 Let's keep it going for John Schleffler, Schneffler, I don't, Schmeffler, got it, I got it, nailed it, nailed it in one. Before we start, this is the last talk of the conference and I just want to say I'm so happy to see all of you nice people here. And how many of you are new, how many of you is this your first, wow, this is all your first time. That is more than half, that's amazing. Can we have a round of applause for the new people? And I also want to say weren't all of the talks great? Just great this year, so good. I'll try to not screw that up now. And also in the spirit of gratuitous pandering I want to show this thematic picture of Jonan. In his constructioneering onesie, as you can see. Now to the matter at hand, in the course of my career such as it is I have made a number of observations about software and based on these it is my conclusion that it fails a lot. Therefore I have written this talk which I am now giving to you the subject of which is, but why? And like all good talks do, I'm going to begin this with defining some terms because that's fun. We're going to be talking about how our software is defined by the processes we use. And how the processes that we use to build our software are defined and governed by our culture. And so I want to go ahead and start by defining culture because it's a word that has been used and misused a lot. And I want to be clear that I don't mean brofists and beers after work. I don't mean preferring to hire people who look like you do. What I mean is the set of behaviors and beliefs and values that we share as a group of people. And then because this is a talk about the quality of software, I also want to talk about quality. Quality is not for me, generally speaking, an ethical problem. Software isn't good or bad. It's just valuable to someone. And since we're talking about failure, I want to define failure as well. And I define failure as a difference between what we expect and what we observe. So failure is a particular kind of change, and it's specifically the kind that we don't like. And before I proceed, I'm going to show you this picture that Amy lovingly drew for me. And I'm going to show it to you for 30 seconds. And I want you to try to remember as many significant details as you can about this picture. And you will be literally quizzed on it later. Significant details like it is adorable. So what I want to start talking to you about today is a concept that I call process patterns, that Gerald Weinberg calls subcultural patterns. And these are ideas that we share within our culture about how software should be built, about what we value when it comes to building software, about how we define our processes. And there's no one process that's followed by all organizations. And that's fine because different organizations face different trade-offs and different challenges. But we can identify some common patterns, and we can start to talk about those. And we can start to see that if you need to increase your capability as an organization, sometimes you have to reach for new patterns. And so Gerald Weinberg wrote a book called How Software is Built, where he talks about these software subcultural patterns. And I recommend you read it in all of the other books in this series. There are, I think, six of them. And they're available on Lean Pub in a really nice bundle that you should go check out. And so the first pattern, or in fact the zero with pattern we're going to talk about, is the oblivious pattern. And this is the pattern where we don't actually know that we're performing a process. How many of you have written some sort of small program shell script for yourself on your own machine? You didn't check it into version control. You didn't open up. You didn't create a ticket system for it. You just wrote a thing and you use it and you're done. You were following an oblivious process. You were following no process at all, the null process. And it's not that following an oblivious process is bad. It's good for certain things, but not so good for other things. If you need a more capable system, you're going to naturally sort of evolve out from this into a new pattern. But if you want to know when this oblivious pattern can be successful, you need a few things. You need to be capable of solving your own problem because that problem isn't too big for you to handle and you need to know exactly what you want to build. So you're the source of the requirements, you're the person building, and you're the person using it. The next pattern, which is the first pattern in the software... Anyway, so Phil Crosby, who was a management consultant, wrote about his own version of these subcultural patterns and he started with the pattern that I'm about to show you and then Gerald Weinberg added this zero pattern. The first pattern or second pattern depending on how you're indexing is this variable pattern, which is one where we just do what we want to do in the moment. And this pattern is characterized by poor adherence to plans and processes and by dependence on the performance of individuals. It's also the first one where people are aware that they're performing a process and since it's the first pattern that separates the developer from the user, it's also the first pattern where blaming appears as a significant software development activity. This pattern is especially common among small young companies that build software products for microprocessors. In order for this pattern to be successful, you need to have a good enough relationship with your customer that the requirements process isn't garbage. You need to be a group of very competent professionals so that because we're dependent on individual performance, you can have some expectation of individual performance. And you need to be solving a problem that isn't too big for your team that doesn't require any additional effort to solve. And there is a myth that variable organizations use when they try to figure out how can they be successful as an organization and it's the rockstar programmer myth. They put out ads for rockstar programmers because hiring a rockstar is the only hope they have of shipping a successful product, so they believe. And because performance of their software depends on the performance of their rockstars, the software quality is about as variable as human performance is, which spoiler alert is quite variable. And so when you have these people, these organizations who need to look for specific rockstars to, by the way, the managers, generally speaking, aren't actually capable of selecting from among their funnel of hires. They don't know what to look for. And so even though they need to find rockstars, they also can't find rockstars. And the other problem with this organization is that if they have processes, they only follow them when it's convenient. During a crisis, they're not followed at all. And the best indicator of success for a software project is which programmer happens to do it. And that means that the programmers get all the credit, but they also get all the blame. So the next pattern is what Jerry called the routine pattern. The routine pattern is one where there's a lot of emphasis made on having a plan. If we can just come up with a plan, that'll solve all of our problems. So let's figure out what our plan is and then force everyone to do exactly that no matter what happens. So if the first step in statistical control is to achieve predictability of schedules and costs, for organizations where this variable pattern isn't achieving that for them anymore, for teams that need to coordinate with other groups so that they need to have some defined process for doing that, they start to move into this routine pattern. And it's no longer sufficient to just lock the programmers in the closet and wait and see what they do. So routine organizations depend on plans and on managers to carry out those plans, but they struggle to deal with the actual changing requirements of software development and the weird things that happen that are exceptional that throw off plans. If a routine organization is to succeed, the problem can't be bigger than what a small team can handle. They have to be able to use their routine process to solve the problem. They have to get the developers to follow the process and they have to magically not run into any exceptional circumstances that invalidate the process and the plan. And these organizations have a different myth. Their myth is the rockstar manager. If we can just hire the right manager, that manager will make sure that the developers follow the plan and everything will be fine. And he looks like that. Organizations that are characterized by this belief in a rockstar manager can achieve a repeatable level of control and quality by very rigorously managing commitments and costs and schedules and changes. But the problem with that is that the key word there is repeatable and not repeated. Routine organizations don't always know what they know how to do, usually because they don't understand why they're doing it. Without understanding the principles and the reasons that underlie a process following it blindly is generally not successful in exceptional circumstances and routine managers in crisis tend to make things worse rather than better. Routine organizations also believe in what anthropologists call name magic and name magic is where you believe that just by saying the name of a thing you gain all of the power of that thing, whether it's agile or behavior-driven development or lean or hexagonal architecture, there are a bunch of names that we think can view us with power if we just say them enough. And we all know, on the other hand, we know that there aren't silver bullets that offer magical improvements to the quality of our software, so why do we keep investing in name magic? So the next pattern is what Jerry calls the steering pattern. And this is really the pattern where you start to gain control over the quality of the software that you're building and you choose the processes that you want to use to build that software based on the results you have already gotten from using them. So these managers, these steering managers of these steering organizations they don't depend on magic, they depend on understanding. And while routine organization managers often come from a successful programming background but they have no particular talent training or interest in management, steering managers have usually had some management training. And I know, for me personally, moving from an individual contributor or programmer position into more management, it's what was Jonah saying before about how you can't just assume that because you know how to write a program, you know how to manage people, I see that a lot in especially new managers. So the processes in these steering organizations are not always super well defined, but there's always a shared understanding of what the goals are and why the processes were chosen. And even if the process doesn't work, there will be a way to recover from that because of this understanding. And that's also why steering managers and steering teams can take on harder, more challenging, more risky projects. And so if you're finding yourself in what you would characterize as a routine organization and you're having trouble solving harder problems or shipping software to more or more demanding customers, then maybe leveling up like this is what you need. So steering organizations are for problems that are too hard for a simple routine to work. And the steering organization managers need to be able to negotiate with their bosses and with their vendors. They need to have control over some control over externalities. And they also need to be able to reject arbitrary schedules and constraints. Now there are two more patterns that Jerry talks about in his book. And I'm not going to cover them and here's why. There was a Department of Defense study in, I believe, the 70s that conducted, that pulled a bunch of organizations where they would rank in this what was at the time called a maturity model. And by the way, I'm not a fan of the term maturity model because I don't think that when I wrote that shell script I was being immature, I just thought it was the thing that solved my problem at the time. So I'm not going to call them maturity models, but they didn't end the survey. So this survey found that 85% of projects were at the lowest level of maturity, what we're calling variable. 14% were at level two, what we're calling routine. And only 1% was at level three. And there were no projects or organizations in levels four or five. So I'm not going to waste your time talking about unicorns. And we're going to move on. So how do you conduct a steering organization? What is the thing that you do that separates it from a routine organization? And that is the concept of cybernetic control, which is the concept of taking in feedback and using that feedback to make good choices, as Ernie would tell us, that will affect the system and then taking in new feedback, new observations about what your changes did to the system and then making new choices and then doing it again iteratively. So how many of you remember, or maybe probably, okay, maybe some of you do, AAA used to have these tryptics. And before there were Google Maps, you would go to AAA and you would say, I need to get from here to here. And they'd compile you this spiral-bound book. And what it would do is it would list out the route and it would list out some landmarks along the way, but it wouldn't list out anything else. And it didn't give you the context that you would need. If you turned off of the route, you didn't have the context to recalculate and figure out where you were supposed to go. You'd have to backtrack to get back onto the plan and then keep going. So this one is actually not a AAA tryptic. This one is from 1789 in an atlas called a Survey of Roads in the United States. So this is a pretty old idea. And the tryptic is limited by that lack of context and understanding and by its single purpose. And you either follow that plan or you get lost. If you take a wrong turn, you can't figure out where to go forward, you just have to go backwards. So in our modern era of Google Maps, we have access to a complete map which provides both context and understanding of the route. And we also have nice robot controllers to help us recalculate if we get off throughout. And so that is the essence of a cybernetic system. Your GPS figures out that you went wrong and then it figures out what course corrections you need to get back on track. So that's a cybernetic system. And you can look at different effect models or diagrams to think about this. And the basic one would be a system that has no controller. You start out with just a simple model of the inputs and outputs of a system. And so for a system that produces software, the outputs are software, but also other effects, other outputs, which include things that aren't the direct goal of the system, but might be things like greater staff confidence with the programming language that you're using. Thousands of failure reports from your customers. A general sense of anger towards management. So the inputs are the three R's. Requirements, resources, and randomness. And when I say randomness, what I mean are all the sort of unpredictable things that would tend to set you adrift. It's the entropy in the system, the things you can't plan for. So this is the entire model of software development as understood by variable organizations. They have no concept of the controller in such a system. There's no opportunity to exert control. And what you find is that requirements go in and software comes out and you can't explain that. So for routine organizations, and if we want to exert control over the system, we need to connect that system to a controller. And the controller represents all of our efforts to keep software on track. Managers, developers, everyone has the chance to exert some level of control to keep the software on track. And at this level of control, however, the controller doesn't get to make observations, so it's pretty limited. It can just say, no, no, no, you need to stick to the plan, but it doesn't know how far off of the plan you really are. So to get to the next level, to the steering organization, you need to start taking in feedback from the state of the system itself and from its outputs so that you can use that feedback to change the requirements and to exert some control over the system. And so an effective method of limiting productivity losses due to sickness is to send staff home when they start exhibiting signs of illness, but you can't do that if you don't notice that they're sick. Right? So the routine pattern controller that we just looked at can't do this because they're not taking in observations. And a more versatile and effective model of control is this feedback model. And because you can take in measurements and observations of the system, it allows you to make a prediction or to make an expectation about how you desire the system to be. And then as long as you have the ability to actually observe the actual state of the system and the ability to compare the desired with the actual and the ability to act on the system to bring it into convergence with what you intend, then you can start to exert cybernetic control. And cybernetic is a big word for the domain of science that talks about this stuff. It's actually fascinating, and there are a bunch of books that I guess generally don't even talk about software, so the human body is a cybernetic system. And when the human body's internal systems fail, we go see a doctor, and so that larger system is also a cybernetic system, so they're all over the place. We just happen to have a few of them here in software land. One of the problems with working with cybernetic systems is that many of these systems are nonlinear. And what I mean by nonlinear is a nonlinear system is one where the inputs don't always correlate to the outputs. Sometimes when you put in the input, you get this output, but sometimes you get a different output. And for our purposes, that's a close enough definition of nonlinear that we can work with it. So at the beginning of the talk, I showed you Amy's picture, and I want you to try to, in your mind, remember how many cats were in it. Don't call it out or do. Just try to remember it, and now I want you to tell me if you were correct. Now, how many of you notice that this is not the same picture I showed you first? A lot of you didn't, and we can talk about why that is. Some of it is change blindness, inattentive blindness, but that's another talk, and I don't have time for it. What I actually want to show you is that what we have here is a Spot the Difference game that maybe some of you played when you were kids. And Spot the Difference is failure detection in a nutshell, right? We're trying to find, we have an expectation on the left, and we have something we're given on the right, and we're trying to find where they don't match up. And so if I had you all play this game right now and record all the differences you found, and we graphed how many differences you found over time, what do you think that graph would look like? Would it be a straight line? Allow me to show you my highly technical data visualization that took me at least 30 seconds to describe the actual behavior of that system. So you start out strong noticing a bunch of differences, but then it tapers off. The rate at which you notice differences decreases over time, and this is a nonlinear system. So why does that happen? Why is it not linear? Well, there are two simple answers. One is that once you've found some differences, fewer differences remain, and the other is that you start out by noticing the easy differences because they're easy, and then the only differences that are left are the hard differences because they're hard. And so if you pick a spot in your progress and you make a mark and you say, well, I've noticed 10 differences in two minutes or whatever it may be, if that's the only information you have about the system and you try to make a guess about how long it'll take you to finish spotting all the differences, you're going to end up with a prediction that looks like this, and it's very different from how long it will actually take you. And humans are really good at doing that thing and really bad at understanding this part. We are really good at making predictions about linear systems. We can catch balls when you throw them. We can do a lot of... Our brains are good at linear systems, but they just fall apart when we have to deal with nonlinear systems, especially when we don't bother to try to understand what is causing the nonlinear behavior. And if you think about this, when you're looking at a spot-the-difference game, you don't have a selection mechanism that you can employ to say, no, I want to find this sort of difference first. You see them as you see them. But if you think about which features should I work on first, if you think about which bugs should we fix first, you can employ a selection mechanism. And the first duty that you have when you're making selections is you have to be better than random chance, which sounds obvious, but it's actually hard to do in some of these cases. And if you are fixing bugs, a natural bias would be to fix the easy bugs first because they're easy and save the hard bugs until later because they're hard. But what does that do to the amount of risk in your system over time? Do we want to work on systems that are inherently more risky over time because we picked a faulty selection mechanism for sorting out our bugs? Think about if you picked the riskiest bugs first, and that's all you did. Do you think that working in a system that became less risky over time would be better for the overall outcome of your project? And I'm not saying that risk alone is the only factor to consider, but it's better than random chance and the opposite is not. So requirements gathering, feature development, bug fixing, everywhere you look you can find these nonlinear systems and you can find managers using bad, and programmers using bad linear models of those systems and getting bad results. And it gets even worse because not only are some of these systems nonlinear but the complexity of the interactions of the entire system makes it nonlinear as well. And so this is an example of the composition fallacy, which is the idea that the composite system is never more complex than the sum complexity of its components. But it turns out that the interaction between those components causes increased complexity. And then while we're here, I should also mention the decomposition fallacy, which is that if all of the subsystems work, then necessarily the entire system must also work. Just in case that's useful. So we ignore interactions between these systems at our own peril. So I want to talk a little bit about how we can deal with nonlinear systems. How can we prevent our plans from going awry because of unexpected events or poorly understood systems? And there are a few things that I can recommend. The first is to introduce governing actions into your feedback models. So if you make choices that regulate the add-on effects of positive feedback loops, if you prioritize work that's risky first, for instance, then the system is still nonlinear, but we've inserted a governor into it that prevents the explosion that you might otherwise get. It prevents it from running away with itself. And another idea is to act early, small, and often. Because the earlier the act to correct a system, the smaller the change you need to make. Smaller changes are less risky, and it's easier to predict the consequences of smaller changes. And the more frequently you make corrections, the more chances you have to catch problems before they become too big to solve or too risky to solve well. So these small corrections, these small choices that we make when we're trying to understand and correct our systems, these are the fuel of this engine of cybernetic control that I'm talking about. But in order to find problems, you have to see them. They have to be visible. If you are unaware of a problem, then you're not going to be able to see it and fix it. Controllers need feedback to operate effectively. So anything that you can't see or measure can't be taken into account. So how many of you use the GitHub pull request model where as soon as you have something worth showing, you put it up there, you create the pull request, then you get a bunch of feedback? I think that's a good model. And GitHub likes to stress that this is the place to hold conversations. And I agree, and that's super valuable. But I also just think the fact that it makes work products visible when they wouldn't have otherwise been is very valuable because managers can't fix a problem with work they can't see. And you'll be less effective in controlling the system that's producing the work if you can't see the work being produced. And that leads to the next point, which is if you can't understand it, you can't fix it. If you don't have a model of the system you're trying to work with that is sufficient so that you can make predictions that are somewhat accurate, then you can't make good choices and you don't know what the result of that choice will be and you're just driving blind. It's a random walk. And so let's talk about models for a while. Each of these cultural patterns use models to guide their thinking whether those models are implicit or explicit. Models support the need for understanding and for clear, correct communication. Explicit models also supplement the vehicle of words with these diagrammatic pictorial things that are dynamic and possibly nonlinear, and they can help show problems and show interactions that words alone struggle with. So Frederick Books, how many of you are familiar with Frederick Books? He wrote this book, The Mythical Man Month, it's decent. It's one of the best software books I've ever written. So he said, more software projects have gone awry for lack of calendar time than for all other causes combined. Now do we really think that the actual problem was they ran out of time? No, neither does him, neither does Brooks. So what he's doing is he's presenting a particular model that people use to understand why their software failed. We ran out of time, right? He's making a model explicit so we can examine it. Because most of the time, of course, calendar time isn't the root cause, it's just the symptom of the underlying problem. It's really the result of a bunch of other failures in your process that led to you not having enough time. And so, of course, Brooks knew this and so he spent basically the rest of the book providing you with better models than that one. And if we don't make our models of the system explicit, then when someone asks why the project failed, we won't understand the real sources of failure better, well enough to say anything other than well, I guess we didn't have enough time. So managers use models to govern their management decisions, whether those models are implicit or explicit, and if you make your models explicit and you share them, then everyone can gain understanding of those. And they become open to criticism and improvement. So here are some examples of some implicit models that managers often have. Let's talk about this one for a second. So managers like to think that if you input 10 units of pressure, you get 10 units of increased productivity out. They like to think that there's a nice linear system there. How do you think it really looks? Allow me to present you with another finely crafted intro diagram here. But that's not the whole story because not only is there this nonlinear effect, but there's also burnout. And what happens when you burn out is your productivity doesn't stop increasing, it drops off a cliff, and not only productivity, but your happiness as well. So here are some more models. Okay, only there's actually the one. So implicit models are a characteristic of variable organizations. Organizations, so this one in particular, I know how to identify the best programmers is characteristic of variable organizations. Variable organizations need to find rock stars because otherwise they won't succeed, so they have a lot of incentive to believe that they can in fact find rock stars. You won't find a lot of managers of variable organizations saying, I mean we sure don't know how to pick good engineers. Because the entire premise of your success is based on picking good engineers. And unfortunately for them, there's very little evidence to support their belief that they can in fact pick good engineers. So to sum this up, because I want you all to go have some drinks or not, as is your choice, we need these feedback control systems, these cybernetic systems at all levels. They should inform the way we write our tests and they should inform the way we make strategic project or product decisions. And with these models, we gain the ability to think and observe our systems in terms of our understanding of nonlinear effects. And so I guess that's really all I have for you. I hope I've sort of convinced you to spend more time thinking about how you understand and guide your software and your processes and your culture. And the last idea that I want to leave you with is if software is governed by process and process is governed by culture, well, culture is governed by people. And if you really want to make better software, we need to spend more time thinking about people. So that's all I have for you. Okay, I have one more thing. I also have this gif of Vader and a little girl. No, no, no. No, still no. Okay, good talk everyone.