 So, yeah, my name's Mike Feathers. I've been in the industry for ages and work with Optiva, but Optiva was recently bought by Groupon, so we're calling it Grouptiva right now and that's the way things stand. The keynote, basically, is about something I'm terming code blindness. It's something that I basically see as an affliction in the industry. And it's really kind of strange because I don't know. I'm not sure if other people notice this as a problem. I tend to and I just want to talk about it a bit in ways we can kind of move past this. How many people here have bosses that look at their code? Okay, how many people here have people two layers up in the organization that look at their code? Kind of rare, right? Okay, I'm guessing this being bolder that many of you are part of startups. Is that fair, some of you, right? And quite often, you know, we're doing consulting work where you're basically going and making, you know, like web apps and things like that for bigger companies and stuff along those lines. The thing that's really kind of striking to me, though, in our industry is that we've had this disconnection between software developers and business for ages and ages and we've tried very different things to go and move past it. But we really are dealing with the effects of this day in and day out and I think it's something we really have to kind of address. For me, this was kind of an odd thing to go and kind of get into. I have this little slide here basically saying that we're going to go and get a little bit deep right now after we've had our meditation. I kind of got involved in the industry in the 90s and I actually joined a biomedical company and it was very striking for me to go and basically see how software development was perceived within that company. The thing that was kind of tough was that essentially we'd have people developing medical instrumentation and they have to get all the science right and they have to go and piece things together, multi-year development cycles and then we'd get down to the point where basically they had to go and put in the software that was going to go and basically do the analysis that was necessary for the biomedical stuff to happen. Very key integral part of what was part of this. The thing that was so funny, though, was that we were the very tail end of the process, right? Develop all the chemistry and all this other stuff, the hardware and everything like that and then it's kind of like everybody turns around and says, okay, are you done? Where's the software? And it was very tough for us in that organization to really explain to people what it was that we did because we were essentially in a hardware company. They understood chassis and machines and mechanical assembly lines and stuff like that and all that software stuff is just ethereal. It's just stuff that's the stuff of thought and we can't really basically deal with it. So I kind of noticed that sort of institutionalized hardware culture in many different areas as I became a consultant. I spent a good deal of time working in telecom and it was striking to me after a period of time to go notice that I was feeling the same thing. I can't say which telecom company I was working with but the thing that was so funny was that they had an incredibly large software development organization but still it felt like the people who were the people who were above the software development organization were kind of like baked into the culture. They just didn't really think of themselves as being a software development organization and they didn't quite understand software development thoroughly and as a result ended up making some very strange decisions. So I don't know it's easy to look at this sort of thing and say oh that's just a side effect of something strange that happens in some segment of the industry but the fact of the matter is I've seen that same thing in other places also. Some of my most exasperating experiences as a consultant have been working with banks and insurance companies. I won't say all banks and insurance companies because some really have their heads on straight but there are cases there are places I've visited where it's kind of like you know they look at themselves and say you know our core business is selling insurance right and we do all this actuarial stuff and all these other things and stuff like that and yes by the way we need software so we hire a bunch of software developers we put them in that room over there right and they basically do all their stuff and we have to kind of somehow manage them and somehow contain costs and then we hopefully end up with some good stuff happening. Sound like a recipe for success? Yeah it's really really kind of tough and banks are the same way sometimes. There is this area of the world right where basically you have people who need software they aren't really software savvy they hire software developers to go and do work for them and that's the paradigm we've been living with for ages right. Now agile software development right it's been around for about 10 years or so I first got involved with extreme programming back in the early 2000s scrum became ascendent a bit after that it's funny about all this because the thing that agile was really after was this agile when you look at it from the outside it was really a way of going and getting business and development to go and talk together more right to go and get them on the same page about things because essentially the guys who started these agile software development methodologies you know they basically were encountering very dysfunctional situations like going various places and having you know development organizations that can't deliver anything for their customers having customers that are basically just sort of begging for any kind of software at all or getting the wrong software back stuff along those lines and so both Kent Beck and Ken Schwaber you know the founders of those two methodologies kind of put a enough process in place to make development and business kind of play well together right and you know we see that these things are quite successful you see them spread all throughout the industry and stuff along those lines I feel though that we really don't really go deep enough for where we want to go with this sort of thing okay you want to have a good relationship between business and development but the thing that's really kind of tough is that it's not just as simple as that okay think that ultimately what we really want is we want organizations to be able to make good decisions going forward right we want them to be able to make decisions about software development that are rational okay that make a lot of sense for their current needs and their future needs now it's kind of funny how do you transmit that information across an organization I don't know you know let's think about let's think about agile stuff right let's think about scrum right what do we do we basically go ahead and we determine a product backlog and we basically go ahead and sort of like do estimates for the various things decide what's going to be inside of a sprint and we just march on and go right and we do that sort of thing so like the primary piece of communication between the people who want the software and the developers is okay that's what you want yeah we can do it it's gonna take this much time right doesn't seem like a very rich form of communication somebody ever worked in a code base that started getting bad after a while really okay started yeah okay this is the very funny thing about this because when agile first started one of the promises that we had in agile was that by using certain technical practices what we were going to do is keep our code bases healthy for a long period of time right and I'm sure plenty of you in here use testing methodologies you use refactoring and all these things you only keep things going does it tend to fall apart anyway I know you can be successful I mean you can be pretty successful in an organization that understands it takes time to develop software and basically you know tries to go and do things in a very sustainable way and sort of like listens to the developers when they say no it's going to take this long to do this and if you rush it it gets bad and that sort of thing but the thing is it seems like you know without having that kind of understanding across the organization you really get into a situation where it's easy to cut corners it's easy to go and target something very quickly and never really make up for it and as a result you just get into the spiral of deterioration and we all know what that's like right we always get into that we always but quite often we get into that spiral of deterioration and this is where things get really exasperating because over a period of time then people say oh you know I need feature x y and z to be able to compete in the market and the developers are like yeah you can have that it's going to take three months and they're like oh why would it take so long it's like oh because it's like the code isn't really situated to do that sort of thing right so very strange thing with this it feels to me like what we're missing is basically a view of the code across the organization okay that basically people who are non-technical need to understand what the ramifications of their decisions are and how that's going to affect the value of the asset that we have right now okay and what it's going to be in the future now the funny thing about this though is I've been accused by some people when I talk about this of being thinking that the entire world should be t-shaped right everybody should be technical and I you know it's like as a personal development background it's very easy for me to see things strongly from a development perspective but I can't help but notice that it seems like many successful organizations tend to have technical knowledge flowing up and down the entire organization right there are places where inside of the company they kind of see themselves as tech-saving companies as opposed to like well you know we're a retailer but and we need a website now it's like oh no no we're we're an e-commerce company and we understand this and this and this and this and then when people are making trade-offs they can understand what the ramifications are against the asset the code they've got right am I working in an organization like this they have that kind of thing yeah a little bit there's I'm seeing a couple of hands but I think this is the thing that we need to go and really really dig down deeper into so I'm gonna talk about a couple of things related to code blindness what I've noticed is it seems like there's several different levels to those okay first thing is total ignorance okay I've visited organizations where past the certain past the line managers it's kind of obvious that people don't really realize that people are developing software it's like they see on paper what people are doing and they know what the milestones are and various other things but it's kind of like they make decisions based upon not based upon software concerns but based on things like staffing concerns a company I visited like three four years ago they had many many many software developers and then when you see how they were doing things it was kind of like oh well we have this project starting we're gonna need five people over here this one over here okay we're gonna move you know 10 people over there and these were all projects but they were all working against the same huge code base you know we talked about the code base at all they just talked about the planning of particular projects and how to do resource allocation across all those things okay and it you know I think if you think about it it seems almost insane right that nobody had a vision of the underlying asset that they were working with and they made some really poor decisions right it's like oh why are things slower why is it not working as well oh but nobody's tending the asset right so total ignorance once you get past that what many organizations try to do is this they try to go and basically target metrics to move forward and go and make things better okay anybody working in an organization that does this at all okay some people do and it's a funny thing with this it seems like it's an easy way to go and try to get some sense of control over what's happening in software development the thing is it's got many many different pitfalls right you can use you know various tools to go and sort of like figure out what the maximum complexity of various methods that you have in your code bases and what the cohesion and coupling are and stuff like that and then quite often people will try to manage by those numbers and say ah you know we really shouldn't have any methods longer than 50 lines no reason for method logging with 50 lines and then people sort of go and start doing the little things it takes to mutate the code base to go and basically meet these particular metrics one favorite happens to be things like test coverage right have you ever worked in an organization where people say oh let's go and increase test coverage let's do it every sprint we're going to go and have end percent increase raise your hand so many people don't have that okay does that work out well no okay and the first time I saw this it was just really striking I was in Texas visiting a company and um they wanted me to go and look at the quality of their tests and they were really poor tests they didn't make sense they didn't have good naming they tended to be rather long it was hard to go and make a sense of what get any sense of what they were testing and over the course of the visit I realized what was going on is that basically um about a year prior to my visit somebody basically said okay test coverage is important because quality is important and stuff like that so let's go ahead and start ranking the teams based upon how much test coverage they have okay now the funny thing about this is that essentially developers will do what you ask them to do right and it's a funny thing it's like even if you know that you're being ranked on that at a higher level of the organization it's hard not to go and basically say look that's just part of my job it's getting that number high right and you get that number high but the thing is since it doesn't really it doesn't really connect tightly to the goal that you really wanted to go and achieve things kind of fall apart you end up with tests that don't mean very much so that's the danger of metrics essentially metrics are great at times but the thing is that it's very easy to go and since people call this gaming the metric I think it's really just the case sometimes that people just you know rationally going and behaving as if the you know the metric itself is a goal and you can end up with really really crazy effects with this sort of thing so yeah some organizations will try doing this sort of thing I tend to think metrics are okay the thing is you have to always look at them with a bit of an eye towards well maybe it doesn't really you know what's the context around this particular thing if you don't things tend to fall apart so metrics craziness I went a little saying I have about this is that there's nothing more abstract than four right it's like once you basically take anything that happens in the real world and abstract it down to a number it's kind of like you lose all surrounding context so we shouldn't be surprised that by targeting metrics we can often get ourselves in a bit of trouble so the next thing along the path is basically what I call qualitatively assessing what you've got right like I said with numbers we can kind of get ourselves in trouble but the thing I've been wondering about quite a bit is can we get into a situation where we can express to the people who we're working with that are not technical what the ramifications are of making quick changes or dirty changes or doing things in an odd way or why certain estimates take longer and stuff like that so the thing I've been playing around with with this is basically to go and throw numbers out entirely and instead to go and make little pictures right so this is essentially a picture of a class okay and the boxes basically are fields in the class and the ellipses are methods and you have a directed edge between you know each of these things when one uses another does it seem like an okay class very classy right so I you know I said numbers are very abstract this is pretty damn abstract also right but here's the thing so to me this is you know it could be better could be worse it looks like we've encapsulated the fields we have you know a small number of methods I mean this one seems okay too looks like we've got a recursive relationship there between that method in itself we've got the odd thing of a public method that doesn't use anything else in the class but that's actually okay sometimes you have utility methods what about this one right here how much training do you need to know that that's bad okay and I you know I wonder about this could this be a means of communication with the rest of an organization you know you go and say okay well you know it's like yeah you know we're going to do this and this and this and you want that feature but the thing is if you want that we have to touch that and that's why our estimates are higher right that's an interesting conversation because then the next thing is like why did you let it get so bad right okay and that's a fair question also right but the thing that I think is interesting about this is that we can start to have those conversations and pretty much educate the people in the rest of the organization about you know this thing which is so important which affects you know what the business is able to do in the future right right now they've given us that responsibility as developers you want to basically tend that asset but we also know that we're constantly doing this thing of fighting back and forth between you know people who want something tomorrow and our responsibility to tend the asset I think distributing the knowledge can kind of help us out a bit more next thing I see is a progression in this walking out of code blindness is life cycle managing and I don't want to use like the ALM word I've you've heard of like application life cycle management but it's something which happens kind of a bit in the enterprise world where people use tools going manage things and stuff like that but quite often it turns into project management in that sense you're looking at things through the veil of staffing rather than what the real asset is what the code is behind things you know when I talk about life cycle management what I'm really talking about is as if we were doing biology right it's kind of like it's very striking to me that I don't know if anybody else feels this way but do you ever feel like your code is this living thing at all okay it has its own growth laws you add to it you add in certain areas it's easier you add in other areas it's harder and sometimes if you go and you carry the analogy but further it's like oh no this is the monster in the box right this is the thing that's kind of difficult to deal with and it's only difficult because we kind of let it grow beyond its bounds so maybe we can get to the point where we're really going intending the life cycle of particular things that we're working on and paying attention to what their quality is and making very good decisions about hey it's not just refactoring sometimes sometimes it's actually tossing something and saying hey we want to go and say look you know the cost benefit ratio of dealing with that particular thing is so bad we're throwing that piece out and we're rewriting it okay now rewrite gets an awful lot of bad press right maybe we can go and find a better way of doing that it's kind of striking I'm kind of noticing now that we have this built-in assumption that code is supposed to live forever right maybe it just gets so bad sometimes that you have to go and sort of strategically deal with things in a in a more you know violent way so anyway I think the last stage for all this really is integrated knowing I think that organizations that can do the best of software development are ones where the organization as a whole understands the asset that they have and to a way to know that you have this sort of thing I've seen some organizations which approach it a bit let's skip that for a second is to go and notice how they basically deal with staffing issues in the team okay in organizations that are doing this well what they do is they have people who understand the code they understand how you know what the architecture is and what the real qualities of the various pieces that are there what's complex what isn't what's troublesome what isn't and what they do then is they basically start going in allocating talent to different areas of the code base based upon those particular decisions okay and it's kind of like looking at the entire you know system of code and developers as being like this big symbiotic thing right so you know it's kind of funny that's not just going and saying I need 15 developers working that area of the code that's kind of like saying I need Jeff and I need Sally over there because they know this particular thing and if we can have them over there for three months we can basically go and meet this goal and do that sort of thing so it comes down to knowing the people as well as knowing what the asset is that you're tending knowing that code base okay making decisions based upon that so why doesn't this happen in many organizations any ideas yeah really okay I don't know if I got gain control here but I'll put it up okay so better better okay thanks a lot one reason why this doesn't happen quite often in organizations is because of the way organizations are broken apart in essence right you can say well you know use this as an example I use like Joe and Sally if we had them on this particular team doing this thing things would be better but quite often you have a manager for Joe and Sally and the manager goes and says no I know you need them over there but we need them over here right and it's kind of like it becomes a fight between people competing for resources and quite often that sort of thing becomes subject to politics rather than basically very rational decision-making about what needs to happen within the code base so anyway the the social structure we have within an organization really does affect the technical structure of things I have to have a slide about this every time I talk any place you know Conway's law at all how many people have heard of this before okay some not everybody okay so I'll just explain it for people who have never seen it before okay there's a guy named Melvin Conway in the early 1970s he basically came up with this idea and it took him a while to get published but he published the idea that essentially the structure of software or the structure of any designed artifact tends to be a copy of the communication structure of the organization that produces it is that hitting a light bulb for anybody yeah okay uh yeah okay and you know quite often you know when I you know mention this to people I sort of like see a couple of faces in the audience go oh no you're right never thought of it that way before but that explains all our trouble you know that kind of thing but you know it's and the thing is it's kind of like it's like how could it not be that way right I mean if you have people that are talking to each other every single day you know they're gonna have much you know much more concerted decision-making inside of things that they have you know like weekly conversations then you end up developing an API between the teams because you need to go and reduce the amount of decision-making across you know that in your development so it's a very typical thing that happens you know and it's something that you know quite often many organizations don't realize that their software ends up being shaped like their organization and that's just you know a bit of a shame but it's something that you know when you're basically trying to approach software development from a very holistic point of view you need to be pay attention to you want to talk about something else also and that's just the notion of technical debt you ever heard of this one at all yeah okay this is a metaphor that was first put into the industry by word Cunningham years ago and he meant it in a very I guess optimistic way saying that if you're a development team you're working on something sometimes you can basically afford to go dirty a little while in order to meet a short-term goal but that's like incurring debt and what you need to do is pay back that debt very quickly otherwise it tends to escalate right today people use technical debt to mean that but they also use it to mean just the natural deterioration that happens in the code base when things start to go crazy right when it's really hard to refactor things when there aren't many tests when it's really hard to add features and because things have just kind of fallen apart now the thing about this though is that essentially technical debt is one of those things which is hard to measure right but it's very easy to go and see that it happens that it quite often ends up being a runaway cost like we're not really even aware that it's happening I talked to somebody at Google a couple years ago and they said that they do a lot of tracking internally of how their developers are working and the effects on the code base and stuff like that and they had this very interesting thing that they noticed you know they tend to hire very smart people who know what good design is and can write good code and everything like that but they kind of noticed that as a team is working in a particular area of code it seems like things would start to fall apart a little bit in terms of certain you know qualities like okay cohesion would fall apart coupling and stuff like this but the team itself wasn't noticing and the reason the team wasn't noticing is that they were kind of like burned into it a bit they kind of see this code every single day and since they see the code every single day they don't notice the problems or they don't see them as problems because it's like oh I understand this I was here yesterday that kind of thing and then it kind of gets to this particular point where it's kind of like things have fallen apart so bad that the refactoring becomes more difficult than it would be otherwise and it's a strange thing we can see this as being basically a you know a side effect of you know this thing here boiled frog syndrome you ever heard of this at all the old anecdote with this is that if you stick a frog in a pan of water you know that's boiling it will jump out if you put it in a pan of water and you basically slowly raise the heat it's just going to die in there because it doesn't really notice the incremental differences and it just boils alive right in many ways I think we end up being in a state where we can be boiled alive in our code basis turns out though there are things that we can you know do we can basically track various things one thing I wanted to go and show you is this this is a project a couple from friends of mine were working on and you know very good developers very conscientious and everything like that what this picture is of is basically the amount of code complexity added per commit okay and this is basically just a timeline from the very beginning of the project so you can see like a massive increase here I think they just pulled in some external code at that point but anybody noticed anything else about what's happening here on the on the baseline looks like the complexity is increasing over time the complexity per commit does that happen in every project you'll notice that there's not too many little bulbs floating around on the top on the left we have a lot more over here on the right okay slowly over time the amount of complexity we add per commit increases is that true of all projects yeah very possible right so it's kind of like you know and these are these guys are diligent refactors right they're always going back in the refactoring things and they're working with them there are certain qualities certain growth laws their software basically obeys and we just don't really know what they are yet yeah so as you're saying it's basically as as time goes on you're adding more features as you add more features then you have to adjust for them and then you might go and be adding more complexities you go on over time doing that sort of thing there's a funny thing about this too because if we think about this as all being like a biological system in a way it's kind of like a gross it's growing like moss on a tree and all this other stuff there's a funny thing that happens in biology that never happens in our source code what happens to biological organisms after a period of time they die they do right and you know it's a funny thing with this do we ever go ahead and kill code on our code basis sometimes we'll take code that we find that we determined just isn't really you know needed at all and we'll delete it right but we were never really aggressive about that right we're never really aggressive about going in and saying hey you know it's like this this code right here let's really really find out whether this is really being used or not and if it isn't let's get rid of it okay and there's a step beyond that too the step beyond this this is going to say look here's a feature in our code base and maybe it's not generating all that much revenue right and maybe what we can do is we can say look you know we can afford to lose that part of the market okay and kill this feature remove all the code out by its roots okay even though we're going to be losing a bit of money but the amount of money that we lose is going to be negligible compared to the carrying cost of that code right the cost of carrying that code and it getting in the way when we were going in trying to add in new features later that might be more profitable okay but nobody's doing this sort of thing as far as I know nobody's going and sort of looking at the big picture and saying look what can we do to go and sort of like say oh time to kill this particular thing instead quite often we end up with these these geriatric code bases we try to prop up forever you know and I I could tell you stories you know I've been to China and I've seen code bases that have been propped up for years and they have like an entire ecosystem that comes with them of tools that aren't made anymore and you know trying to find people that will actually work on them it's just crazy stuff so anyway um I want to leave you all with three ideas even though we've kind of meandered around a bunch of different areas um I think that if we're going to try to go and get past this thing of code blindness which I consider to be a problem I hope you do too um it requires us basically going and doing a bit more monitoring of what we're doing in our applications okay um I'm not talking about metrics per se as much as like you know just being able to go and find early indicators of problems in code bases so you can sort of like you know send in the attack squad right it's kind of like we notice the things are going a little bit crazy in this particular area let's go ahead and focus attention on that particular thing and that kind of decision making is not something you're going to get in individual teams it's something you have to go and basically get as a whole in the organization you have to have some people recognizing when things are about to go bad and taking action upon it and because of the boiled frog quite often the teams that are closest to the code aren't the ones to tell you this right another thing I think is important that we just don't really have yet is really assessing what the value of software is economically right so we can basically make better decisions about this sort of thing and the third thing really is planning for replacement um I think we can probably get to something at some point in the future where we're designing applications in such a way where it's easy for us to go and say this particular piece of code it's not that we can't refactor we can't refactor we've got tests and stuff like that but it's small enough for us to replace and rewrite and more cost effective for us to rewrite this little piece than to refactor it and that could come down you could get we can get to a point I think where people are designing applications in this way for the potential replacement of pieces of themselves but it's just something that nobody's thinking about as far as I know right everything is all about prop it up keep it alive or do the big bang rewrite which we know is problematic so anyway so came here give you a keynote some things that I thought might be kind of useful and then it's up to you whether you find them useful or not and um thank you for listening so thanks any questions comments at all a few minutes questions or comments yeah sure yeah I was using um flog in ruby sorry oh yeah okay so the question was how was I assessing complexity in that graph and I was just using flog which is part of metrics foo for that sort of thing but you can use just about anything it seems like what's interesting is most complexity measures tend to correlate with each other pretty well I don't quite know yet the question is you know I mentioned vital science what are good examples one thing I'm really trying to dig into now is doing a lot of repository analysis I'm hoping we can get to the point where basically we can start to see little indicator patterns in the commit history of code bases that indicate when a place is about to go bad and we can give it attention before things start going wrong this is another place where the medical analogy is kind of useful right you go to a hospital and they can basically monitor your heart they might be able to go and see something there which is an indicator that things are about to go wrong and I'm guessing that that sort of thing is in there I don't know what it is yet I just think it's a really interesting area yeah yeah so the question is it seems like I'm downplaying metrics but don't I need to do some metrics in order to go and have the vital science and stuff like that I think yeah I think the thing metrics are dangerous when they lose context right so it seems like it's terrible to go and basically take one rep metric consider representative of everything and then use it as a goal so the answer to this sometimes is just basically more information so you have more information in front of you and also never basically making things goals and also making sure that the team is aware of a particular metric and you know you're hoping that they kind of targeted a bit make sure you don't do that forever just retire the metric at a point in time so thanks for your time