 Welcome to Agile Roots 2010, sponsored by Version 1, Rally Software, Vario, AmirSys, Agile Alliance, and Xmission Internet. Toxic Code, Technical Debt Analytics by Israel Gatt and Joanne Hines. Thank you, Andrew, and my apology to everyone for the set up of the crisis of 2008 and 2009. He's really the person who first coined the term toxic asset, and later on through his behavioral and various actions, taught us a lot about what toxic assets mean and what they can do for us. The reason that I'd like to begin with maybe a lot of way to subject city is because many of the lessons of the financial crisis, the toxic assets that lagging, are actually a significant software, and I'm content that they are more applicable to software than we usually realize. So I would start with speaking a little bit about those concepts, namely the financial concepts, the technical concepts. My prediction lines will move on and show specifically how we measure some of those concepts. We should then move on to how we can use those governance of the software process and finish with a few quick takeaways that everyone can implement tomorrow. Now, starting with the notion of technical debt, this was coined by World Coming Home years ago in one of the software projects in a financial company. And the core idea is that you grow money. I'm sorry that you grow money. Namely, you have a deadline, you are like to come fast to market. You do this shortcut, you do that shortcut, and you manage to come to market on time, over another time, and get all the benefits of every feedback, every experience, possibly some evidence. So the notion on the market side is I grow some time, and it's a group of core. Namely, once I cheat, I am to come back to my core, refactoring, fixing, and so to speak, payback the debt that I have taken in order to expedite it. So the notion is limited by our growing time. However, if we are to listen to our normal language, each solution in equivalency between time and money, and you know, we save time, we waste time, we invest our time wisely, and of course, time is mine. So, for example, I do some last coding in order to come to time one week sooner than we had expected. I have to definitely conduct one week. And you can look at one week, and what is the monetary equipment. So maybe it's $2,000 or $3,000 or $4,000, then it's the cost of my week. But the amount of money that I owe for the project is $4,000. So, going from borrowing time into the equivalent of borrowing money lacks the good between the financial concept and the technical concept. Namely, how precisely you borrow money and how precisely you pay it back is a little lucrative, while if you look at money, there wasn't so good how you borrow money, how you return money, how great you were at this point. Now, this actually is quite important, I think, of the thing, the one between time and money. Because once you look at your technical debt as the amount of money which you owe, all kind of interesting things can start happening in terms of looking at your debt versus your cost versus the value of the project. For example, if you go back to the statement here, the one about toxic assets in the financial context, the characterization of a toxic asset is an asset with the nominee is the equivalent of the value. So nothing is left in it when you take the loan which is 100% of the value of the asset. The same thing happens to us and again more frequently than we might realize in terms of cost. We expect the cost to generate 10 million dollars or 15 million dollars or 20 million dollars in revenue. However, the situation might be that the cost will only generate 5 million dollars in revenue. So all of a sudden we have a code or a code debt where our technical debt money exceeds the amount of money that we can expect to get in the market. And this is what we call a toxic code, namely a little coin in which I put so much technical debt on my code that the revenue stream in the future will not compensate for it. And you of course have to assess the question if it was fixing it in the first place, even though everyone never gets an exit. So the relationship of time to money is a conceptual letter. Is this thing the certain amount of time that they need to pay back and then the dollar amounts that go with it. Specifically, when you look at the code at a certain point in time, the way you go about calculating it is follows. You start with code analysis and the code analysis in this case can give you all of the quality deficits that you have in your code. Against each one of the quality deficits, you take some industry norms as to how long it would take to fix the deficit. For example, it might take on the average 15 minutes of time to fix missing braces in a journey statement. You sum up all of the time 20 minutes to one or the other three hours from another deficit etc. and you get a grand total for the amount of time that you need to pay back. And then you use your cost data to monetize it. So for example, if you don't code analysis and find that the grand total of all the quality deficits is equivalent of one year, then the technical debt in terms of money is something like $150,000 or $200,000. So what we have here is convergence between two areas of quite different disciplines. On the one hand we have the financial discipline and on the technical side we have the code analysis techniques and metrics which enables us to get the two things together. Now, how does our technical debt get generated? And it gets generated in an explicit form as part of this also location. We are going to allocate so much money for doing new features. We are going to allocate so much money for post-cutting, so much money for tactical sets of activities etc. and we allocate no money or very little money to the business of pushing back on the technical debts and reducing it on an ongoing basis. So the explicit form is basically some resource allocation that we as managers or managers take in terms of where would the money available to our enemy go. Now, the quote that you see on the board, on the project of debt, means to one of my clients, this was from the city of debt, when I really asked him about the situation, how it became so bad in terms of the quality of the code, the answer that I got was basically the resource allocation decision. Namely, they put all their resources into testing functionalities. Assuming that this would be good enough that they would be able to get into the other 36 for him and completely neglect that, anything related to unique testing in this process. So as you, in the allocation of your resources, even if you are doing testing, can create a situation that the technical debt owes you. Two other forms which I would say are the nature of things. One is, I have the best intention as an organair to come back to my code after I shift it and really do the refactoring etc. But the purpose of the pressure on me to put out this function to put out that picture is so relentless that I never managed to find a way with time to go back to my code and fix it even so obviously one year down the road. My memories of what I need to fix are much more than my memories right now. So what happens while the obvious economic advantage is to go and fix the code as soon as you can find the time. The ongoing pressures often lie standing in the way and that's like the ongoing economic pressures to put the function picture out in the market in a situation where I'm not able to come back and in fact after an hour you'll be able to do all of this like an ordinary one week worth of effort to go back and fix the code. So the other thing that happens and is a phenomenon that perhaps we don't pay our attention to is the software decay. Software like any compound object in nature decays over time and while you don't really see with the large amount of software the effect of the matter of the structure of the software decays over time because you do these things and you do that things and the original logic and our structure are deteriorating as function of time. Now this has an effect to which it becomes worse and worse over time. What you see here is a slide from Jim Meissner which on one at least shows cost of change in software system and on the other hand shows on the other dimensions shows the time since a project, a product was released. Now the global health heal, the optimal cost of change is for the people who are on the basis of the technical debt let's say spend 25% of the time on refactoring and refactoring and refactoring here today. However, teams with the technical debt happen to find themselves on the actual cost of change health which is a highlighting experiment and what happens if you are not good at pushing hard back on it at earlier stages when you find yourself at this point in time on the right side a few things are very painful. A, your ability to respond to your customers goes down the drain because the code is decaying to that extent now that whenever I do a fix I interject 0.3 new bugs or 0.5 new bugs etc. So the work of fixing the software creates its own problems. Second, you don't really have an option an option once you are on the right hand side of the slide. If you do things as is then they just continue to deteriorate and become worse because the exponent continues to grow. If you say something like oh I think I'm tired of this code and of this theme I want to acquire a new company with beautiful software and run for me with each one now on the back of the matter is that even under the very basic circumstances for a couple of years you will have two code bases and you will have all the problems of migration from the code base to the new one on your hands. So this is not as attractive even so I think it's shiny it's not as attractive as it really sounds. And last but not least when you are here you can say okay okay I get it I should have pushed back on the technical level for a moment and I improve it now. Whatever it is however now you are staring at a mountain of technical level and there is a huge difference between being able to push me on an ongoing basis versus needing to take the whole mountain under the new economic level and with the simulations. As a matter of fact the situation is so bad that for lagging projects and lagging you can think like I'm speaking about projects who do not push hard on technical debt they lose their value within three to five years. Now things about products in your company or in your company's portfolio I am sure that everyone here has a whole bunch of products that are older than three years five years more than I do. So what happens is by not pushing back on the technical debt you are creating toxic assets assets whose value is already zero because of the technical debt and furthermore assets are toxic because other components in your software's portfolio depend on those and those are very, very big. So the phenomenon is a very, very painful phenomenon and as John and myself today exemplify are more prevalent than perhaps you want to realize but before we get into that the natural question to us is okay how do we go about measuring technical debt namely at this point in time we go back to our own offices how do we know what is the amount of technical debt and for that I will hand it over to the capabilities of John. The slides are fine. It's a simple product. The first job that I have is the handheld microphone and there is no sound from it. But if you have some going on it is a first distributed database and there is confirmation from Google and you can see the one in Yahoo's distributed database system it's developed and established and it actually is a top level product. It's got 25,000 points of code it's interactive development as independent. It's fairly accurate and well done. The good question in front of this database is where are we today in terms of who to do is to run a pool analysis on that database on their accounts for the code and for the answer to the question yes. Comment, it's actually very quiet. Is it still quiet? Try to bet on it. Okay, I will talk like this. So where are we today in terms of this code? And before I show you the slides I want to say that this is going to be pictured on the tool. Tomorrow at lunch time I'm going to be demoing the tool with the system and a couple of other code bases as well. How I did this was by printing the anti-build files and introducing some native build files into the project. So that's how I did what I did with the tools. Let's jump into some parts of the dashboard. At the top we see the fragment of the dashboard that shows the big picture of the project. This is the roll-up of the roll-ups all together. This is the technical bet as calculated. We see that it is 251 days of effort calculated against this project. We can also see the lion's share of this pie chart is in coverage and then the design also has a significant part of that pie chart. We can see that the number of the dollar figure is $125,000. And that's really just a I think the tool defaults to $500 of a person day times 251 days. That's a big number because that number when you first look at that project it can give you a particular shot. It is real as a story but don't shoot the messenger of doctors that keep relating to me for understanding this. A metaphor I suffer from atlantic and from time to time they send me to take exonies and when I come back to the doctor sometimes the news is not too good and I have my pain etc. There is no reason to shoot the doctor because he or she is interpreting the ex-wife and there is no reason to shoot myself because arthritis is somewhere in the genes and you know being tough on my own genes is not the best strategy. So my overall philosophy about measuring the technical that it's of the same nature and this is a phenomenon that you need a monitor to respond to any possible event but this is not something over which you get to med and sustain or you get to med it yourself and getting med in the cold you probably is not an effective strategy. So some of the other characteristics that are shown out of this there is a percentage of there, 19.5% that is a ratio of death in this project all the things that could have gone wrong represents 100% and that 19% represents how many things code violations, undocumented, EPIs, non-covered code filling unit tests that represents how many things did actually happen that raised a red flag in terms of the tool. Other details about the code base 350 classes 25,000 plus lines of code and so on. A few more details this is some of the more static analysis that this tool shows this is the rules compliance 74.6% rules compliance these are rules defined in terms of groups of coding rules efficiency, reliability not going to go full of details especially given our time we can see that this project is showing rule violations it does have a set of critical violations 34 of them both of those are fairly low numbers for top level high-end rule violations for a project this is all for the dashboard of the system now given that you just looked at where are we today with this code base something else we can do is we can look at where we come from and how we are passed and how are we trending and one of the things this is a timeline that I put together by instrumenting about the last year's worth of public releases top end releases from 0.2 3, 4, 5, 5.4 and so on to the latest 0.6.2 we can see I put three different things code coverage technical debt the code coverage has been rising from 0 upwards the complexity for class and technical debt they dropped off considerably where the technical debt bottled out from 0.4 release with data on 9.109 that release measures having the lowest technical debt and that technical debt has gone upward since then and something that Israel's going to talk about in the next section is the governance and risk management associated with knowing these types of things and that's something that we'll get to next here though is just to demonstrate that you can use this instrumentation over time and I didn't quite get universal control repository and instrumenting these versions given that we know these things and we have these x-rays what are we going to do about it we'll focus on for the next work that we're looking at this is a this is two things from the dashboard component model and the heat map of the system on the right hand side the diagram is a heat map this heat map is a ratio of lines to code to technical debt ratio so the bigger the block the more lines of code and the colors actually yeah that sickly green yellow color represents higher technical debt a nice vivid green would be lower technical debt actually that's the hub rover in one of these components to show it the Pachy Kisabra Avro is a remote communication protocol this one package of Java code this component has a 29% debt ratio and we can see also that that shows a component model in this case sort of like complexity is the highest complexity component in the Kisabra system and that complexity and also its unit test coverage both gave it the highest ratio of technical debt or one of the highest ratios so we can see that this heat map of this component list gives us a place to focus on it gives us some insights on where we are to look and it's not a sure thing that that's the right place for us to focus on next to the project but this provides the potential data that we can use to help make those decisions with that just to quick it up tomorrow at lunch we'll be demonstrating that tool and with that we'll hand that all back to Israel same good job I would just mention that this sheet is out on the tables so you are welcome to read some extra details that you don't have the time to address in the presentation so perhaps most important thing about technical debt is the ability to use it for governance and I would actually use the word enlightened governance namely we are not really talking here about command and control we are talking here about our governance which is responsive to indicators which are in the course and the thinking about it is follows what you see on the left hand side is a traditional island by Arngel with costs, cost and schedule as a three parameters that we focus on and this suffers from two major defects at least in my humble opinion very difficult to put focus on what you see on the right hand side is a triangle that clips all the sleep on the previous one namely schedule, cost and cost into the category of constraints on the right hand side of the triangle has a value at the top of the triangle and quality at the bottom of the triangle and the premise is that one can very very effectively run with his or her sort of project by way of focusing on just those sleep and the way that this is done is follows if you take a look at the right hand side cost is obviously done in dollars or number of days of it is spent and you convert this into dollars this traditionally is done through dollars the view in this example is expressed through the net present value of the revenue streams that you expect to get under the product and this data is usually readily available from the business case quality or let alone we express through the technical data that we have spoken about so far so for example you may have a project where you expect a value of 100 million dollars you invested in it 10 million dollars and you currently conducting is 500 million dollars and the great power of those three simple in unison is that you can compare one with another you can have many other frameworks that say for value and they are all very well however when you use those three you can make direct comparisons and you can actually open up the dialogue for one technical people to be part of the game to be as always technical people and let me show you a few examples the first example deal is very important dialogue and I don't even think that we need to know who is the product owner and who is the project lead or the scamster etc but what you basically see in the project is the distribution of the technical data at a certain point in time and obviously in this example the lack of unit test is the number one they think at least in terms of something now out of the dialogue deal they reach a conclusion that they will bring up test coverage to 70% of the current deficit and they will then shift and do the rest of the investments now you may or may not agree with this strategy you may say oh we should start duplication I don't have any problems with that but what I would like to get a close view is here you have a simple way by which you business people and you very very technical people can understand the magnitude and the nature of the problem and what to do about it because if you are going to go they will have you know it's lack of cohesion among methods probably you will not get too much dialogue under this kind of statement when you the unit test around complexity for example I don't really know who the person is going to speak he or she will not understand a half a million dollar complexity for whatever the figure is is something to pay attention and then you are able to then reach and open up the dialogue to other constituencies you have a wonderful opportunity to work with technical debt one bit at a time instead of at the end when you need to stay at the mountain and here is a story what you see in the picture in the chapter is the process control view of this process you have the inputs or the requirements on the left hand side you have the output on the right hand side in the form of the common data and you have the control unit in terms of the scrum meeting at 8.15 in the morning of the night now what happens you can tie the technical debt criteria into the bill and just like you throw the bill when it does not contain you throw the bill when it fails to satisfy some technical debt criteria so for example one of my clients whenever the technical debt goes higher from the previous reading to the next reading they stop the line and everyone goes to work on it and the philosophy around it is that when things are fresh on other lines when we can really see checking in this piece or that piece etc the amount of effort that is necessary to eliminate or reduce the technical debt is much smaller than the amount of effort to put in in 3 months to 6 months or 9 months down the road and you can put in with all kinds of parameters of technical debt as part of the process and actually if you are can you see integrating in a very aggressive manner you can expect that you will have big sales a few times a day and you will like the process much more now by the big sales rather than by the scum processes so once you submit the new code you can do the technical debt reading as part of your big process once you do it as part of your big process you can easily use it as a criteria to decide to be faster in sales etc in addressing one bill but now this could be this I don't know the correct mechanics but this way of addressing technical debt namely when entering close higher I stop the line maybe too much of a good thing for example if my technical debt is a million dollar and the new leading dollar and one does it really make enough of a difference to stop the line the beauty of it is that you can take statistical versus control methods and apply them to the technical debt readings and what you see here is what the statisticians call control chart namely you have the mean of the reading are in the middle you have a lot of online two standard deviations about it and you have another line three standard deviations about it and you establish your economic policy in accord to those readings for example the economic policy could be if the technical debt is higher then two standard deviations then you stop the line and go to the company so you have the flexibility not just to take a one million dollar and one bigger than one million dollars you have the flexibility to play with standard deviations in accord with the quality of the process that you have at any point in time and people have actually experimented with the values additional rules that I don't have the time to get into but for example if you have five or seven consecutive readings which are between one standard deviation and two standard deviations this could be a reason to stop the line and start working on the technical debt so you don't need to do it mechanistically you can do it under statistical process based on the readings of the technical debt now John was kind enough to mention the business or the risk because you know the unass of the projects are filled with spectacular failure of so many projects and I would compare that the major reason for that is that there was no way to understand the risk at the right point in time and you know at the point in time that we could still that the past of this is actually a wonderful way to tell you about your risks for example if you had one million lines of code and you got a technical debt of one million dollars then your debt density is one dollar per line of code maybe this is not great but you could certainly leave it on the other hand if you have a million dollars in debt on one million lines of code the debt density is ten dollars per line of code so this might make you think a little bit about what is really happening here because for those environments the amount of technical debt is higher than the cost that you would be producing this code and this is equivalent to instead of building equity taking more and more debt so you can look at the technical debt with respect to code as indicative of the risks that you are facing that you are facing other more technical debt has a wonderful predictive capability which is demonstrated in the upper chart here what you see here is a study of 12 Java projects where one axis was the promise of a file to be able to own and the other axis was a complexity of file and what you see is that when it reaches 74, complexity of 74 you are pretty much guaranteed to have an error code class on your head on the other hand it increases the level of cell behavior then your probability is only one more insight now this is a sense of economic value because you know usually on models follow a 559 of the rule mainly 5% of the models create 50% of the vaccines and things now when you have this kind of correlation metric in the face of development you don't need to wait until you get the chemicals from the field and then start thinking about working on the error code classes when they are still in the map so it's not only a good risk measure at the point of time it's a risk measure with a productive ability one last example and I will just touch on it very very quickly because it's similar to the hit map that Jordan has already spoken about what you see here is a hit map for a bunch of models and size represents complexity and total represents quality of coding so for example, hit to hit is not complex in P5 because it's bigger in size than P5 however the quality of the coding in P5 is quite heavy when P2 has a much better quality of coding however we go with the P5 it's just characteristic instead of going up to P2 or maybe you go up to P1 so you have as an argument as to what you need to do in terms of selectively addressing you technique of that instead of looking at unbelievable amount and saying oh my gosh what am I going to do about it and this brings us to what I consider in some ways the most interesting model build technique of that which is it's rooted in agile philosophy and principles but it applies to many projects whatsoever so if you are in continuous integration in agile among the rest of the good stars and you are doing 12 to 9 a day it probably applies to you however if you are doing an old-fashioned model for any other method in your environment the effect of the method will continue to look precise with the same thing because course value and technical value have nothing whatsoever to do with the software method underneath and you can go into your wonderful code at any point in time that you desire and run the technical method so if you take a look at the project here the project shows three elements the cost, the technical value and the net present value at the start of the project we have a certain net present value from the business case we start working on the project and we have good cost in a linear manner another week past and another week past etc at time t1 we take the measure and we find that actually we include a lot of technical data at this point in time two things happen hey, you probably need to adjust your net present value because you are likely to be late or with res pitch ups and certainly your risk is higher so net present value has a singular point here going up the other thing that you do you probably store new function features and as long as you are within the zone v1 you are focusing on reducing the technical net present value at a certain point in time t2 the intersection of the cost line the technical net line the two intersect each other and while at this point in time not all everything is valid you are in a much better shape at the new point v1 and you can expect certain velocity to be picked up due to the fact that the technical net is low this is represented by stopping to increase the net present value maybe you know that you still have work to do but you are saying worst is the minus now, what is happening at time t3 when the technical net is pushed down the more accepted the better is that actually your velocity takes up possibly in a very significant manner and if you really push the technical net down real well with respect to net present value to be ultimately higher than that that you originally thought about because your velocity is probably better than the velocity that was stipulated in the original business so this is one way in which various projects evolve there are not many other patterns but in terms of being able to govern the software process you don't need to join these continuous integrations with any software method that you might be working this will do the job so what does it take away from this discussion we just actually oftentimes have an allergic reaction to the world governance and what I want to hopefully convince you that we are talking about enlightened governance here maybe we are talking about me as a developer as a person who writes a presentation getting data about what they do well what they don't do well and it can be specifically it can be specifically going around for me to go to the individual line of course and say oh my god I messed up here I need to fix it so we are hoping that about enlightened governance or in fact the user world following some GIFs of the GIF presentation includes a few governance it's not arbitrary it's what the code tells us about its quality so it's not an ugly world it's not a command control world it's about enlightened way to steal the code second as I mentioned it's rooted in a gile philosophy and principles but it's applicable to any software method your environment may improve any number of software methods you are able to do able to learn from the strong GIFs by way of technical depth across your work so even if you are planning to be 100% GIFs and you are in the inferior in which you are working or whether you plan just 50% GIF of the less other kind of methods you have a way to root to other worlds of the project portfolio in a quantitative manner in a meaningful manner across the board and last but not least the difference of the toxic code are if you look at the financial at the financial industry in 2008 it's obvious that for years they were sitting on a lot of toxic assets and it's just that the visibility was not there and we all find it in a very hard way in a very simple way something like that I am becoming unfortunately more and more convinced of the situation within the software industry particularly with respect to legacy code it's not any better than if there is a journalist code that I run into and I think you might be running into which in the lexicon behind here I will design a toxic code and what I am very very worried about is that we can let investors instead of having to share lightly and start taking steps to use it before we have a signal like this in the software industry I am sorry to have you run I had it a bit and thank you very much if there are any questions can people come in here so if there are any questions John and I will simply take it thank you