 Is it culturally OK to start early here? No? I'll be in trouble if I start early. I packed a lot in here, so I'm going to go ahead and get started. I think we have Vice President of Product Development for IHS. IHS is, we have an office here in Bangalore. We're a global company. Data and analytics is our area of expertise, particularly strong position in oil and gas, and then a lot of areas in the supply chain very close to oil and gas, automotive, technology, chemicals, defense, and aerospace. We are probably a company that you have certainly heard from, but probably have no idea who we are. We're the ones that are behind the scenes. We're reporting on forecast of oil and gas prices, a lot of the economic risk. If there's something in dealing with analytics and forecasting, most likely IHS has been involved. Today, we're going to talk about myth-busting software and estimation. How many people are familiar with myth-busters? It's the American TV program. The idea of these guys, a couple of scientists, they get together and they design really cool experiments, usually involving explosives and blowing things up in order to take a look at myths and figure out whether those myths are real or not. I wanted to do that. I asked Noresh, can I get a really big budget and blow things up? He said, sorry, it's not going to work. Instead of blowing things up, what I'm going to do is go and look at real data. As much as possible, go with real scientific data. There's a lot of postulation of different ideas out there. That's what creates many of these myths. I'm going to try to take a look at some real data, some data of my own data, other data from other researchers, to actually take a look at the various myths that we have. We tend to have a lot of them in software estimation. So as a good Agilis, what we're going to start with is test first. So when the myth busters get together, they look and they have three potential outcomes. Either it's busted, the myth is absolutely not true. It confirmed the myth, or they go in between and say it might be plausible. So I'm going to go through, we'll start with 10 myths, and then we'll go through them each one. So the first one, estimation challenges are well understood by general management, project management and teams, and it's normal to be able to estimate projects within 25% accuracy. How many people think that's absolutely true? Come on. How come when I ask my team, they say absolutely, it's going to be done. No problem, sir. Ah, okay. So now, how many people think it's going to be busted? Mostly. How many people think it's plausible? Maybe, okay. All right, keep moving on. I'm in trouble with it. What's going on? All right, estimation accuracy significantly improves as the project progresses. How many people think that's confirmed? All right, how many people think that that's busted? Okay, and how many people think it's plausible? Estimations are frequently impacted by biases, and these biases can be significant. Confirmed, busted. We're pretty good at estimating things relatively. How many think that's confirmed? How about busted? A couple of people, okay. Plausible. All right, okay. Velocity or throughput are some good tools for adjusting estimates. That's a good idea. Confirm, busted, plausible. All right, this one's good. We're a bit behind, but we'll make it up in testing since most of our uncertainty was in the features. I don't even need to ask. Who's going to be that hero developer that makes it up in testing? Okay. Seven, scope creep is a major source of estimation error. Oh yeah, sounds like that one's an easy one. Eight, having more estimators, even when they're not experts, can improve estimation accuracy. Busted, you think it's busted? What's that? How many confirmed? Project success is determined by on-time delivery. Success is on-time delivery? Nope. Busted? Busted, mostly busted. And number 10, estimation is waste. Confirmed? All right. We're going to have some fun. Okay, first one, estimation challenges are well understood by general management, project management and teams. Well, what do you think about that one? Well, you've heard this before, but I got to do it again. And even if I've done this a hundred times, I still have fun with this one. So we have our little beginning of our story, the project kickoff. We have our poor little Dorothy saying, when will we get the requirements? All in good time. My little precious. All in good time. But I guess it doesn't matter anyway. Just give me your estimates by this afternoon. The team binds together. Not so fast. Not so fast. I have to give them after a little pause. Go away and come back tomorrow. Does anybody believe me? I can't believe this anymore. The developer hero. I mean, I come out alive, but I'm gone in there. It's just about got things under control when... Flaws has got mad as well, in hand. My, people come and go so quickly here. And finally, they get it into testing. I wouldn't hear of it. Why, my little party, just beginning. Unfortunately, this happens too often. So, why is software late? I came across, I've done a lot of various research in this area, and came across this paper by Ghanunchan a while ago, 1991, but I found it pretty interesting because he did something really interesting. He surveyed both general managers, the business side of the world, and the project manager, and had a number of different things that came up. And one of the things I'm gonna go through, they're not quite the same. And I'm gonna highlight a little bit about that, but before I do that, I'd like to give you a little story on the context of feedback. So one of, during the World War II, the British were trying to study, they were doing some advanced data analytics. I think Dave talked about analytics and all these different phases of analytics. They were trying to do analytics on airplanes. It was coming back from bombing episodes in Germany. And they started doing analytics, and they said, and they did all these wonderful things and they found some patterns of where the bullets were. And they said, wow, this is pretty cool. All we need to do is reinforce all those places where they're shooting our airplanes and our planes will be stronger. Everyone's in agreement. Yes, that's what we need to do. We need to reinforce all those places where they're shooting us. And a Hungarian mathematician looked at this and said, you've got this all wrong. You've gotta remember that these are the planes that have come back from the war. These are the ones that didn't get blown up. You've gotta reinforce the places where they didn't get hit because the ones that got hit were the ones that got down. The places that were the fuel and things like that. So it's the context of feedback. Understand the context of feedback. And this is the type of thing that we have to do. So in this case, what is our feedback telling us? We're asking a general manager. We're asking a project manager. Couple of places they agreed entirely, called those out in the green ones. They both said, yep, customer management changes were big issues. They also said unrealistic project plans. Yeah, probably pretty much unrealistic. We went really aggressive in here. A couple that looked a little bit different. General managers thought staffing, what wasn't really a problem? Yeah, I pretty much gave them all the people and all the money they needed. Project managers said, no, no, no, you don't remember when I kept telling you I needed more people, right? Overall complexity. Business side, yeah, no, no, we just asked for this simple little thing. Project manager actually had to deliver it. Much higher complexity from their perspective. The other side, insufficient. General manager said, our only problem is if we would have just done a better job planning, we would all of this would have been in great shape. Project managers said, no, planning wasn't our problem. We did as much planning, right type of planning up front, but things changed along the way. We had to deal with that, so more up front planning wouldn't have made a big difference at all. You've got this disconnect. This disconnect comes out all the time. It's a quote from Upton Sinclair. It's difficult to get a man to understand something when his salary depends on his not understanding it. So let's take a look at the space shuttle challenger. When the space shuttle blew up, they did a real detail analysis of this. And they went back and they looked at, what sort of probability of loss of life did we have from this program? Look at it, the engineers, they said that there was about a one in a hundred chance of loss of life. Management, one in a hundred thousand. Okay, now what really happened in the space shuttle program? 135 flights, two disasters, 14 deaths. What does it tell you? Engineers are optimists. Okay, and management is somewhere in outer space. So let's look at some real data. I was at a company, Landmark Graphics, and we had been collecting a lot of data. Not really as a program that was focused on anything, just we were collecting data because we were technical geeks and we like to collect data. So we were collecting some data and we had a lot of data. We were a product company and I'd been doing a little bit of research in this area and reading about on it. So I said, well, let me take a look at our data and see how it compares to the rest of the industry. Can I validate some of the things out there? So I actually started out with the idea to validate some things. Came across a couple of unintended side effects. So let me go through a little bit of what I found. First of all, if we want to look at the accuracy of our initial estimate. So plot the initial estimate on the bottom and on the y-axis is the actual what came out with. If we'd been perfect, we'd be all along that magenta line. Well, what do we see? We saw data all over the map, right? Most, almost all of it to the point where the actual ended up taking more than the original estimate but scattered in a fairly dispersed pattern. I looked at that, that's sort of interesting. Also it looks a little bit a lot like data that I'd seen in a book by Tom DiMarco. If I look at the data and cross-plot the data from Tom DiMarco, almost identical pattern. And this was totally different industries and probably 10 or 15 years earlier data. But that's a pattern that I was seeing and I started looking around and saw this a little bit more. This is almost similar data by Steve McConnell that was published in his book. Very, very similar types of patterns. When you actually look at this and then you try to figure out, well, what does this mean? It sort of boils down to this. About 10 to 20% of the projects is actually what's coming in at or under the initial estimate in what we're seeing. If I wanted to have a 50% confidence, so 50% of the project, I'd have to go out twice the estimate. I'd have to double the estimate in order to have a 50% confidence of making that of what is based on that initial. And if I want to have a 90% confidence, I've got to get out closer to 4x. So that's the nature of the pattern that we see of what is actually taking us relative to that early initial estimate. We see a couple of things. We see a very strong optimism bias because we're very optimistic. The other thing is we have no clue as to the range of the uncertainty that we're dealing with. I mean, this is, when I have this conversation with people, they have just really no clue about that. They're thinking, oh, plus or minus 25%. I've added 25% contingency. I'm all in great shape. No, you're not even close to great shape. You haven't even covered the 50% probability case. And this pattern of one to 4x, I've seen repeated and repeated in data throughout. There are some companies that get better and better at it and they can get it. The about the best I've seen is where they can get the range down to about 2x in that 90% size. Still much larger than most people expect. We heard Dave talk today about connection framework. This is a plot that I saw from a person, John Helm, who presented at Agile 2013. This explains a little bit of what's going on here and that is that we have these long tails. We have these log normal type distributions that we're facing. And particularly, this is an earlier conversation with Dave about this. And it's a little bit of a, I should bring Dave to tell this story too because the challenge here is that estimation itself is one of those things we don't tend to be very good at. And when we're not very good at it, the area of say the complicated, we might have software development activities which are more in the complicated domain, but if we haven't got the experts or the people who really know how to do it and really know how to do the estimation, we may be actually behaving more like the complex where we have these very long tails. And very long tails are the things that push us out into that 4x category. Some more data. This is some recent data from a guy in Langdeg Jorgensen from Sintep in Norway. I'll actually be using a lot of his data. He's one of the best researchers in this area. I really like this project because he put software development project out for bid online on a place called vworker.com. It's a virtual worker environment that bids on these small projects. They received 16 bids for this project and they pre-selected it down to six bids that were from vendors that have very high client satisfaction. So they're only making sure that they're gonna have people that had already started from a high client satisfaction. The piece I really like about this is all six bidders gave their estimates and actually did the project. So we have real data from at least six people that went and did this work. So what we find is that the data's all over the map. The highest estimate to the lowest estimate was a factor of eight to one. The fact that we could try to get it, this is an amazing range of how different we see between projects and estimation. If we look at the actual that it took them over the estimate, we see this power of four if we look at the range. It ranges, one team was actually, one group came in at 70% of their original estimate. Another, the worst case was 2.9. The overall range was about four to one. So same type of data that we're seeing elsewhere. And then if we look at the actual performance range, the worst team took 18 times that effort of the best. Huge ranges of uncertainty, huge ranges in performance. So the idea that we could estimate so accurately, given the fact that we've got this natural uncertainty, is just unrealistic. So are they understood by general management and it should be accurate? I think this one's pretty well-busted. So let's look at the next one. Estimation accuracy significantly improves as the project progresses. So how does estimation accuracy improve? We have, how many people are familiar with the cone of uncertainty? Not as many as I thought. Okay. So the cone of uncertainty was first postulated by a very famous researcher in computer science, early pioneer. And he published this. How many projects do you think he used to validate this cone? One? How many? Actually it's zero. He published it and said this is subjective. It was based on his idea of what it was, but he had no actual data when he published this. He republished it about 10 years later. And how many projects did he, and he validated it with a couple projects. How many did he validate it with? What? So he had seven projects that he ran at University of Southern California, a university program. They were all the same program, project. And he had one time estimate that he had for me. It actually, if you go really deeply into it, the methodology is quite flawed. He also had four or five bids, I think that he had for Department of Defense. So overall there were about 12 projects and 12 data estimation points. And from that it was declared that this is absolutely validated. Let's see. The other thing though I think that's interesting is just wanna point out that he did pull out this forex. That forex behavior, sort of around the concept of operation, he was subjectively thinking that was about what it was and it looks like our data is also telling us that it's about the same. So I had a whole bunch of data at Landmark and so I plotted this up. It's a little bit different because the timeline I used was not, in the cone of uncertainty it's these phases and in my case it's relative time. So I put everything on the same relative time basis. And so I get something that looks a little bit like a cone. It's half a cone maybe. What's that telling us? Well it's telling us we've got a very strong bias towards optimism. That's one of the things. And it tells us it does actually converge and the reality is that it's almost guaranteed to converge. Well it is guaranteed to converge if you're only looking at projects that ship. Because when you ship there's no uncertainty. Once you've shipped you have no uncertainty. But I looked at this and I said okay that tells me something but does it tell me anything really useful? Because what's the real business question we're asking? And the real business question is how much work do we have left to do when we ship? In the Wizard of Oz we've got this witch and she's gonna do something really nasty to us when the hourglass is empty. Am I looking at it? Do I care how much sand there is that's already passed? Not too much. I care about how much time is left. So I wanna look at what's my uncertainty of what I have left. Not what's the uncertainty in what I've already done because what I've already done has no uncertainty. So I plotted this differently. I said what's the remaining uncertainty and how does my profile change? And I no longer have a cone. In fact I pretty much have a pipe. It stays pretty much the same all the way through. That was what I found with my data. Now subsequently of course I've been working with a Dutch racer Chris Verhoff and Lawrence Evelyn. They have actually got a lot more data than I had and they found that there are several patterns some patterns where there does converge some. There are some patterns where it actually diverges and gets worse as you get further away. In any event you actually, if you go back to the original cone it will converge. But what I'm finding in general is that unless you're taking active things to change it the uncertainty that's remaining really doesn't change. You still have the same relative uncertainty. So let's look at why that might be. If I look at a set of stories each one of these stories has some uncertainty associated with it. And if I probably use things like the invest principle to keep them separate, when I do one story what's happened? I've removed the uncertainty in that story. What's happened to the remaining stories? Really nothing. All the uncertainty of the remaining stories is still there. And that just continues. So if I look at only the remaining part there's no reason for uncertainty to get less. It's really relative perspective the same amount. So if I'm one month away from the project from where I think I'm gonna be done there's no reason, and I've historically been a 4x type organization. My chance of success is somewhere between one to four months. And there's no reason to think that that's gonna be drastically reduced. So if we look at this from a perspective of accuracy significantly improving as the project progresses. If we go to the real business question we're asking of what's left I think this is pretty much busted. So number three, estimations are frequently impacted by biases and these biases can be significant. First go back to our data. There's a clear optimism bias here. We're only looking at the upper half of the cone. So yes, absolutely there's an optimism bias. Let's look at another bias. This is some more work by Mungley Orkinson. You think the exact same specification to four different groups. So group D is our control group and he gave them no guidance whatsoever. And they came back and said it's 160 hours of work to do this activity. Group A was told the guidance was 800. They thought this is, here's a project and by the way we think it's about 800 hours. Group B, this is the project and it's gonna be about 40 hours. And group C, same project, pretty easy we think it's about four hours. What sort of results do you think we got? What did we get from group A? They were told 800, what'd they get? A thousand. A thousand? Okay, any other guesses? No other guesses, huh? Okay, so 300, I thought that was a good one. Okay, so anyway, 300 from the 800 group. The 40 groups said no, they went up to 100. Group four said you're kidding. One that was told four, you're kidding. It's really 60. Lo and behold, that's still way below what would have been otherwise. Two aspects here are going on, I think. One is a concept called anchoring. I've heard it a couple of times at this conference. The idea of anchoring is if I start in with a number and use that then when I am told to do something I'm sort of got myself anchored in that perspective and so my results are influenced by that. So that's one part of it. I think the other part of it that happens is that because software is a very malleable thing the people that were given the guidance of 800 are probably gonna think well they want something more than what I see in the specification. Similarly, you can always take the simple way to do it or the more complicated way to do it and that's gonna influence things as well. But these types of biases can be very significant. They did a number of other studies just changing a little bit of words here and there and got vastly different results. So what looked like it was the same set of specifications ended up with quite different estimates. So what's that? Yeah, this was a group saying just you're estimating for this group so it was the same set of groups. They were estimating for the same group. There was estimators that were doing the work and it was a group that they were using so it was the same pool so they normalized based on that, yes. The only thing that was different really was the person doing the estimation itself. Yeah, in this case they were using an estimator rather than the actual people doing the work. So it was a professional estimator doing the work. So you could say that if it was actual development teams I think if you did this with actual development teams you would get very similar results. The behavior is going to be the types of things that are shown in the experiment I think you would replicate in other places. And that's because what gets us into trouble is not so much what we don't know. It's what we know for sure that just ain't so. So these biases can be very significant and they can impact your estimations quite a bit. So here's one of the agile things we say we're pretty good at estimating things relatively. There's an example of it. Let's look at this. So Juergensen also anchoring I just talked about a little bit this is one of the problems we have with relative estimation is the fact that whatever we start from is an anchor for what we're doing relative estimation on. Juergensen did some other work in this area and the problem is when you do it because of anchoring A relative to B is not symmetric with B relative to A. This is the challenge we have. An example, you did a number of examples someone's software, some with just general questions. You get the things like Austria's population is 70% of Hungary's when we're using Austria relative to Hungary while Hungary's population is 80% of Austria's with Hungary relative to Austria's. So both are less than each other. It's nonsensical, right? But that's the type of behavior that you get as a cause of the anchoring. The data that's often or the study that's often referenced to by Agilis as to relative estimation is an excellent piece of work by Eduardo Miranda, but it looks absolutely nothing like any of the relative estimation approaches that we use. This is actually a very, very complicated, pair-wise estimation process and it goes through a fair number of moderately complicated mathematics. It would be much more complicated than I think I would see in any Agile team because most Agile teams are taking approach towards doing it simpler and simpler, which is not necessarily a bad thing. So I'm not saying relative estimation is bad in any way. I do have my concerns about it. So let's look at this particular one. This is one I think is interesting because she's an example to point it at. We're pretty good at estimating things relatively and it says this is about twice as big. So the big one is about twice as big as the small one. How many people think that's about right? Big ones about twice as big as a small one. Look about right. Anyone else have suggestions? Not twice as big. Same size? Four times? Why do you think four times? Width and height? Okay. Close. What about depth? I'd say this is low by four X. That it's eight to one. Because the problem is, this is actually a problem with the mind. The mind is visually trained to work well in one dimension. We think, and actually it overemphasizes height over width. So if you get a glass and you have a very tall glass and most restaurants and things know this, if you've got a very tall glass, it looks like it holds a lot more than a very thick and wide glass. When in fact they often have the same. So the mind plays tricks on us with relative estimation. My view of this is, I'm not saying that relative estimation is bad because I think it's actually just about the same. If I were to look at that specifically, we're pretty good at estimating things relatively. No, I think we suck. We just suck in relatively different way. So yeah, it's busted. But now we're gonna come to the reason why we do relative estimation. Velocity throughput is a good tool for adjusting estimates. And this, I get my team, one of the things I find amazing about teams doing agile development is that how few of them can actually produce this, which is of course a release burn-up chart. People focus on so many different things, trying to get all sorts of things right, and they miss one of the most important things which is telling me, as a business person, I wanna know where am I at and when am I gonna be done? These are pretty simple questions that business people have. And what do we do? We spend all sorts of time tracking tasks. We do all sorts of things to make sure our iterations are meeting our deadlines. Does a business person really care about iterations? Not so much. I mean, I care about the business object that delivers value at the end. The fact that I've got an iteration, if I'm in continuous delivery, continuous deployment, yeah, then I wanna know how things are going through all the time. But if I'm in a situation where most projects are where there's a final delivery and then something's mobilized, I care about that mobilization object. And when that's mobilized, that's what matters. Giving this type of picture and showing the ability to look at how's the team's velocity projecting out and how is scope pre-projecting out, I can get a nice intersection and it'll give me a very, very good thing. Velocity's pretty good for that. And what it's really good at, predominantly, is removing bias. That's its primary function is to remove the optimism bias. Because it's not a silver bullet. It actually doesn't help us that much with uncertainty and why. Because our uncertainty is still there. So the uncertainty is still there. But what it's helped us with is to remove the bias and remove the bias and also to help us incorporate in scope creep. So is it a tool for us that are improving things? Yes, absolutely. So now we go on to, we're a bit behind but we'll make it up in testing since most of our uncertainty was in the features. Everyone seemed to think that was gonna happen. So this is an article by a paper by Langkau, Estimating Software Project Effort, An Empirical Study. And other than the, well the picture wasn't in there but it's, but I love this article for three reasons. There's three great reasons why I love this article. First, she references my work, which I think that's awesome. Second, she more or less comes with the same conclusions that I did that the estimation process ends up with a log normal distribution. But third, I learned something new because she had just slightly different data than I did. And that is that she had data for both features and bugs and was looking at the comparison and compared that data between features and bugs. And what she found, or actually what the data showed and when I looked at it with it, is that the uncertainty in the defect is about twice as great as the uncertainty of a feature. Why is that? Well with features we can actually pretty much get our hands around what a feature is. With a defect, we're trying to do, we don't know what we're looking for, right? So they may not be big problems but when it's a big problem it's really nasty. And the uncertainty is great. So what it means, a couple of things what this means, means if you're deferring your defects till the end, you're putting a lot of uncertainty at the end and we know that, right? Get your defects dealt with early, find them fast, remove them. This is actually probably more important of a feature of an issue than what's often reported as the cost of when you fix the bug. Because from an overall management perspective, you're trying to understand where's my uncertainty, can I pull that in? Can I get rid of that uncertainty early? So I think we know this one's busted. So scope creep is a major source of estimation error. Let's see how this plays out. We want this and that. We demand a share in that and most of that, some of this and all of that, less of that and more of this and plenty of this. The other thing, we want it now. I want it yesterday, I want more tomorrow and the demands will all be changed then. So stay awake. A little bit of scope creep. So what is the challenge of scope creep? Well, so Capers Jones has been recording a lot of data and there's some people who have a little bit of concern over the type of data he's collecting but in general I think it's a good perspective to start from. He says the average project, if you have nothing else you could look at is about 2% per month in scope creep. Which if you look at it projected out gives you almost, it's about 27% per year. And you know, I've looked at that and it's not too bad for what I see. So as a starting point, that's what I look at. What I find a lot of teams do when they're looking at a project, they say well this is about our story points that we have and this is our velocity that our team has been able to do. Our gross velocity. Now this is total points we're delivering. And they say well they take the existing story points and divide by the velocity and they say I'm gonna project out this. What they have totally ignored is the scope creep. They've not even included, if you've got to look at the importance of net velocity, net velocity is far more important than gross velocity. Okay, one of the key things. I see all too many teams that are doing this. And they've got included. And if nothing else, just assume that your net velocity is gonna be decreased by an increase in scope of 2% per month. I'll go through an example of this a little bit later. The impact of this 2% scope creep per month, the thing is it gets worse as the farther out your project is. So if you look at something, if you look at how much impact it has over, if we're out here in the 30 month project, we're gonna end up, if we didn't include it, we're gonna look at this 4x type of behavior. It's huge. Which is why this is a report from the Standish Group, also published by Craig Larmann, of Project Success versus Project Duration. Now in this case, success is defined by the chaos definition. Chaos definition is delivery on time. But if you look at this sort of over time, if your project is up to 36 months, your chance of success goes down to zero. Your chance of being on time based on that, based on the database that they have. Pretty awful, yeah? So that's one of the reasons we try to keep things short in an agile cycle, is keeping the shorter we can get it, the better chance we have controlling it. So scope creep is a source of estimation error. You know, that's pretty much confirmed. So number eight, having more estimators, even if they're not experts, improves estimation accuracy. Pull out my prop here. So this is an exercise I've done. And usually, if I have more time for this session, I would include it in this session. I'm gonna be doing a game tomorrow. So if you're here and wanna participate in some of the fun stuff we're gonna do there, we'll be doing an estimation exercise of jelly beans in a jar. Do this for a couple of reasons. One is, it's a good estimation exercise. It also shows the power of potentially groups getting involved in it. But one of the important things I find from doing it, that one of the unexpected consequences of it that when I started doing it, is I realized really how bad we are at estimating even something as simple as jelly beans. I've just anchored you with bad. So how bad do you think we are? How, in terms of estimation? Anyone wanna venture up? If I looked at sort of the range between the 90% and the 10%, the highs and the low, how much difference do you think it would be in the estimation, as a ratio between the high and the low? You wanna offer anything up? 70%? We got 70%, I'll go straight to it. Six to one. Six to one between the high and the low. When we're estimating jelly beans, and I've repeated this consistently, if you get a group of about 20, I've done it probably close to 10 times, groups anywhere from 20 to 100 people and just have them do this, I end up with a very consistent distribution. Tends to be log normal, they're very close to log normal. With the worst case, the 90% is about six to one, six times higher. So if I had an estimate of 40 on the low end, I'd be something like 240 on the high end. That's when individuals do the work. I then get groups together, it's the groups of about six or seven. When they do that, their ratio goes down to two to one. If I just take the average of the crowd and then I compare average of crowd versus another average of crowd, that range goes down to about one and a half. The plus or minus, plus or the plus, yeah. This includes, yeah, it includes not experts. I mean, there's no, yeah, I mean, anyone with an expert at jelly bean estimation. So it's not experts, so this is one of the things. You would think that this would be pretty easy. I mean, it would be a pretty easy thing. I always work with business people with this one as well. Business people do this and they come up with these numbers and they do the same thing, six to one or sometimes worse. I said, okay, so is this easier or harder than estimating software? Easier, right? And software, we were able to get four to one. So it's correct. So you would think this is easier. Now, the granted, I do this in five, I give them about five minutes so I don't give them a lot of time. Software, we do, software, we can get at least 10 minutes to do our estimates, right? Okay, so this is written up really well in The Wisdom of Crowds by James Cerro Ecke and that's sort of what the source of were. I started working the experiment. It was interesting to come across this book because I had read this other book called The Extraordinary Popular Delusions and the Madness of Crowds. So here we have one side which is The Wisdom of Crowds and another part, another guy which wrote this back in the 1700s, I think, on the madness of crowds. So who are crowds, are they crowds? Why is Eurocrats mad? And the reality is that they can be either way. So some examples here on The Wisdom of Crowd is Jelly Bean Estimation. The Who Wants to Be a Millionaire. The most accurate thing to do is the audience, ask the audience type of thing which gets a 91% correct answer. The examples on the madness of crowds are things like sometimes stock markets go mad. The Dutch tulip mania in 1637 where an individual tulip bulb was worth multiple millions of dollars. Just craziness. So crowds can go totally mad or they can be incredibly smart and the key part is diversity. If a crowd has diversity, it will tend towards being really smart. If a crowd does not have diversity, it can go off into groupthink and be totally crazy. So the idea is bringing in multiple opinions, making sure that those multiple opinions are diverse. This is an example I've used with some teams is ask the team and get a distribution from the team as to where they are. So ask them individually, anonymously and then plot this out. And this is just an incremental distribution. So in this case, we've got some groupings here. We've got some distribution that tells us something. It also tells us sort of a range of where it could be and it gives us also the possibility to have some conversations. I think in Jeff's talk yesterday, there's something about you can get stuff from data but you can't get the empathy. And actually in this case, you don't have to be a rocket science to get some really interesting information from this. You get some patterns here. So who do you think that was? That's all the developers. That's all the testers. And that's the test manager. Not on my watch. So what this worked out with this particular team is this was a great conversation because the developers had no clue as to why the testers were worried. And then they got together and actually having this, they had the framework to have a conversation and they worked pretty well and they came in pretty much at that inflection point. So in general, having more estimators, even if they're not experts, improving estimation action, that's pretty well confirmed. There are some, you have to have just enough knowledge but you don't have to be experts and getting more people involved is generally a good idea. So project success is determined by on-time delivery. So certainly that's how things like Standish Group says yeah, on-time delivery, that's the important thing and they give this like often reported challenged project that didn't meet their date. But why do we really care about on-time delivery? Is that what we really care about? One tool I use for this is something called the cost of delay. What I wanna do is I wanna have the adult conversation as to what it really means to be late. What's gonna happen? Is everything gonna go, is the world gonna fall apart if we don't deliver? Are we in a situation where the market forces are such? Maybe in the US around Christmas season, if I'm a game company and I don't deliver in the market window, I'm in real trouble. But in the other hand, I could be have a very good position with a product, not really strong competitive forces. And if I'm a little bit late, it's not gonna have a big impact. So it's really important to have that conversation with the team as to how does the market value fall off relative to the date? And if you have that, we have the storage of this idea in Agile sometimes comes up. Oh, the delivery date is fixed. Whatever scope we had is exactly what we're gonna deliver. Who's making that call? Is that a really, is that an intelligent business decision? It may or may not be. It's just sub-optimizing on one thing, saying delivery dates is the important thing. What's really important is we're trying to make, solve an essentially complex, uncertain differential equation as to what's more important when we deliver or satisfying customers. Sometimes you get your priorities wrong. This is the Ford Taurus. Back in the 1980s, Ford was a struggling automotive company. And the Ford Taurus, first version of the Ford Taurus came out. Project manager that I'm gonna make sure this is good, did all sorts of focus groups, all sorts of things to make sure quality was in great shape. Did all the things we think really important to get the product right. Car was a huge success, save Ford Motor Company. But it was six months late. What happened to the project manager? Fire demoted, recording history is not easy to tell, but anyway, not rewarded is the important part. So then go on with round two of the Ford Taurus, new project manager. What do you think he did? Delivered on time, right? Cut all sorts of corners, no focus group. Don't need those, it's gonna hit our schedule. Quality, act, quality is good enough. We're gonna hit the schedule. Now, what do you think happened from market perspective? Not very good, right? So you gotta make sure your priorities are right. Look at this, this is one thing I look at. A poker metric. You think, if I were to say I'm gonna have a metric called percentage of hands one. You think that's a good metric? Percentage of hands one? Good, not good. But it turns out it's actually a pretty good metric for determining who's the loser. Because what do you have to do to win hands in poker? You have to stay in. The really good poker players do not play many hands. They get out early. And in order to win, you have to stay in. So if I were to just arbitrarily say that's the important metric, I'd be in trouble. Where we say on time percentage is an important one. This is some data I included in my paper. The really ugly red project there is Microsoft Windows Word, WinWord 1.0. You wanna remember WinWord? Yeah. You wanna remember what the old enough to remember the word processor market back when WinWord came out? It was a while ago maybe. WordPerfect was a dominant player in the marketplace. WinWord was a brand new product that's gonna come out on a brand new operating system Microsoft Windows, but it was horrendously late. It was one of those, we have to be out in one year projects that ended up taking four years. But if you look at it, WinWord took over market share very rapidly. So it was late, but from a value delivery perspective, enormous. So I like to work more towards value metrics than cost metrics. Douglas Hubbard says that our industry is totally messed up in terms of what we look at. We tend to look at the most measured tends to be the lowest information value such as cost and time. But what's really important is things that value delivery and we just tend to not measure it. And part of the problem we don't measure it is because it's a lagging thing. It's hard to measure and it tends to lag. So instead we measure the thing that's easy, but useless. So if we look at outputs and outcomes, Jeff talked about this yesterday. If we're bad and we're late and we produce a bad product, yeah, we're pretty much on thin ice. We're a rock star if we've got on time and a great outcome. If we've got this good outcome like the Ford Taurus and we're late, I can't predict anything because politics is more difficult than physics. But if we're on time and produce something bad, we call that crap on time. And you joke, there was a time when our customers were saying, oh, you've got crap on time. It wasn't the thing you want your customers saying. So projects is thus determined by on time delivery. Yeah, not so much. So estimation is waste. You gotta go back to the real business questions. Is it worth doing? What's the priority? What's the target time to ship? What's the critical scope? Do we have the right investment? What's the cost of delay? If your estimation is answering those questions and you're doing just enough to get the answers to those questions, then your estimation is potentially, that's what you should, that's pretty good. If you're doing estimation that's well beyond that, it most likely is waste. So as estimation waste, it's plausible. And I think I like a lot of what I'm seeing from the no estimates camp. I'm still curious as to how it all works. My perspective is you do just enough estimation in order to do things to answer those questions, particularly to do prioritization. People from the no estimates camp say, well we don't do estimation, but we do prioritization. Well, if you're doing prioritization, you are doing estimation in some form or another. So prioritization and estimation I see as just two sides of the same point. So now what? So I think we're out of time, which is not a surprise. I have a little bit more that I go through. I'll be going through it tomorrow at the game. So I leave you in suspense now as to, I scared the hell out of you because we have to do all this estimation. And I have a simple answer to what to do and what I've used that works really well. We're actually gonna live it in the simulation game tomorrow. So if anyone's still here tomorrow and wants to be involved in that, that'd be good. But I think I am supposed to be done now, right? All right, thanks. And I'm happy to talk any questions.