 Everybody still have energy at the end of the day? We're still here attending? I'm sorry, what? So all right, I want to start this with what is going to seem like possibly an unrelated question. How far do you live from an airport, the closest airport? Think about what you were going to say about that. You could say it out loud or not, or just think about it. I, when this question comes up, I have no problem saying that I live 20 minutes from an airport. That's basically factual as far as I'm concerned. I will drop that into conversation. But of course, that's not really true. That's dependent on a series of assumptions. It assumes, for example, that I'm in a car walking to the airport. In my case, O'Hare would take a lot longer. It assumes the weather is good, which in Chicago is dicey for about half the year. It assumes that it's not rush hour, which means that's dicey for another 10% of the day. Nothing untoward happens like blowing up a tire. On my way here yesterday, I learned another assumption, which is the assumption that the cab driver takes the optimal route to the airport. If the cab driver messes around in the back roads of my suburb, it takes a couple minutes longer. Despite that, despite the fact that all of these factors would conspire to make my trip longer, I still say that my trip to the airport takes 20 minutes, even though there's no way it could possibly take less than 20 minutes unless I teleported. And all kinds of ways for it to take longer. In other words, and I still find this 20 minute estimate, for lack of a better term, to be so valuable that I use it, and I don't even know the actual distance that I live from the airport in miles. I only know it in minutes, even though it's problematic. So, my name is Noel Rappin. This talk is 20, 30 minutes. I realized I had more time in the slot, so it's 30 minutes, give or take 10. It's going to be very embarrassing when I go over at the end. So, let's get that out of the way right now. I work for TableXI, which is a consulting firm in Chicago. If you need consultants in Chicago or you want to be a consultant in Chicago, come find me at some point over the next couple of days. Or if you just want a nice green circle sticker with an XI on it, I have a bunch of those as well. You can also follow me on Twitter at Noel Rapp if you wanted to say something about this talk or just say hi in general. My qualifications for talking about estimates, well, once upon a time I gave an entire full day workshop that ended exactly at four o'clock on the dot as scheduled, as evidenced by this one tweet from one guy sometime. And since I actually got an estimate right once, I think I'm very qualified to talk about estimates. I'm also, this is also slightly idiosyncratic about this. You will probably disagree with some of this. Many people do. People are somehow successful even if they do estimates a different way than I do. I don't know how that works, but apparently they are. And that's fine. This is a way that works for me and it's a method that works for me and you may do something different or have different opinions and that's fine. So one thing that you hear about estimates is that developers are bad at them. I think that's something that almost everybody in this room has probably heard, that estimates suck, developers are bad at them, we can't do it right. And sometimes the question is like, are software developers uniquely bad at estimates? Is this, sometimes you hear it phrased that we are so bad that we have problems that no other industry has because of our estimates. And the only thing I can say to something like that is that people who say something like that have obviously never been involved in any kind of home remodeling or construction project or involving any kind of home contractor because that is an entire industry that is dedicated to time overruns and cost overruns and has an entire vocabulary dedicated basically to scope creep. And it has a lot of similarities to software in that a lot of times you don't really understand the problem until you try to solve it. You don't really know what's behind the wall until you tear the wall down. And it has very similar pressures in terms of a pressure to deliver an initial estimate that's low for obvious reasons, to make a sale or for other reasons. Or I can point you to this. This is Boston's big dig. Did people here live in Boston, near Boston? I lived in Boston for a pretty good chunk of this. This was the most ambitious public works project in America in the 2000s. It was an attempt, a successful attempt to replace the North-South Overpass Highway through Boston with a series of North-South tunnels. It occurred to me as I was putting this together that in reference to the airport analogy it also involved an exciting game of how will we get to the airport this week for about two years as the access roads got messed up. The big dig was originally estimated at $5.8 billion, which is a lot of money. It came in a tiny bit over that at a mere $21.93 and counting because they're still fixing a couple things that weren't installed correctly the first time. So estimation is not just our problem. There are other industries and there are other kinds of engineering that have similar issues when trying to come up with a time or a money estimate for the work that they do. But software, we have a slightly more unique solution because it's at least in theory easier to change software estimates and move software around than it is to move walls or giant tunnels underneath Boston or things like that. So potentially there's something that we can do about it. Some people, sometimes you will hear now the suggestion of people who think about agile projects that we don't estimate, that software estimates are so bad that they're actually a net negative to project success or project happiness. And so you can find these people, sometimes the no estimates is sometimes a hashtag or a sort of rallying cry on blog posts or whatever. And they suggest that we don't estimate and that we just do stuff and get paid. The ideal world here is that you continually are in a very tight feedback cycle where you take the highest priority task remaining, you do it until you're done or the client runs out of money or gets mad or some combination of all of those things. So you can just do stuff and get paid. And I can imagine this working in maybe a startup environment, a product environment where you don't really have a deadline or customers or clients and there is enough money to cover all of this but I'm a consultant working with clients with tight budgets and it feels somewhat similar to me saying like we're just gonna tear up your kitchen and you just keep writing me $15,000 checks and eventually you'll probably get your kitchen back. It is not effective, it's not useful to our customers. Customers I think reasonably expect to have some kind of idea what they're getting into before they start a project and it is on us to give them some sort of good faith plan to do that. You sometimes hear that projects don't fail for technical reasons, projects fail in part for communication reasons and I think estimates are a very important piece of communication between the team that's doing the work and the team that is paying for the work, managing the work, covering the work but you have to be careful with it because it's communication that has dollar signs attached. Anything that gets a number attached suddenly acquires a certain weight and a certain gravity in discussion and it's hard to move it and especially once you put a dollar sign on something no matter how crazy or half fast your process was to get to that dollar sign, once you put a dollar sign on it in the mind of the person that you were talking to that is a real thing and you need to take that into account in future discussions. So if estimates are communication what are they communicating? Well you're communicating I think at the most basic level you're communicating the investment. In this case not just the monetary investment you're communicating all the things that the people need that need to be provided to the development team in order to get the project done. So that is money, it's time, it's potentially a certain amount of requirements analysis, some kind of data format, some sort of business plan, all of those things can be part of the estimation process. The estimate also enables you to communicate progress. It's kind of hard to say you're 50% done if you don't know what done is. Estimates help you define what is in scope and what is out of scope for the purposes of being done. So that's the other thing that estimates communicate is scope. So these are all things that need to be done as part of a good project and talking about the estimate and planning it is a really good framework for communicating and getting that shared understanding up front. But a bad estimate does have consequences. We've all, many of us have probably been on projects where there was an unrealistic estimate up front and that creates an unrealistic expectation and it creates a lot of bad pressures on the development team and a bad cycle that leads to poor work and slipping deadlines even further. And bad estimates also, or badly used estimates also allow for a certain kind of micromanaging where it becomes critically important that you said that this reporting feature was gonna be done on Tuesday, it's Thursday, and where is it? And that can allow that kind of very, very fine grain managing whether it comes from a client or from your actual manager or something like that can also be really counterproductive in the course of a project. So one thing, one then, so if bad estimates do that really I think we should just do good estimates. The easiest way that I know of this to do good estimates is to wait until the end of the project and then produce your estimate. This is an underrated technique that I've seen actually used at large companies when they produce the estimate at the end to cover and retroactively cover their budgets. However, most of the time you actually need to produce an estimate at the beginning of the project and estimating at the beginning of the project is basically the worst. You don't know anything. Gary was talking about known unknowns and unknown unknowns. The beginning of a project you have the most unknowns of all kinds, known and unknown. This is something that only incrementally gets better but unfortunately clients insist on knowing an estimate at the beginning. This is a little bit like trying to figure out how much time it's gonna go to the airport. It's gonna take you to get to the airport if you don't even know where the airport is or haven't even decided if you're going to need to fly yet. What can make an estimate at the beginning of a project effective even when we know nothing about the project or very little about the details of it? So I'm gonna talk about a couple of different processes by which you can create an estimate. First of all, an estimate that sort of works over the running life of a project and then how to modify that to create an estimate that might work at the beginning of a project. And it's gonna have a reasonably good chance of success. It's worked for me in the past. But first I need to make sure that we're all clear on a couple of agile project software terms because I'm gonna use them and I wanna make sure we all have a common definition. So we're gonna talk about a feature or a story which is the unit of work in a project. Typically they're relatively small on the order of a couple of days or a week or so, ideally less. The main thing about them is that they are, at least in theory, independent of each other and independently verifiable by the customer or the manager as to when they're done. In iteration is the unit of planning in an agile project. It's typically one or two weeks. And ideally the way that this works is you think about how, at the beginning of each iteration, the team gets together, they talk about what stories they're going to complete over this iteration and that's the cadence of which planning gets done in an iteration project. Point is the size of a story and we'll talk more about exactly what points are in a second. Hopefully that'll just come back. And then velocity, which is the most important number, which is the number of points that you can do in an iteration. And the way that you use that is to be able to say, I have done, we've done 10 points in this iteration, we can do 10 points in the next iteration, we can do 10 points in the iteration. After that, we have about 100 points left in the project so we have about 10 iterations to go. That's, those are the kinds of sentences that you say. So here's a theory about how to do this. There are a couple of things that work or that I have worked for me. The first thing to think of when you start thinking about whether, how to create an estimate is that there's two different scales here. We're talking about estimating the entire project, like this will be able to go live on December 15th, versus estimating individual features. I can get this reporting feature done by Tuesday. And the first thing to realize in creating a successful estimate over the life of a project is that nobody really cares about the fine-grained feature estimates. There's a whole lot of effort put into them. And yet at the same time, really, it's not important over the lifecycle of a project. Whether an individual feature is done on Tuesday or Thursday, or it takes a little bit less or a little bit longer. Over time, over the lifecycle of a project, that is not the important part. The important part is when you can actually go live and when people use stuff. The important thing is the aggregate, not the individual. The individual estimates are only useful because they are the building blocks that you use to get to the aggregate estimate. And estimates are estimates. They are imprecise by nature. If they were exact, we would call them measurements. But they are actually probabilistic. If I look at a weather report and it says that the chance of rain is 50% and it rains, was that forecast accurate? It's actually, you actually can't tell from one piece of data. So you actually need like, say 20 pieces of data. It actually may not even, that might not even be enough. But if it said there's a 50% chance of rain 20 times and it rained 11 of them, you would say that that process is reasonable. It's producing reasonable data. If I say there's a 50% chance of rain 20 times and it rains four times, then there's something wrong in my process. I'm overestimating the chance of rain. And as a hint, weather forecasters overestimate the chance of rain because if they say it's going to rain and it doesn't rain, nobody gets mad at them. But if they say that it isn't going to rain and it does rain, people get really upset. So that's, in some ways, it's actually very similar pressure to what we have on software estimates. So similarly, if I say this is a one point story and it takes me six hours, that doesn't really say anything about the process by which I call this a one point story or about one point stories in the future. What I need to do is have a bunch of stories that I call one point stories and see how long they take in the aggregate. So this process only works in the aggregate. And again, I think one place where this kind of, where people make, where people go astray in this is starting to think that individual point estimates, individual feature estimates have more validity than they actually do when in fact they only have validity and aggregate. So when you look at this in aggregate, you see that the kinds of things that can make this kind of estimate go wrong are sort of larger scale things. Like I can talk about having made a bad assumption. Like I may have assumed that this reporting feature only needed to be delivered on the web, but they also need CSV and PDF files. That's an incorrect assumption on my part. I might have misunderstood the problem. You have scope creep where once they see, once the user sees it, then they're like, oh, the columns are in the wrong order. I want to be able to alphabetize stuff. I need to be able to drag and drop columns or something like that. Those obviously, those are outside initial estimates off of the time, a lot of the time. There's unknown complexity. I need this to integrate with PayPal. How hard can that be? That kind of thing. And then there's rework, which developers I think overlook a lot, which is the very sort of the back and forth that goes along when, that's not quite scope creep because it's more like this needs to be blue or it needs to be alphabetized. It's not really adding scope. It's just coming to a shared understanding of what needs to be done. And that's the kind of thing that we often don't consider when we're coming up with a time estimate as developers because as developers, we tend to be optimistic both about our own abilities to solve a problem and about the amount of support time that it takes to get something to develop, to production. As developers, we tend to overlook project management time, testing time, deployment time, that kind of stuff when we're creating individual estimates. And of course, sometimes there's pressure to create low estimates and that happens in a lot of different ways. It can be a manager saying this has to be done by the 15th. I don't care. It has to be done by the 15th or else you're all fired. When's it gonna be done? The 15th. Or it can be like, I know this is gonna be a long conversation if I say that this feature is gonna take three days because the client's not gonna understand that. It's much easier for me to just say it's gonna take a day and deal with it on the other side. Or it can be something like, I know that this is a $400,000 project but I'm not gonna be able to sell it at a $400,000 project so I'm going to say it's a $250,000 project. None of those things are exactly like the kind of shining lights of our profession but they're all things that happen. But the problem with that is that giving something a lower estimate doesn't actually make the problem less complex. It doesn't make it more likely to be finished by the 15th. It doesn't actually make it a $250,000 project and it doesn't actually make it a one day feature. These are all, you're just setting yourself up for much more difficult conversations later on which is, I don't think it's a really great way to continue to run projects. So to do better in terms of estimates you embrace the kind of uncertainty and don't chase false precision. Adding decimal points to estimates don't make them more accurate. They just mean that you know how to use the divide key on your calculator. It doesn't really tell us anything about the problem. And it's also important to focus on the parts of the problem that people actually are pretty good at estimating. So if you think about what, if I take an individual feature, like for example a reporting feature, I need to create a bunch of administrative reports and I think how much calendar time that's gonna take, very broadly speaking there are three factors that go into that. The actual complexity of the task, how good the developer or the team is at producing the code and how much time in that period they're gonna actually be able to spend on the task. Those three factors together more or less combined it to be all the work. So let's look at those a little bit. We're actually really bad at estimating how much time we're gonna spend on a task. We tend to be very optimistic about it. If you've actually measured how much time you spend with your code editor open as opposed to something else open, you probably found that that was an appallingly low number if you're me and got so frightened by doing that once that you swore never to do it again. But it tends to be very consistent over time. Like my meeting schedule is basically the same from week to week. I spend about the same amount of time at lunch from week to week. I spend about the same amount of time on Twitter from week to week. Over time this is a consistent number more or less. The skill of the developer, the second factor. We're really, really bad at thinking about how much time an individual developer is gonna take on an individual feature. And that's fine because a project estimate is a horrible time for your team to be arguing over who's gonna take more or less time on individual features. It's very, very bad for team morale to walk around saying, like, Noel, I'm sure you could get this done by Thursday, but if I gave it to Bob, he would have it done by Tuesday. That is not a good, any metric that's not like when the whole team can deliver this project tends to be very destructive over time. But if your team stays the same from week to week, then your team's skill tends to be consistent from week to week. Ideally, they're actually getting a little bit better from week to week because they're learning stuff, but for the purposes of this, it's roughly consistent from week to week. Which gets us to complexity, which developers are actually pretty good at estimating. If I showed you five stories from a project I was working on, even without context, I'm really confident that we would have a very consistent sense of which of those features was the most complicated and which of those features was the least complicated. And I'm even sure that we would be able to come up with a pretty good, rough, relative complexity. This feature's twice as complex. This is one and a half times as complex, that kind of it. It's something that we have a very good sense of. To think of it a little bit more graphically, almost none of us could tell us how many blue diamonds it would take to fill up this entire slide, but every single one of us will be able to say that it would take more red triangles to fill up this slide than blue diamonds. And most of us would be able to say at a quick glance that it would take, it's about a factor of four, that the red triangle is about a quarter of the size of the blue diamond. That's a kind of reasoning that developers are actually very good at. So if we focus on that, we estimate the complexity of each individual story and not try to think about how much time it's going to take. And we just let the time sort itself out. What we get is the classic definition of a point in an agile project, which is a unit of complexity and not a unit of time. If somebody starts talking to you and they're saying that a point should be about four hours or six hours or whatever, and I'm going to say that in a restricted context in about 10 minutes. But if somebody says that, they are either completely wrong or oversimplifying or possibly both. A point is a measure of complexity and it's basically dimensionless. Teams tend to find their own levels on them. You more or less understand that a point is a simple story that we can get our heads on that doesn't have much individual logic and you sort of work up from there. And what happens if you estimate everything in points is that you get to an understanding that your team can do 10 points a week or 12 points a week or something like that. And then all of the other stuff doesn't matter. Everything else averages out. The amount of time I spend eating lunch, the skill of the team, our meeting schedule, as long as it's all roughly consistent, we don't care about estimating it individually. All that happens is we found out that we can do 12 points of work and as long as everything else is consistent, which is a big as long as, this is pretty robust. It has a little bit of like up and down, but over time it can be pretty robust. So if you think about it, it's a little bit like basing the estimate of going to the airport on how far it is to the airport and how fast I normally drive. Which as, you can see if I put it that way, that it's going to be roughly true but not precise. But that's fine, over a long period of time, roughly true but not precise has a very strong tendency to even itself out. But the important thing here is consistency. And in a couple of different dimensions, like the team size needs to stay consistent. Adding and subtracting people to the team really messes with this or can really mess with this. It's important that you're consistent in how you grade stories and that you also have roughly the same mix of small and big stories at all times. It helps, it makes this a little bit weird if you have a few small, if the number of really big stories changes a lot. And the overall environment needs to be more or less consistent. The hidden, really hidden skill here is the ability to decompose your complicated problem into more or less similar sized chunks. If you can do that, like almost everything about creating estimates gets a lot easier. That's the key important skill. But you need a little bit of data for this to work. Like you really need two weeks or four weeks or six weeks of data to get a velocity that you can have any confidence in. And the problem with that is it doesn't really help you estimate things at the beginning of a project. And it's still, a one-off estimate is still a problem. Like in a completed project when the client comes to you and says, like I need one more thing, how long is it's going to take? Like that's still problematic. And there's, it's almost, for most of the reasons we've already covered, like it's hard, it's still very hard to make that not problematic. And we still have the problem of how to do anything at the beginning of a project before this data comes in. So the technique that I use at the beginning of a project, which I call kind of the worst way to do this, except for every other way I've seen, is to fake it. Not fake your knowledge about the project, but to fake your velocity. You need to, in order to make this work at all, you need to make a rough guess of how much time it's going to take your team to do a points worth of work. But you can do it in a very controlled way and with a sort of understanding of what you're doing. Don't spend a lot of time chasing precision here. You're not going to get precision and spending time on it is just, it's just a waste. What we come out of this process with is an output that is a range of probabilities, a range of a low estimate to a high estimate. And how you present that to your manager or your client is going to depend on your situation. But what you get out of this is a low value and a high value. So the way that this works is you take what you bet your best understanding of the scope of the project, ideally this is part of a process where you're creating this with the client. And you split the tasks down as best you can and assign points to them in much the same way that we've already talked about points existing. So a one-point task is easy and something can get your head around. A three-point task has a bit of more complexity or risk. I usually tend to spread the numbers out a little bit here. A five-point task has more complexity or more risk. Traditionally, people use Fibonacci's from here so the next number you would use is eight and then 13 to indicate more and more complexity or more and more risk. Don't spend a lot of time assigning points to a single task. If you can't, trying to argue over whether something is a one or a three is the kind of argument that you lose just by having it. You can't possibly win that argument, just round up. Round up will help, rounding up will help with your consistency and it will counteract your normal tendency to be optimistic about this stuff at least a little bit. And when I'm doing this estimate at the beginning of a talk and only at the beginning of a talk, at the beginning of a project, I will give stories a range. This reporting feature, if this gem that we might use will actually work, then it's a three-point story. If it doesn't work, then it's an eight-point story. Over the course of a project, I hope to be able to refine that but at the beginning of the project to make up for the fact that I don't know anything, I will give it a range of values. At the end of that, hopefully without too much pain, you have a range of points. You add up the low numbers, you add up the high numbers and you come up with something that says this project has between 100 and 120 points of work. And you can refine this more for high priority versus slow priority or something like that. How you do that, again, is going to depend on your context a little bit. And then you kind of the really hand-wavy step in this part is the point where you have to say a point is going to be somewhere between five and seven hours. I give it a range, again, like I just said 10 minutes ago that anybody including me right now that tells you that points are a particular set of hours is oversimplifying. I'm oversimplifying. But you need to get to a number. And the most effective way that I have found to do this is to assume that there's a range and then that combines with my low and my high number to give me a low and a high estimate for the number of hours. So I have 100 to 120 points and five to seven hours per point. That gives me a very broad range of 500 to 840. This process usually comes up with ranges that are going to be much broader than to be useful in a particular context. So often we present only the middle of the range because the extremes of the range are much less likely than the middle of the range, ideally. And then you can multiply this by a load factor to get talent. Calendar time, you can multiply it by a dollar amount to get a budget. And then you need to present that to your client. And when you present, or your manager, or however. And how you present that is just as important as what the process is. Unfortunately, there's a lot of things about that that I can't tell you because it's going to depend on your context. But generically, it's important to have that be part of a process where you explain the limits of this. At TableXI, we often have this sort of elaborate spreadsheet of graphs that we show that show the range of possibilities if you only choose high-priority stories or if you choose lower-priority stories. But we try to do a really good job of explaining how we got to this estimate, what is in it and what is not in it. The problem is that very often when you say to somebody, this is going to cost between $75,000 and $126,000. What they hear is, this is going to cost $75,000. And that is a tendency that you need to counteract. And one of the ways to counteract that is by really focusing on the things that need to happen to get to the low estimate. The way that everything, the things that need to go right, the idea that it is a minimum and not necessarily a likely. Or in some cases, you just don't present the minimum at all. You only present the likely number. Again, it's a situation that you have to kind of read. So you need to describe the results. You need to understand that this is going to change over time and you need to be flexible to that. One of the advantages of the agile method of points and velocity and things like that is it gives you a really nice cadence to see whether the project is headed towards being completed on time or not. And that gives you an opportunity to have further discussions about ongoing scope, ongoing rework, and ongoing priorities. So that is also part of it. The estimate is not a static thing. It's a thing that exists over the life of the project. Because ultimately, all the people in the project are kind of headed to the airport together. And it's best that you both have a common understanding of where you're going and how long it might take you to get there. So that's it. Thanks. That was 29 minutes, if I'm counting right. So close. I have one image credit and a couple of things that I want to tell you about. Again, you can find me on Twitter at NoelRap. Again, feel free to say hi or ask a question. I work at TableXI in Chicago. If you finish this and you thought, wow, I really want to hear everything Noel has to say about running software projects and agile projects, I have a self-published book called Trust-Driven Development, which you can find at nolrapin.com. It's normally $15, but for the next couple of days with the coupon code RubyConf2015, very creative, you can get it for $10. It will probably never be around at that price again. So if you want to be one of the lucky dozen people or so that have bought it, have at it. I also did a book on testing for Pragmatic, which you can find at that URL, prag.com, to end our test too. And that's all I've got. Thank you for your time and attention. I hope you're really having a great time at the conference and enjoy the next couple of days. I have time for a question, I guess, but thank you for your time. A couple of questions. Well, okay, so the question is, like, it doesn't agile imply that you don't have a master plan at the beginning. I think that that is sometimes, so we sort of get, this thing's in between like, the ideal agile project and the way things sort of exist in the world and what, and how you can kind of move between them. I find it very, very difficult to work with clients on projects where there is no master story plan to start off with, even if it's just like, we would like this to be the first release. Like, these are the things that need to be in the first release and it's not the entire project. I think it's beneficial to set an initial endpoint and say, this is what's in and this is what's out. Otherwise, it's just really hard to understand whether the project's going well or whether it's going badly. I don't know that the manifesto really, the manifesto itself really talks too deeply about that level of detail. It may or may not be a deviation from how a lot of these people think it probably is. I may be deviant in this respect, can I say. The question back there. The question is, do I find it useful to put points on things that are not features that? My answer is no, although I see the argument for yes. The argument for no is that it's very, very hard to estimate bugs and chores are not in an agile sense user facing stories. The math works out as long as that is a consistent percentage of the work. If everybody's doing 25% of their time on bugs and chores every week, then you still have that consistent cadence and the points still hold. In practice that becomes a problem because ideally at the beginning of the project you're doing more chores and at the end of the project you're doing more bugs. It doesn't quite work so it does tend to be it can get in the way of the estimates a little bit later on. Ultimately the point is supposed to be a measure of complexity, the story is supposed to be something that provides user value and actually does provide user value, can come in as a feature. I don't know if that's actually kind of an answer but I don't in general find it valuable to estimate those and add points to them. So the question essentially is that I'm making an implicit assumption that there's a very direct relationship between complexity and time and that may not be true. My inclination and I think the larger the project I think the bigger the inclination is to not try and outsmart the process. That gets tricky. Maybe what that means is the one point story that takes a lot of time is actually it has a lot of pieces that can be verified separately that maybe the thing to do is to break that into several one point stories that can be independently verified and independently deployed. Maybe that can't be done. I was on a project not all that long ago where we had a data migration story that was one point stories. We didn't immediately realize that we were migrating like 35 tables and it took like three weeks. That does tend to maybe what that means is but in that case I think you can say that we legitimately underestimated the complexity of it and should have had multiple single point stories. I think that there are always going to be outliers and I think that that my experience is that trying to do a lot and that overall you're just saying like sometimes there will be a one point story that winds up taking several days and sometimes we'll have a one point story that takes five minutes and that they're just it's just a 50% chance of rain. It's a probabilistic thing. It's not a super satisfying answer. It's like the least bad answer I have to that question. Sure. I mean, sometimes you say like one of the metaphors is like the bicycle car, the bicycle version of this, the car version of this and the truck version of this or like the penny nickel quarter version of it. Yeah, you do need to go to that. Ideally, depending on how you do it sometimes you go to do that. At the beginning of a project sometimes you do it at the beginning of an iteration. The earlier you can do it the better you can fit into the estimate. At the beginning the later you do it the more you can respond to the the practice on the ground when you get there. But yeah, that's something you change. Yes, there's a version of this that's going to solve 80% of your problems at 20% of the time. Is that good enough? That's always a constant negotiation. So yeah, that is something you do. So the question is do you take into account the consideration of choosing the framework when you make the estimates? Strictly speaking, I would say strictly speaking in the don't outsmart the dumb process kind of way I know because the way that would play out in the process is the complexity of the stories is the complexity of the stories and what the framework will do will impact the velocity. So I would, if we're all struggling with the framework then our velocity is going to be lower but the stories are the stories. In an ideal world where you have a very long project you have time to see that play out and you see our velocity here is only 7 and we thought it was going to be 12. It's a little bit more problematic when you have to say something at the beginning. You have to kind of guess. It's another level of unknown. So at some point eventually we come down to the future is the future and we can try to manage that but sometimes we can't. I feel like all I'm saying is yeah, shrug, yep, that's problem. Got me. Any other questions? Cool, I think we're good. I'll be around for the next couple days. I talk about this stuff a lot so feel free to come up to me and ask me more questions about it or tell me where I'm wrong because that's great too. Thanks.