 Hello everybody. This is back-to-back presentations. Maybe the only time during the conference, so I have a half hour, so this will be a bit of a lightning talk, but I'm going to keep it high level for that reason. Hopefully a while of time for questions as we go or at the end, but if not, you can meet me pretty much the rest of the conference anytime. So I'm Steve Wolf. I'm with Raleigh Software, and I'm also channeling a partner of mine, a colleague of mine at Raleigh named Larry Matcharoni. Is anybody here familiar with Larry and Larry's research work on agile metrics, agile analytics? Well, I will give you a little bit of an insight into what he's working on as part of our research efforts and how we're looking at helping teams improve with agile. Now one thing I would do want to say is that we are working at Raleigh on what we call a performance index framework, and this is what Larry's work is largely based on. And the basic idea is to, we could talk all we want about how to measure team performance, but what we're trying to do is do research with our large number of customers around how they're performing, looking at their data and coming up with key outcomes and key metrics that we can use to help all teams using agile improve. So the question is why? Why do we want to measure? Why do we want to see how we're doing at an agile level? How many of you are familiar with the saying culture eats strategy? Has anybody ever heard that one? Basic premise, no one's heard that one. The basic premise of that is that no matter what kind of plans you lay out, no matter what your strategies are, the cultures of the people doing the work is going to really dictate how well that ultimately goes. If the teams aren't motivated, if they're not engaged in the work, I wouldn't say they're going to purposely sabotage it, but the effect will be that you're not going to have the outcomes that you're ultimately shooting for. So it's important that when you look at measuring agile, because it's such a team oriented approach to getting software developed and done, that you build these measurements with them in mind. In fact the whole idea here is that give them measurements to help them to complement what they already know. Put into terms that they can now see and visualize how am I really doing these insights that I have? Are they working? Ultimately let them make the decisions on the ground. We empower agile teams. We give them business ownership. We give them responsibility for solving customer problems directly. We need to give them the measures now to see how they're doing with that. So that said, there are some ways that you can misapply agile metrics. We call them the seven deadly sins. Some of this is the outcome of Larry's work that I mentioned earlier. He starts to look at some of the bad practices as well. We've all heard about the dark side of metrics. What we're trying to do is avoid that and give you some real practical input on how to effectively measure. So why do we want to measure? It's back to this. Sin number one. Sin number one is don't use metrics as levers. Don't use these to be at a team over the head and say you need to do more, go faster, etc. Also don't use them as absolutes. They're just feedback. It's a feedback loop which we build into agile from the beginning. Use it to help a team or an individual understand how are they getting better at their own performance or the team performance. Is this insight that I have really paying off? Is this training that I took really making a difference? Give people some insights into that. Don't use levers again though as they can be a hammer and that's where people start to fear them and that's where you start to see teams not even really want to deal with them. Sin number two I don't know from America. Probably figured that out already. I'm from Colorado. We love basketball there. I think Crickets are a professional sport here. Maybe soccer. Is anybody here watch NBA basketball? A few people? Good, good. Well here's a player who I guess used to be near and dear to my heart until I figured him out. He played for Denver which is where I live and basically the sin here is don't look at a metric and then guide everything around it. You need to look at the outcome or what you're really trying to achieve. Here's a player Carmelo Anthony and there's another example up here. Score is a lot of points. People used to love him for that. I think probably a lot of people still do love him but when you really evaluate the team performance which is the outcome we're really shooting for how's that team performing? He's had mixed success. He takes a lot of shots. Some people would argue he steals shots from the rest of the team. He's so focused on being the star and getting his do and taking his shots but that's not really leading to the best outcome for the team and this is backed up through some analysis. There have been periods of time where he's been out of action due to an injury or what have you and the numbers would show that the teams actually performed better most of the time when he was out because he wasn't we were looking at a metric and glorifying him around that metric that didn't really lead to the right outcome that the team had and that basically is to win. So that's now how we want to present and look at metrics. This is a term Larry's come up with. It's called Odom framework. You don't want to start with the metric. You don't want to look at a scoring average in basketball and think somebody's getting the job done. You want to look at your outcome. Start with your outcome, work your way back. The way this fundamentally works is metrics just help you gain insight into a problem at hand. From those insights you can make better decisions and those decisions will be in place really to drive the outcomes that you're going for. An outcome in this case in this example being better basketball team performance but in an Azure software teams case it would also be team performance as opposed to something very specific. Bad analysis. This chart here is a fairly typical statistical standard deviation chart and a lot of organizations will look at this and look where you sit within this to measure and determine how you are performing as a team. This has application. It works well. We see it work well in manufacturing. Let's say you have an axle manufacturer and you want to measure how many of your axles are going to come in with a certain diameter and what the outliers are going to be so that you can then start to set expectations for the people you're selling to, even a service level agreement. This works great for that type of application. Works great in business where you start to see normal curves emerge looking at the mean and looking at the standard deviation but in software it's a little bit different. This is actually a chart I'm going to talk a little bit more about it later but it's a chart from real data from one of rally's teams and essentially what it is showing us is that we're not normal. If you look at the curve and the histogram on the right you can see that it's not flattening out the tail in fact is starting to get a little thick at the end then fat tail we call it and this is because of the nature of software. Software is a very inherently complex problem you may also have other reasons for some of these stories that you're working on that allow them or force them to get larger over time and so your standard deviation might say that in a normal world we would be at this 98% point which is something that we would then use to predict how we're going to finish off in a normal amount of time let's say two weeks because of the fat tail phenomenon we see in software that can actually lengthen out quite considerably because you do have these outliers these cases these features are these stories at the end of the tail that are taking a lot longer than you expected and really again just to amplify that point it's because of the nature of software and sometimes these tails these outliers are great sometimes you want them let's say it's a black swan situation where you're someone's just exploring and learning and coming up with an entirely better way of doing things it's worth investing that time maybe again it's a simple research element or maybe it's something you need to go investigate why are these stories blocked what cross dependency do we have for example let's go ahead and evaluate and look at those more closely that in turns help the insert insights we gained from that in turn help us make better decisions down the road so cycle time is really a pretty significant measure in software delivery we want to know quickly we're able to get a feature or a work item through a given set of states again to help us with predictability and this is a chart I think taken from the combine world where you try to apply like a manufacturing approach to process control to software and it just doesn't work there are reasons for these outliers and what tends to happen then when you start to use charts like this these red dots become things you want to avoid the people working on the teams to start to fear those because they start to get beat up for them even though there's really practically good reasons or good insights to be gained from understanding why you're having those outliers so it's really not a good approach to use process type control charts when you're managing a software project or looking at a software project and again this is back to the chart I showed you earlier it's got some similar elements this is a chart rally has come up with we call it a time and process or time and state chart and you can basically define it at any work item set of work items you want let's say from a portfolio level down to a team level around tasks and defects whatever you want to look at and then you start to look at it in the context of cycle time and again that could be however you define it it can be from development time through acceptance by a product owner it could be through up front story creation all the way through release into product so you can define both sides of that and then you can start to really evaluate how's this team doing and how where are they at so you have the scatter plot that I showed you in the previous slide on the left you also have this histogram on the right and what you you can see very quickly is that you have this large distribution where 30% of the work is being done in four days or less probably typically what you would expect if your stories are set up the right way and of the right size and on down the line that 98% that magical 98% point that a lot of people use for to declare nearly done or set an SLA or something like that around it's not a normal mean you can see that it's really closer to 40 days which is not what you would expect if you're looking at that first standard normal standard deviation curve that I showed you it's much longer than that and it's because of these outliers that I talked about now this trying to rally you can actually look at each one of those dots and start to look at it in depth understand what's behind it what tasks are underneath it that might be blocking whatever the case may be and you can have a good discussion around those outliers it's not to say these are bad again we're not using this as a hammer to say these three dots are out of cycle or they're taking a long time you know let's go beat somebody up about that no it's to have that good discussion and outliers are the key in software they're very key that's where you learn how a team is functioning what's getting in the way it's where you learn about good research that that team is doing that you want to amplify whatever again those insights might be the discussions usually happen around the outliers the rest of the work is getting done as you projected or as you planned a lot of people also fear you know all this work involved with collecting the data and the visualizations that you would need and I would say that 90% of what you need is already in the system that you're using a system that gives you access to your data and allows you to report on it visualize it either out of the box or through a custom set of reports that you would create not a lot of work you know you don't want to go off and start to create some crazy amount of obscure metrics when most of it's in the tool that you're using it's based on the process that you're using and what you set up in the system so the perceived cost can be a lot higher and therefore become a kind of a fear point for teams that they don't really want to just make that investment plenty of data in the tools the other thing you can do is again simple insights can be gleaned or good insights can be gleaned from simple statistics and what we have this concept of customer and employee satisfaction doing that promoter score one or two questions and I think you get a pretty good idea of how something is going that you want to take a look at so the collection does not have to be hard at all since six and seven are interrelated the first one is too many I think the picture says it all I don't really have a lot to add to this other than if you give somebody five dashboards to look at and ten reports and they have to evaluate that at their stand-ups every day or every week through some sort of review process they're going to shut off they're not going to look at it plus those metrics will start to conflict with each other so keep your regiments simple identify three or four in the key areas that you're looking at you want to look at and then have a cadence around making sure you're evaluating those as you go you also want to have enough to be meaningful and I mentioned the software performance index work that Larry's undertaking the whole goal here is two full one is balance and one is to then figure out the metrics in each box that makes sense so when you start to look at a software teams performance there are elements of productivity around getting things done quickly or being really responsive to change let's say defects come in how quickly are you iterating on those doing it right you want to look at quality but you also want to weave in and bring in customer feedback as well into that process make sure your customers are satisfied doing it on time is it how predictable is it again with the outliers being in the mix as well that you want to evaluate so predictable within that context and how's the team doing really how are they happy are they satisfied with the structure of the team and how things are progressing in the work that they're doing so it's really important to keep everything balanced if you overweight over emphasize one particular box you're going to end up doing well in that box but poorly in the others you can almost think you've probably seen these like spider charts where you can measure in these four dimensions and plot where you're at in each one and you're going to want to have a nice distribution across them so those are really the outcomes that you're striving for you want to get it done as fast as you can but in balance with the rest of these other areas not just getting it done fast because in that effects let's say quality so find your set of metrics and keep them in balance those are things to watch out for lots of specific details we have this performance index research going on that I mentioned to you we are working with a number of our customers very directly with access to their data to help define based on those outcomes where we would want to have or where we could see specific metrics that come out that really that they can then use but that we can also start to look at trends too because our ultimate goal would be to through this research deliver a set of standard reports and metrics that could and papers and such around that that could really give you a nice starting point around an agile metrics regimen so it's not all fire and brimstone here at all it's there are definitely things you can do to measure and you want to measure again teams want to empower teams want to improve they're the ones making the decisions ideally they need to get better they want to get better so you need to help them do that so again start with the outcomes come up with a random metric and say this is what you need to hit what you want to do is understand why that metrics important relating it to an outcome that you have and then making sure that as you see that metric that you're having good discussion around it make sure they're balanced you do not want to overemphasize I think it was mentioned this morning the software engineering tool kit you do not want to burn people out you do not want to overemphasize time to market at the expense of quality take a look at the whole package make sure things are balanced be careful in your analysis right don't fall into standard traps of looking at process type metrics that maybe work in a different field this tip chart is an example that we showed you today that gives you an idea of rather than coming up with the control chart because we're not really trying to control things we're just trying to gain insight so we can make better decisions you just find the charts and the metrics that help you really make those good decisions and then better the collection costs in the end don't don't overdo it get maximum value out of the things the system already provides and then add to that subtly as you go don't overdo it or it's going to become number one it's going to become overload number two it's going to be something people won't do because it just takes too much time amazingly enough that's all I have that was really a lightning talk so any questions at all I think I have time for a few how do you pull it out I can walk through that a little bit more let me back it up here so the tip chart here there's really two sides to it we walk over here the dots that you see on the left the scatter plot those are individual work items that you're basically categorizing or reporting on so each one represents a work item on this chart within the system you can actually go in and open up that you can click on it open up and start to see details what tasks underneath it you can look at just the revision history of that particular item this vertical column with percent number going down the middle that tells you over time or over this particular time slice that you're looking at what percentage of these are done in that amount of time so you have a number of days on the left so what this basically is telling me is that the 50 percent pretty quickly in four days if you look at the 50 percent number near the bottom 50 percent of your work items within this sample set are done in that four day period you go up you can see that 98 percent point way at the top you can see that's right around 40 days so you need to look at those two vertical axes together days and then percent done percent of work items that you're looking at done and then the histogram is just another way of looking at how many days and how you had a grouping of the number of days it takes for the aggregate number of work items so in this case 180 of your work items are done in 0 to 2 days 2 to 4 days for another 120 of them you start to see this normal distribution that I talked about occur until you start to go a little higher and you see the tail start to extend out which is the phenomenon we see in software a couple other points this is actually a really interesting chart that I'll point out you can see to the left you do not have a lot of these outliers the first and there's other insights as you get practice with this that you can like it I just mentioned the one where we didn't have outliers because we swarmed you can see a holiday break in there in December and a hackathon break we run internal hackathons things like that yeah that good question question is how do you analyze in the context of story size it's kind of some of that's built in but a lot of that also is around estimating and make sure you have a consistent way that you're estimating your stories and so this normalizes around a certain size of story or story point estimate and then by doing that then you're also then allowed to take a look at your estimates as one insight that you might glean from some of the outliers for example but that is normalized across these any further questions yes yeah there is you know I can talk to Nuresh and follow up we have some white papers some research papers already coming out around the performance index framework that I've talked about but some of the work we have a great white paper on this tip chart that I think would be particularly helpful if you're interested in it so I don't have a copy of that with me but I can talk to you offline afterwards too and just point you to it on our website and then beyond that I can make sure we get that information out through the conference okay any other questions what can you measure against time time is one axis what do you measure against what is the other axis well the bottom is time there's a time element on the left this is just a sequence of time the two y-axis is on the left one would be number of days and the other would be percent done number of work items done as an increasing percentage and that's also against that time so see the number ten on the left that's ten days time in state for ten days and if you come to the right you see the 81% of our work items were done within ten days so you start to see that increasing it's another way of looking at that bell curve but it's actually on the left now on the right and it gives you a chance to also factor in and see some of those outliers that I talked about does that make sense and then on the right this is a companion chart it's like two charts in one the histogram on the right will basically tell you the total number of stories that were done in a certain number of days so the bottom line here is zero to two days this will give me the number of my stories in this case that were done within that range of time so two days if you go maybe towards the middle you can see that between 24 and 26 days there's probably one or two stories that got done that took that long to complete but you can see again the outlier point that extends on there's definitely one or two that continue on all the way up to 46 to 63 days 63 would be the high end in this particular case we just create one last category with the highest number of days so again the goal is to give you some great data that you can look at and focus on maybe those items that aren't performing the way you thought that they would at a time anybody else have a question? Okay thank you everybody enjoy the rest of your conference.