 Hi everybody. I'm going to get started. I'm Mia, Mia Horrigan, Strayer via Ireland in case the accent sounds really weird to you. Yeah, I wanted to talk to you about evidence-based management for business agility because it's something that over the last couple of years I've been working with executives. And what they've been saying to me is like, Mia, you've spent hundreds and thousands and millions of dollars on this agile transformation. But how do I know if it's working? And how do I know if we're producing better stuff? I just can't see it. I've got no understanding of that. I mean, you know, has our product delivery improved? Are we got happy customers? Are our business happy? Are our stakeholders happy? Do our employees actually feel more empowered? So has all this money that we've been piling into this thing, is it actually worth it? And the problem that we have is because, and if you were just in Alex's talk and I know some of you were, this is because this is how we traditionally think about our projects. We have scope. We have budget. We have time. And our success was that we did everything on time, on budget, and we delivered all the requirements that up front we agreed to in our contract. And the problem is that we're tracking a project as if it was going to just follow a nice little plan and it was very, very predictable. Because it assumes that there's going to be minimal change. We just plan it, we do it, and then we're successful. But the problem is that this forces us to focus on activities and output, but not on the outcomes and the values. And that was the thing that we're missing. We were doing lots of stuff. So we could show our CEO, that, hey, look at all this activity we've done. Therefore, we must be good. We produced lots of documents. But did we get the output? And did we, sorry, did we get the outcomes? And did we produce anything that was actually valuable? So value is the reason why we do things in an agile way. And we've really had to shake that triangle up and actually evolve from this traditional mentality that all we have is scope, cost, and schedule. And we can actually plan that to the nth degree and it'll be fine. In agile, we think we care more about value and we definitely care about quality. So the things that are our constraints are those scope, cost, and time. And we tend to see the scope in our product backlog as the things that we can actually change because as we learn more, we know that what's valuable might change because of what we've learned. So we need to think about things a little bit differently in agile. Because what you measure and track today could be very, very different. And what you measure talks a lot about what's important to you. Or at least it should. So just next to the person next to you, just for a minute, this is, I promise, not too much interaction. Just for a minute, just talk about what are the sort of things that you're tracking right now in your company? What are the things that you say to show you what you're measured on? What are the things they ask you to track? So just a minute to talk to the person next to you. You can go in threes if you can't find a friend. Okay. So at least you know the person next to you now. So that's great. What are the sort of things, anyone happy to shout out? What are the sort of activity, what are the sort of things that you're measuring? Yep, just velocity. Yep. Yep. So the progress of the project, the velocity and the last one was? Oh, yep. Excellent. Any other things? Yeah. Yeah. How are we tracking against the actual in the forecast? Yeah. Plan versus actual. Yeah. So you can help improve your forecasting. Cool. Yeah. Time to market. Love it. Excellent. Okay. Co-quality. Anything else on this side? Yeah. Cycle time? Cool. Yeah. Stability or predictability. Excellent. Impact on operations. Lead time. Are you guys advanced? I can probably go. Now that's excellent. Because a lot of times when people sort of are thinking about their activities and you ask them to sort of put them on a post-it note and do this with your teams. Do this exercise. Ask them what are we measuring and ask them to track whether it's an activity, an output or whether it's actually something that's an outcome or something that's an impact. Because you'll probably find you have a mix of all of them. And, you know, the activity outputs and have their place at the team level to help you with forecasting and improve and things like that. But are we thinking as an organization about what we want to achieve and we think about our impact and the actual outcome? Because the activity is just reflecting what we've taken, the activities that we've done, the outputs just reflecting what we produced. But where we want to get to is, well, what outcome? Did that change our customers thinking about how we're doing things? Did it change users thinking about us as a company? And the impact on the company itself as a result. They're the sort of things because just measuring activity isn't really telling us if we're delivering value. So, yeah, do that activity with your teams and your leadership and just see what you're tracking because what is important and what you're tracking are the same thing. So, if it's important to you, track it. Because what we find is what we measure is important. And if we're measuring the wrong thing for the wrong reason, we can go down the wrong track. Because typically our teams are used to, if you're tracking it, it means that it's going to be used against them. And velocity is a classic one. It's called the observer effect and you can see it in there. That, you know, oh, your velocity is not good enough. You need to increase your velocity. The teams will game it and they'll give you a bigger velocity. It won't be meaningful and they won't be using it the way you were talking about to see if it's forecast versus actual getting better. You've told them more velocity, they give you more velocity. It's not a measure for an organization to be tracking. It's to help the team improve. Help get them better at their predictability and their forecasting. The other thing we find is the streetlight effect. We're counting code. How many lines of code did you deliver? If you didn't deliver enough lines of code, then you mustn't have been productive. Well, just because it's measuring the number of lines of code, does that actually mean that anything you produce was valuable? It could be one or two lines of code that are the most valuable pieces of code you've got. So think about those things when you're doing measures because what you measure is important. And how is value measured? This is from the Chaos Report. This comes out every year. Big survey of 30,000-odd organizations, big, small, medium enterprises. And what they found was that all of that software that we're producing and the features that we're producing consistently, 65% of it is rarely used or not used at all. That's a lot of waste. That's a lot of things we're doing that haven't produced anything valuable. Something that we produced up front, we thought was required, when it got out to our customers, this wasn't something that they were interested in at all and rarely used. We also sometimes think about practices. And if we're doing these new cool practices in DevOps that we're talking about nowadays, are our developers busy? Have we got test automation, continuous automation? We sometimes get focused on the practices and the effectiveness of those. But do they actually mean we're producing value? We need to also think about the next layer. What's the impact that has? Why are we doing that? And can we measure if it's had an effect? Because if we're measuring the wrong things, it could have really undesirable consequences. Because measurement is really strategic. We have great dashboards we can do with some wonderful tools nowadays. But if we're not measuring value, the success of what we're doing is really just based on our intuition and assumptions. We think it's working. We're not quite sure. But you know, we have no idea whether actually closer to achieving what our users want and whether we're having an effect. So oh, so that's we've done that. Okay, so evidence based management is something that it was a couple of years ago came out with Ken Schwabber. In more recent times, it's kind of been reinvigorated with a different thinking around the outcomes and the impacts. But it's really about measuring value delivered as evidence of organizational agility. And hence in this business agility day, it's kind of fairly interesting because we're now finding that as businesses want to become agile, they also want to look at metrics across the whole organization that they can actually use to see whether they are agile as an organization. The model is fairly simple. It's we look at a lot of different quadrants. So at the top, we're talking about customer value, market value. And that's all the things about the current value and the unrealized value. Down the bottom, we have the organizational capability. So these are the things that are going to help us achieve those things. And it's our ability to innovate and our time to market. How quickly can we get things out there? And then they are measures that we can use that help us work out whether we're delivering value or not. And to help us guide our improvements. So we're going to drill down into the model in a little bit more detail. It's very much fits with that agile model of inspection and adaption. So we measure that key value metrics, we select the key value area that we want to improve, we'll conduct an experiment to improve that value, we'll look at that outcome and results and then keep going. I'm going to talk about a lot of measures today. I think there's about 40 odd measures. If you try to do all 40 at once, when you're successful, you don't know which one it is. So do little mini experiments and work out which ones you need and then move on to the next one. Trying to tackle it all at once. So let's talk about current value. So this is really the goal is to maximize the value that we have in the present day. It's measuring what you're worth today. It's not thinking about the future value, it's just what you're worth today. And it reveals your actual value in the marketplace. And there's a lot of different measures that you can use to actually help you with that. Some of them are leading indicators and some of them are lagging indicators. How happy are your customers, your stakeholders, your business, your investors? And are you improving or are you going backwards? So the leading indicators that you've got there are things like employee satisfaction, customer satisfaction and also usage index. So remember that chaos report I showed you of 65% of it not even being utilized. What's the usage index of your products when you actually get them out in the marketplace? Lagging indicators that are also helpful here, but unfortunately because they're lagging, you're not going to see them for a while, are things like revenue per employee and product cost ratios. These costs are really important. I was actually talking at breakfast with Scott and he was talking about if we measure velocity and it increased, but then it took you extra people to deliver that extra bit of velocity, you've actually increased your costs. And that's something your CEOs and CEOs want to know about because you deliver it, but at what cost? So also think about the revenue per cost because if it's getting too much of your margin is being eaten away because it's taking you more people to actually deliver that extra. Is it actually of value anymore? It might be something that's not valuable for that cost. And it really comes down to understanding our customers. So I'm assuming most of you use personas or something fairly similar. Really good starting point for your product owners and your teams to sort of start to build this picture of who are we really building this for? You've probably seen those user stories where as a systems developer, I need this where it's very much focused on the technology in the system. User stories all about thinking about the user and the value that we're giving them. It's talking about what's their story? What's the trigger? What's their goal? What's their motivation? And how can we help solve their problems? What do we know are their pain points? Another thing I really like to use when I'm working with teams to kind of work out what the value is, is doing the value proposition canvases. What's your value proposition? Why do our customers actually want to work with us? You know, it's so hard to get those customers that being in sales and marketing for a long time. It takes a long time to woo those customers. If you've got existing customers, you don't want to lose them. They're gold. So why do your customers actually want to work with you? Do you know why? What are the outcomes that they actually get from your products? What are the things that really delight them? And how do you measure that? So your value proposition canvas that you've got there looks at all those customer wants and needs that you've got in your personas and the pain points. But then looks at, well, how does that match to the product offering that we've got? What are the benefits? And what's the match that will give us our value proposition to that customer? Impact mappings, also a really good one in this particular era for looking at current value. And you're starting with the goal. You're thinking about those actors which are represented in your persona. You're looking at the impacts that you've got and then the deliverables. It's very goal driven and you can prioritize your features based on those goals. A really good way to sort of think about it. And you can then start to track your progress towards the impacts you're having as opposed to just the activities that you're doing. And if you've got an existing backlog and you're starting, once you've got your goals you can actually track it back and validate things as well. I like to actually use the modified impact map that Jeff Patton tends to talk about. And rather than thinking of the goal persona impact and deliverable, he then breaks it down into what's the outcome and how do I get my user stories from that. So you've got that beautiful traceability from your goals all the way down to the PBIs and user stories that you're doing. So much more impactful. And it also helps you to keep your user stories very outcome focused. So if you've got that outcome focus it's going to be easier to track what those outcomes actually are. So it's hypothesis driven development. So here's a good way to do it. We believe in doing this which is your feature for these people which will achieve this outcome. And we know that will be true when we see this measurement changed. Just a different way of thinking about your user story at a feature level. Because we're running many hypothesis and many tests and experiments all the time in an agile environment. Here are the key measures again and some definitions of them for current value. So employee satisfaction, customer satisfaction. Simple things like just doing a sentiment analysis. Very very powerful stuff. The usage index. How much are we being used. Revenue which is basically your gross revenue divided by your costs, the number of employees and your production costs. Really really key things to understand. Let's talk about unrealised value then. So we've talked about what we're currently worth. Let's think about what future might hold for us. And something like the opportunity canvases is a really good tool to try and find out what are the opportunities there that if we looked at this idea that we've got, what could that potentially be worth to us? Would our customers actually even like it? Would they value it? What dollars could we get from that? What unrealised potential is out there? And the goal is to actually maximise the value for the organisation by bringing in some of those new ideas. Again there's leading and lagging indicators that you can look at with unrealised value. So the key thing here is in the leading indicators what are the competitors strengths and weaknesses and is there an opportunity there for you to capture some market share because of that? What's the customer acquisition or defection? A lot of people leaving a particular product because there's a gap in the market is that something that you can actually fill. So tracking those sort of things is really critical. Lagging indicator is when you actually get your results in your market share valuable information but again it's a lagging indicator so having those leading indicators to help guide you is quite critical and whether the market itself is growing or declining is also an important metric to be tracking for your organisation because you want to actually know if I invest money in this is it actually going to give me a return? Is it actually worth the risk that I'm going to take on particularly if it's a new or novel market? And there's lots and lots of different ways to think about your marketing strategy but you probably want to get a balance of your current value and your unrealised value because if you've got all of your money in a cash cow which you know today very high value for you as a company if there's no innovation in that area or any way that you can reinvigorate that product over time your it's a crowded marketplace there'll be other competitors that are coming into it so you need to start looking at other things you've got some dogs in your market that probably are things that are legacy systems that you've got to let go of they're costing you money they're taking up time that you could be doing I'm spending it on other things so think about mapping your products and where they're at in their life cycle in this kind of quadrant your stars high current value good high potential for growth they're excellent and your rockets are ones that have got really good potential you might have done a beta or tested something in the market people loved it and if you put a bit more money into that that's going to be a future star for you so think about your your products in those sort of things to see which are the ones worth pursuing and what value can we get from that there's also the concept from Don Wright's sort of cost of delay and this is something that I put together for my client who's actually a government agency because we couldn't really quite understand in the government situation that we don't sort of have revenue or care about revenue it's more about compliance and getting people signed up to different programs and things like that so we have to think about what business value time criticality and opportunity enablement meant there but the key thing in cost of delays you're actually looking at if this is valuable what opportunity does it open up for me and if I put money into that is that going to be worthwhile versus other things that I currently am doing that a business value as well so thinking about it as cost of delay or wish of which is the formula up the top there helps you understand of is this worth pursuing is this an idea that will actually give me a return if I get it out to market quicker so when you test your hypothesis for its success kind of think about things like how would it take for you to know whether you've actually achieved this is it the outcome you wanted to achieve and is it the impact that you wanted to see so always be testing your hypothesis and running those many experiments each and every cycle and the unrealized value that market share and the customer user satisfaction gap are the key things that you're looking at in that unrealized value area okay so they're the things that from a customer value point of view a business value point of view we think about when we think about time to market these are things that as an organization you can actually influence in your organization today so time to market is how fast can we learn from new experiments that we're doing how fast do we learn from the information that we've gathered to inspect and adapt what we're doing how fast do we live a new value to our customers because the thing we're trying to do here is actually maximize the amount of title like sorry minimize the amount of time it takes to get to our business value to get to our goal because if we're not maintaining our time to market and thinking about that we actually could be lagging behind our competitors and we might not be able to sustain the value that we're wanting to get out there here's a idea map that we did we mapped with one of my clients you know how long and what the cycle was and what were the steps that it took to actually look at how we get value and improvements out there so we started with just the simple ideas you know applying our strategic filter understanding the problem and but and then we eventually got it out there measured our outcome and our trends but the key thing here was we were always looking at the feedback loop trying to find out was that valuable do we keep going have we proved it or not okay is there something that we want to get information on because if we're not getting that feedback it's really difficult to understand if we're if we're on the right track if this is actually an idea worth pursuing and we want to know how long it's going to take us to get it to market so being able to map that end-to-end process is really critical so how long it takes to get to market the sort of leading indicators that you're looking at there is the frequency of your build success build pass fail trends release stabilization trends and the meantime to repair the lagging indicators very valuable as well but things that you only find out after the fact cycle time release frequent let's see lead time and time to learn if you've seen Jeff Patton's talks I think he was here last year he talks about time to learn and time to earn and each of them are equally valuable so how long does it take you as a company to learn that's going to influence how long it takes you to get your time to market value stream mapping is when you get that end-to-end process of what are the steps then you actually start to look at well actually end-to-end what does that look like what's our current time to market what bottlenecks am I seeing what do I need to improve or remove is there a lot of waste or technical debt in this I know that this example is really small for you to see but it was mapping a particular government agency and it was an initiative that they had from legislation and it actually took 238 days to get from the idea to actually in production guess how much oil you can see it on there if you can read really small guess how long it actually took to actually develop that particular product about 30 days so 30 days of actual work by the development team took 230 days to get through that system but doing this value map was really powerful because we actually stood up we're able to show them you've got a bottleneck here in design because you're doing upfront design you've got a bottleneck here in testing because you're doing enterprise testing after the testers have already done all the functional integration testing so there was duplication there was waste there's a lot of things up here I'd call this water scrum fall was what they were doing their development was agile but everything else was very very waterfall but the value of the value stream map was to actually show them how long it took them time to market to actually get an initiative once it was announced into production and the last part of the quadrant is ability to innovate so that's all about what prevents your organization from delivering new things of value what are things that are slowing you down and what's your stop your competitors as a result from benefiting from these new innovations it's helping you to avoid things that are overloading your system and really thinking about the system perspective because we want to get new innovation out here so these are some of things that are impeding your development it's inability to hire the right people maintaining multiple code branches and things like that complex or old application architecture insufficient product like environments to test on multiple levels of environments to test on lack of decentralized decision-making lack of operational excellence and actually spending too much time on fixing bugs and defects and technical debt that's really every time you're spending on that it's stopping you from actually working on innovation so the leading indicator is here and there's quite a lot of them in this particular section because a lot of them are still things that today we're still grappling with our technical debt trends our architecture our defect trends our production incidents our dial time how long is our downtime number of active branches that we've got that we spend a lot of time merging time spent on context switching if you're at Doc Norton's talk you would have seen what happens with context switching and velocity trends that we've got some of the lagging indicators are innovation rate installed version index and usage index but the success and your ability as an organization to truly innovate are going to be determined by how you handle these particular things lower the lowering the cost of support means you can do more innovative things just an example from that organization I talked to you about with that value map that we did we're two years into our journey two and a half years now when we started we looked at that value map and we identified some areas if we look at this particular one over the 12 months in the if you look at the line that says 17 Q2 that was the previous release it was done in a very waterfall way there was no agile development it was all waterfall and as you can see they thought about things in a 12 month program and towards the end the three months before the big release there's a lot of time spent fixing defects in fact it took six teams offline for three months just fixing defects because they were identifying things just way too late in the system the following year in 2018 was when we moved the teams to an agile team and in this first six month period this is number of defects per day we hardly had any defects do you know what the CEO and CEO asked me at that about you've moved to agile and I'm not seeing any defects do you know what their concern was that the teams weren't capturing the defects they said we've got defects but the teams aren't writing them up so this is a problem we actually didn't have defects because we moved to agile the teams were picking them up and actually fixing them so there were no defects to record the defects that we started to get towards the end when we were integrating with some other parts of the system outside the teams control but as you can see the agile teams dramatically decreased the defects from what they did the real thing why this is quite critical is this was a large program so this organization has a revenue we'll call it revenue of five hundred billion dollars a year to put that in perspective Google, Amazon and Apple all about a hundred and twenty billion so this is a five hundred billion dollar organization with this initiative and moving to agile we actually saved them five to ten million dollars because the teams weren't doing all this overtime that they previously did to get this over the line we saved three months of overtime by moving to an agile way teams were much happier so a happiness index with employee satisfaction went up people that back here didn't want to join the agile team because it was scary it was going to be a change and they didn't want to do it they didn't want to leave the team after that they were really seeing that I'm getting a good life balance here I like the work and I'm doing really well and I'm not having to do all that overtime and the revenue that that particular program brought in extra was five hundred billion so that cost-saving and that employee satisfaction were really good hard metrics that we could take to our CEO and say this is what we can achieve with agile really really powerful message that organization now 18 months later there's six other programs that have moved to agile ways of working so how much should you spend on innovation this is some for us to research that come out that sort of said that in 2010 people were spending about 29% on business innovation and then when they looked at it again in 2016 2017 innovation innovation rate had gone down because the business operation rate was about the same but there was a lot of a lot of waste in the system where incremental business change was just fixing technical debt and things like that so the innovation rate when it's lower it means that you can get less things out there in the marketplace that are actually going to be useful for your customers what's it innovation rate is better is 52 better than 29 look it'll really depend on your industry and what it is 29 is about the average that most companies are doing in the industry nowadays and it depends on your risk appetite as well so your innovation rate is going to be a combination of all of those but if you're tracking your innovation rate and you're seeing that it's going down and that your operational costs and fixing and bug fixes and technical data going up that's probably a sign that things aren't going well for you in your ability to innovate this is that same organization that I was talking about we've been tracking these teams for about two years what we found with that is that leadership and culture were really those things that were going to enable you to have a good innovation rate and be able to do continuous improvement because the continuous improvement only got a little bit better as the team started to mature and as the leadership servant leadership of the organization improved it took a while for our innovation rate to get to a good spot so what can you do to improve will really increase your investment in your idea generation get your executives on board if we didn't have the executives on board for this particular program it would have been extremely difficult it was an organization or wide transformation so we did have that CEO buy in and support get that innovation culture happening dedicate actual time for innovation use outside sources of creativity if you need to bring in some inspiration or talk to other companies that have done it bring them in and they can tell you how they've done it as well develop really good understanding of what your customers are doing so you can help together solve their problems think of your customers as part of your team and partner with suppliers for new ideas if you need more inspiration because you want to lower the cost of supporting your products so you can start to to invest money in those new ideas that in the future are going to be the future of where your company is going we do hackathon days as an example of giving that time to do something innovative we make sure we make it very visible on the wall who's done it and the team can kind of choose who they work with and what problem they want to solve some of the problems they're solving are things that are tooling that will help them do their work better others are processed things they really have quite a wide a wide berth on what they can actually do on the hackathon day they submit the idea we don't restrict it and we have about 10 or 15 teams each and every cycle so every three months we do a hackathon day and people really look forward to it it's a lot of fun we are geographically dispersed across every capital in Australia but we also have our team in Manila so we actually the Manila team actually always a part of the hackathon we do it all over Webex so that everybody can actually see it it's quite inclusive and it's something the team love and they look forward to if we took it away from them I think they'd be a bit of a revolt now because they just really get a lot out of it and some of the things that they've come up with have actually really helped the teams they've come up with some really cool initiatives so in conclusion if you could change just one measure that you're doing today to make it more about the impact that you as an organization you know think about what that would be and really think about how you want to improve that so you're not just measuring your activities you want to think about what's the outcome that I've delivered and deciding what to measure is critical so if you're measuring activities and outputs that's okay as long as you're using them for the right purpose to helping guiding your team or helping you improve with your forecasting your predictability but start to think about the things that are the impacts start to measure those things as well don't just focus on the activities and outputs the things that might be relatively easier to measure at a team level start to think about those more critical things that will help you as an organization be more agile and very much it's an inspection and adaption you're running many hypothesis on these things all the time to see if what you're doing is going to actually make an impact so you need to constantly be assessing your results and Peter Drucker if you can't measure it you can't improve it so make it really visible and and what you're actually measuring why you're measuring it and the and the impact that you're expecting to see the outcome you're looking for so current value is the most important but if you've got a product that doesn't offer value it's not going to last long in the marketplace measuring your ability to innovate gives you insights that you going to need to remove those barriers remove that technical debt that time it takes you to actually get your product out there use experience as part of sustaining and improving getting engaged employees getting happy employees they're the ones that are going to help you produce better valuable products that people are going to enjoy looking at improving value requires frequent delivery of new features new value so always think about that remove the impediments and understand what they are and what you can do about them to actually help you get faster in getting to market and improving your organizational performance looking at your iterative process looking at your cyclical process to see what we can improve each and every time each and every sprint and the EBM model actually provides you that holistic view of business agility for your organization so rather than just thinking about one particular quadrant try and think of a couple of measures in each of those and help track those along the way to help you understand what's going to be good for your market and celebrate the success focus on in the improving results take time to enjoy those things take time to acknowledge them and maximize the learnings that you've got around that and really building up the knowledge in your organization to make it a truly innovative organization and that's the end thank you all the slides are in configs engine but they're also on my blog as well but I think I've got two minutes for questions maybe if there's any okay on the template there's a which the hypothesis best delivery so where you have the persona and the outcome and also the measure yeah just more interested in that and I also want to understand when you actually talk about the agile best delivery and all of that from the requirements point of view I see that it is more applicable for an epic or a capability level description of that right so as a bank I want to go completely paperless so then you can say okay it can be measured through maybe the number of customers who are still visiting the branch and things like that yeah just an example so but how can that be expressed in the form of let's say further granular level let's say user story level or something like that yeah and that's where you take that sort of first example and then you use Jeff Patton's tool to actually take it from that outcome to the user story so I in the paper I can point you to where that is but it sort of shows you how you can get that traceability from the goal all the way down to the user story and Jeff Patton's works got a lot of patterns there to use patterns patterns yeah so we'll probably discuss yeah so yeah let me give me your card and I'll forward some to you from Jeff Patton yep anything else I think we'll close cool come talk to me later bye