 Mae'r ddaf yn siarad. Yn gyhoedd ychydig yn ei fod yn bach. Ychydig! Felly, yna gydag yna ar y teimlo yn yn 45 mwy o ddod disoedd gydag. Felly, yma'r parodydd ym Mhwyl Gysylltu'r Gwasanaeth, ac mae'n gofyn i gydag. Felly, rydych chi'n gweithio'r gysylltu'r gwasanaeth, eu bod yn gallu'r gweithio'n gwynt yn ei ddod, ac mae'n gweithio'n gweithio. Yn cydwch chi'n gweithio'r gweithio, gwahodd o'r rhan o'r hyn sydd wedi bod yn newidio'n fawr yn eich hunan, ond we don't know what we're going to invent because otherwise it wouldn't be innovation. Projects are unsuitable. Instead, what we should be doing is starting small and focusing on learning fast and changing what will undoubtedly be the bad idea we initially come up with. The most important thing to work out is how you're going to actually measure value and start doing that. For those of us working in large organisations, Mae'n siŵr i gyd yn ystrym soedd cymaint, ac mae'n 4x3. Mae'n cael fwyllgor ythryd yn gwybod. Ond ydy'r rhai ystyried sy'n ei gwybod, darwch chi'n fwyllgor y stainless? Dwi'n ein bod yn ymddindol iawn, oedd? Yn ymwysgwyd, ymwyng ymddindol ni'n meddwl yw dda'r adsod yn y 1,000 yma. Beth yw mewn 1,000 yma, gallwch yn fwyllgor. Dyna cyfans i bod nhw bydd ymwysgwyd portfalio. That's managing all the work you do at the very large scale. We should be looking at measuring value using cost of delay at the portfolio level. Finally, I'm not really going to talk about this today, but this is an advert for my talk tomorrow. Innovation is a mindset, what you need to do to innovate is to create a culture of innovation throughout the organisation starting with the leaders. So, let's talk a bit about projects. Mae'r grondau bod yn ei mewn i bobl yn ddechrau'n gwneud teimlo'n gondol o bob i'w cyd-reidio'n bach. Yn gyfans, mae'r ddechrau'n gwneud i'w wneud sefyddiol amdogiaethol o gondol, sy'n gweithio'r trofodi, a wnaill fod yn ôl i wneud inni trofodi o bach o bob i wneud i wneud eu gwneud, mae'r ddweud i eu gwneud i wneud i'w cefnoddigtol yn y gwaith yn y gallu argyllidol o unig o growl. ac ac mae gydaint yn y lyfodi y ba weithio sydd ar y cwnfrymau yn bwysig ar y nato ym wneud bydd y gallu i casf ei ddweud, ac ddweud y dyfodol y ein ysgrifennu software mae'n fwyaf ar y nato cwnfrymau o hynny o'r lle cymrydd, yw sy'n ddweud yn bwysig o'r ll perchwy phaswe bwysig ar y nato, ac mae'r lle yn bwysig ar y nato. Yn y cwnfrymiau yn bwysig ar y nato, a'r lle yn bwysig ar y cwnfrymiau yr aethlach gyda cyfroedd cyfroedd cyfroeddiant, oedd y gallu ff Waydog, sydd wedi cael ei wneud i'r oes i'r gweithio i gyfryddio'r branches, oedd y gallwn y ddweud i gweithio'i gweld felly llawer o dda i gweithio'r gweithio yn y dal i'r gwiriau pan fydd honno. Ac oedd ymwneud, yna bod rhai gweithio'r gweithio, mae rhai gweithio'r gweithio'r gweithio, i gael bod yna'r llawd yn unig i'r gynhwys yn ymweld. Mae'n ffordd y meddwl sy'n dod i'r gynhyrch gwybod yn fawr cysylltu i gyd-gwybod yma gyda'r meddwl. Rwy'n meddwl sy'n dod i'r gynhyrch gwybod. Mae'r cysylltu i'r cysylltu i gyd-gwybod. Mae'r cysylltu i'r gwybod, oedd ydych chi'n gweld y bwysig i gael gyda'r projectol sydd yn symud o gyfer y cychwyn. Mae'r cysylltu i gyd-gwybod, Oherwydd'r bydd yn cyfw皇 o mewn cyfwüg, bo contentsyти iddyn nhw'n arweud yn gweithio'r pethau cyfan. The software project can start delivering value long before they are complete in the sense of the entire set of specifications that we have in a typical project are actually completed. So for the three simple reasons, projects are very unseasible for delivering software innovation. I'm just going to explore briefly some of the consequences of that. We actually saw, this is something that Martin talked about in his keynote yesterday, we saw in the 90s there was some discussion about whether or not the typical project waterfall method was the right way to do things and of course we all know what happened, we had the agile manifesto and we decided there was a better way to deliver innovative software-based products. But what we see today is typically a process that I like to call water-scrum fall and water-scrum fall is pretty much what happens at the enterprise level. We have the water bit which Don Rynison calls the fuzzy front end where basically someone comes off the golf course and decides they have a fabulous new idea for a product and we go through a study phase and we get a business case and then budget approval and then somebody sits in a room and spends several weeks drawing large requirements documents and breaking them up into little bits and then we estimate those things and do some further work on the budgeting and finally at the end of this we actually start working on software and if you're lucky there's this nice scrum thing or some other iterative method where we're delivering in nice iterations but actually we're not really delivering because none of this stuff is in fact going into the hands of users and what happens after this is we have integration and the testing phase and in enterprises who works in an organisation where the software is tested by a different organisation or by people in a completely different place. Okay fabulous so about a third of you and then the whole thing actually has to be released to users and if you're working on websites or software as a service or something like this typically that's a pretty unpleasant time for the operations people we have to try and make the thing that works on people's development machines actually work in real life with real sized datasets and realistic production hardware and real user load so that bit from dev completes to released in production, live in production is called the last mile and my experience has been that it doesn't matter how agile the bit in the middle is you can make that as agile as you want but it makes no difference to the business outcomes and the customer outcomes if you leave the other bits the way they are. Two fun facts which should help explain that particularly about the fuzzy front end. Number one we spend about 50% of project lead time in the fuzzy front end. What's the main activity we're doing in the fuzzy front end? A big part of it is estimation and the reason we do estimation is to work out costs to work out how much money we're going to have to spend to do this. There's a great book called How to Measure Anything by Douglas Hubbard and he wrote an article for CIO magazine about 10 years ago which had this really memorable statement. Even in projects with very uncertain development costs we haven't found that those costs have a significant information value for the investment decision. So in terms of actually whether or not we should spend the money the development costs actually aren't that important. The important variables that have a high information value are whether the project will be cancelled and how fast people will start using it and whether indeed they'll start using it at all. Those are the things that have high information value for an investment decision and we typically don't measure those and there's many projects that I've seen where we don't even measure the return on investment at the end. So in terms of actual kind of fiduciary duty and doing the right thing in terms of our investment this is really really sadden depressing and my colleague Ross Pettit says that in many cases we'd be better off taking the money we're going to invest in software development and putting it in unit trusts instead because it would have a more reliable return on investment. So let's return to the agile manifesto. The agile manifesto says that our highest priority the first principle our highest priority is to satisfy the customer through early and continuous delivery of valuable software. So apart from being a subliminal advert for my book this poses an important question what is valuable software and the first thing of course is that value is something that belongs within objects value is value for someone so we need to start by thinking about who is our software going to be valuable for who are our customers going to be. Now I was once a product owner and when I became a product owner I basically you know read the stuff about Scrum and thought that Scrum would tell me what to do and Scrum was pretty much useless at telling me what to do as a product owner it told me that I should come up with a bunch of stories and prioritise and estimate those things and I did loads of that I was really good at coming up with stories and I could prioritise them and estimate them when I spent ages doing that well not the estimation the team did the estimation so I just to be clear but I can have got them ready and helped the team to estimate them and we did lots and lots of that but I still to this day I'm not really sure about the extent to which we actually delivered valuable software other than you know in terms of our kind of overall bottom line which is a leading indicator that's not very useful at telling you which of your features actually valuable to your software and just by you know hearing people say nice things or otherwise about what we did. So after kind of a therapy after I finished being a product owner as a way to kind of recover from the trauma of that job I started to look into what value was so there's this concept of shareholder value. Shareholder value is very important in the US and other capitalist countries the directors of a public corporation in fact have what's called a fiduciary duty to maximise profits that means legally they're supposed to act in such a way that they maximise profits for their shareholders. However research into competitive performance of organisations determines that the companies that placed the highest priority on profits were universally less profitable than the firms that didn't so the more you focus on this being your guiding principle the less profitable you end up being. So it's actually a really poor strategy for in fact delivering customer value to shareholders and in fact the shareholder value model has presided over a decline in the rate of return on equity investment and capital. However there has been someone who benefits from this model there's been an eightfold increase in CEO compensation from 1980 to 2000 as a result of people who follow this model so somebody's winning but it's not the shareholders. So let's look at some companies which you know do some interesting things around delivering value and look at some of what the foremost entrepreneurs of our time think about value so in my opinion one of the foremost entrepreneurs well let's start with Jack Welch Jack Welch was CEO of GE and he had this fabulous quote shareholder value is the dumbest idea in the world it's a result not a strategy your main constituencies are your employees your customers and your products and this as with many things Jack Welch said is is pretty much spot on I think my number two entrepreneur of our times is a guy called Elon Musk who said of Elon Musk okay so Elon Musk founded PayPal and when PayPal got sold to eBay he had a ton of money and he decided that what he wants to do with this is send rockets to Mars so he founded a company called SpaceX SpaceX was the first private company to send a vehicle to the ISS this is that vehicle his mission of space for SpaceX looks like this the company was founded in 2002 by Elon Musk to revolutionise space transportation and ultimately make it possible for people to live on other planets now that's what I call a mission this is a man whose stated goal is to retire on Mars and he's pretty serious about it I mean from 2002 it took them 11 years to go from nothing to this vehicle which was the first private vehicle to dock with the International Space Station so that's pretty cool but we don't all have to be Elon Musk's in order to be entrepreneurs and to innovate so I want to I want to introduce you to this guy Jack Andreco who was the winner of the 2012 Intel science fair he created a diagnostic tool for pancreatic cancer after his uncle died of pancreatic cancer and he basically did this by researching things on Google and finding a local lab that would let him kind of play around and he researched things like carbon nanotubes and various proteins and how to detect those proteins and he came up with a way to detect a protein commonly used as a biomarker for pancreatic cancer using carbon nanotubes coated with antibodies that was 100 times more selective than existing diagnostic tests 168 times faster 26,000 times less expensive and 400 times more sensitive so this is a 16 year old kid who was playing around with carbon nanotubes and various proteins and bits and pieces I mean that that's pretty amazing and you know quite rightly he won this this prize so I have two kids I'm pretty interested in how I can make my kids grow up to be like this guy and so I was reading a bunch of articles about how he ended up doing this stuff and I came across this this quote which was fabulous you know I'm like well what did his parents do so his parents he says never really answered any of the questions that they had him and his brothers and sisters go figure it out for yourself they would say I got really into the scientific method of developing a hypothesis and testing it and getting a result and going back to do this again so that's a pretty interesting parenting method you know whenever your kids ask you a question just say go out work it out for yourselves I think that's something I can I can probably manage in fact um but I think this thing is really interesting this idea that it's the scientific method of developing hypotheses and testing and getting a result um and actually you know Jack Andraica was one person who was pretty much the self invented uh or self developed entrepreneur um the the chip that's in all the iphone phones and many other phones is called an arm chip the the arm cpu is developed by two people in 18 person months uh by these two people in Cambridge and the guy who was in charge of the company which is called acorn computers uh he said well I gave these two people something that no other company gave them I gave them no money and no people they had to keep it simple I think resource scarcity is actually really important to innovation one of the ways we kill innovation is by investing a ton of money and spinning up a really big team and imagining that somehow that will enable us to finish more quickly and in fact the opposite is normally true when you have a large number of people uh and you grow a team very very rapidly what happens is people start doing things and it all comes a bit uncontrolled and the cost of communication massively overwhelms our ability to actually control the growth of what's happening and innovate in this way and instead we need to start small and this is pretty much what the lean startup is about so he's familiar with the lean startup uh hopefully most of you there's a ton of talks about the lean startup of this conference which can only be a good thing but basically the lean startup is just the scientific method um you know there's these fancy diagrams that you see everywhere and the kind of trendy colors of modern times you know the kind of line green and blue which indicates that you don't believe the diagram um and you know the bill of measure learn loop that most people I'm sure are familiar with but really it's just you know a method that's of years old called the hypothesis um and we designed an experiment to test the hypothesis and that experiment is called the minimum viable product and we get feedback and we repeat the thing is people working in enterprises they see that and um you know this idea that we should start small and grow slowly and not do any of this stuff on financial management because that's kind of pointless and certainly not spend months and months working out in detail what the requirements are and trying to get budget and they often say well that's all very well but it basically sounds like a load of crap so I went on holiday the first holiday that I had after my kids were born without my kids and in fact the only one to date that I've had without my kids me and my wife went to Barcelona for a weekend and we saw this church called the Sagrada Familia so anyone been to the Sagrada Familia? Okay I mean it's beautiful um it's designed um by a guy called Gaudi um and it's been under construction for over 150 years and it's still not finished yet I mean it's really a stunning piece of work um and Gaudi actually had a quote you know people would ask him well it's taken me a long time to build and he would say my client is not in the hurry and Gaudi actually invented a number of new architectural techniques to build this church um a lot of people around the time this was built most churches and large buildings were built using a perpendicular style so people were very keen on corners um you know they've kind of slightly wacky people would use arches uh which of course were invented in the um in the Islamic world uh a bit earlier um but in general it was straight lines and arches if you wanted to you know push the boundaries a bit and Gaudi was basically a bit of an awkward child and he would go and play in nature a lot and look at how natural things plants and so forth were constructed and he found these kind of hyperbolic patterns um and he started playing with the idea of parabolic structures and hyperbolic structures um and in fact this is built there's lots of hyperbulas in this uh in this church if you go inside and look it's modeled on how natural um living things um grow the problem is he didn't want to just build it in this way because it's a bit risky when you build something with a completely new architectural style in case it falls down and kills everyone and you really don't want that in a large building um and so what he did actually was experiment with um first of all scale models if you go down into the crypt of the church um what you can see is this is a uh an upside down model for an earlier smaller church he built in the hyperbolic style and what he did is he built this model upside down and hung these little weights to simulate the loads on the structure and used this as kind of a low tech way to simulate what the loads on the structure were to try and make sure that the thing would actually stand up so he had all these techniques low cost experiments to try and determine if the basic premise would actually work and deliver the expected value i.e staying up and he built lots of little scale models and then he built a smaller church in this style long before he built the Sagrada Familia so this is just one example of where when you're doing something innovative i mean if you're building a trust bridge a trust bridge has known um engineering qualities and there's models that you can build that pretty much predict uh how the thing should be built in such a way that it will stay up and you just punch the numbers into the model and outcomes you know what the structure should look like and what the material should be uh when you're innovating in architecture you actually need to use these small experimental techniques in order to work out whether the thing you're going to build is is going to stay up so this idea that even for civil engineering projects we shouldn't be doing innovation uh we shouldn't be doing small experiments to test things before we hire tons of people to actually build it um that's not really true and you can see that in in this fabulous experiment that he did the other example that people who are kind of technologically savvy like to give about why the whole kind of experimental approach is kind of crap is the iphone so this doesn't look much like a minimum viable product i mean when it emerged it emerged fully formed at uh the wwdc uh conference um i can't remember which one it was um but there was really no inkling that this thing would come out it looked like it came out pretty much fully formed who knows what apple's first product was anyone huh the lisa no there was there was a couple before the lisa it was that anyone mouse no the mouse was actually designed as part of the macintosh the first computer apple built was called the apple one and it was built in steve wasniac shed in palo Alto and it looks a bit like this this looks much more like a minimum viable product and actually the keyboard in the case were not supplied you just got the logic board and you had to get your own power supply and keyboard and so forth to actually make the thing work um and then if you look at the macintosh which was really the the product that took them out of kind of geeks into the mainstream of design the apple macintosh was actually built as a reaction to the lisa so the lisa was built in a very traditional project manner where they had an enormous team of people and it was very expensive to develop and steve job's got very frustrated with this process and left to create something which he actually was more passionate about which is the mac and the mac team was much smaller and they worked in a very different way and this is a quote from one of the people who worked on the macintosh he says instead of arguing about new software ideas we actually tried them out by writing quick prototypes keeping the ideas that work best and discarding the others we always had something running that represented our best thinking at the time and this idea that we should always try something out and then integrate it all together and and see what happens and see if we like it that that I think is very central to this idea of tinkering that's crucial to effective innovation so I want to move on to this idea of measuring value and how do we measure value there's actually a number of different ways to measure value ab testing is is really powerful and I'm going to talk about that in just a minute there's very simple ways to do it as well just showing people your prototype and seeing if they're interested in it and crucially if they'll actually pay money for it is a very powerful way to measure value without actually building something and the one of the problems with projects is the way we test whether or not our initial idea is valuable is we actually build the whole thing and then find out if it's valuable that's the most inefficient way we can possibly test if the thing we're building is valuable and there's much cheaper ways to do it like showing people prototypes and seeing if they'll pay for them measuring business metrics my favorite quote on value is from Donald Reinerson he says the way the world tells you whether what you're doing is valuable is whether they send you money and you should try and find that out quite early on rather than at the end and really you know we have this word requirements I have a problem with the word requirements because whose requirements are they are they are users requirements users don't know what they want users know what they don't want once you've built it for them but they don't know what they want the requirements are actually the requirements of what's called the hippo the highly paid person's opinion and the person who's the person who plays lots of golf and generally decides what the requirement should be and I think if we're honest we shouldn't be talking about requirements we should be talking instead about hypotheses there's this idea that a bunch of people in the linear x community have come up with of hypothesis driven delivery we believe instead of stories I mean everyone's familiar with stories white as a hmm I want hmm so that yay right we're familiar with the story format everyone written stories or use stories okay so maybe instead of that we should think about hypotheses we believe that building this feature for these people will achieve this outcome we'll know we're successful when we see this signal from the market what's the signal from the market we're looking for that will actually demonstrate that the thing we plan to build is actually valuable and can we find ways to measure that more cheaply and quickly than actually building the thing so I want to show you what a company called Etsy and does Etsy does something called ab testing and ab testing wasn't pioneered by Etsy it was actually pioneered by the people who send you junk mail they send you different versions of junk mail they send different versions of the junk mail to different people and work out which version produces the highest response rate and this idea was copied basically by software people who thought well instead of showing you just one version of a website we'll show lots of slightly different versions and see which ones actually end up with people spending the most money or whatever you care about in terms of your goals and organisation and so when Etsy develop a new feature they don't build the whole feature out and then put it into production what they do is find a way to build an experimental version of the feature which is much cheaper so we don't worry about making sure that it scales we don't worry about covering all the corner cases we find cheap hacky ways to achieve the outcome without building the nice elegant code that you would build for something that you actually really wanted to go into production and they just switch it on on the website for a small percentage of users so this is a feature that Etsy run an experiment on says show similar items link on unavailable items in the car this actually was a feature where so Etsy is an online website that people an online website of course it's online it's a website sorry it's a website where people in the US who who make handicrafts can sell the handicrafts to the general public so anyone who has relatives in the US it's a great place to buy presents for them so this was an experiment where if you searched for a particular item and it wasn't in the person's shop who you were looking at it would show you items from somebody else's storefront instead and they turned this on for a small percentage of users first and they built this fabulous tool that shows you what the impact is on the business metrics that you care about so blue is the control group who sees the website without the experiment turned on and green is the group of people who sees the website with the experiment turned on which is a small percentage of users and what they track is the business metrics they care about the number of people who visit the car the number of people who bounce off the site the number of pages that you get to see and the number of visits that end up with something being added to the car and what you can see here that there's very simple statistical model you can use to get this data it's called the student team model and what they look for is a 90% confidence interval in the statistics and you can see here that this grayed out number means that we haven't reached statistical significance for this particular measure yet but we have reached statistical significance for page count and the improvement with the experiment turned on is 0.26% in the number of pages people visit on the site and that's pretty average for an ab test a really good ab test might give you a boost of a two or three percent anything bigger than that is really surprising and it probably means that your experiment was badly designed and you did something wrong so that's the first thing to test now as a product owner this is like crack having this data it's amazing it's the coolest thing ever to be able to actually see a causal relationship between an experiment and the top line metrics we care about and that's the great thing about ab test it's not just a correlation it shows you causation this feature caused these improvements in business metrics and you can get this data in a few days i mean and it doesn't take long to build the experimental feature because it's very very cut down and simple so it only takes you a few days to build and then you can get the data before you invest the effort in actually building out the whole thing so before they actually you know spin something up to really build out the feature they get this data that proves the impact on top line metrics and you don't need a lot of data actually to reduce uncertainty on what the value will be because when the uncertainty is very big the amount of data you need to reduce it is very small so this is a very effective way to actually measure value before you go out and build out features um and they actually have a number of experiments running in production at any one time to try and gather this data so there's many you could end up there's effectively many different versions of etsy and depending on which buckets you fall into for the various experiments you could see any combination of these experiments turned on or off uh bing which uses the same technique of ab testing there's about i think 10 to the 5 different versions of bing because they have hundreds of experiments going on at any one time that you could possibly see now experimental design is hard working out how to design experiments make sure they don't interact for the monitoring in place to actually detect if there's interaction or if there's regressions in other parts of the site that's really hard and it's something that actually is probably the most important thing you could possibly learn about product management in my opinion but it's something that i mean there's no talks on it at most conferences it's something there's hardly any books on or literature on for me that's a real i mean unexplored territory in product management it's how we actually do experimental design and it's something i'd like to see people looking a lot more at the guy who developed the ab testing framework for amazon and then went on to work at bing guy called Ronnie Cahavi gathered a lot of data from doing ab tests and there's a slide which i showed in martin's keynote but for those of you who weren't a martin's keynote i just want to show you again and re-emphasize this because it's pretty shocking evaluating well designed and executed experiments that were designed to improve a key metric only about one third were successful at improving the key metric so these kinds of experiments that we run most of them demonstrate that the experiment delivers zero or negative value to our customers what that means is if you generalize this that means that two-thirds of the features that we build delivers zero or negative value to our customers we could be spending two-thirds of our time at home or on vacation in goa on the beach and deliver the same value to our customers if only we knew the two-thirds of the features we build that have zero or negative value that's a pretty shocking statistic another way to quantify value at the portfolio level is to use what's called cost of delay cost of delay is a system that was developed or popularized i should say by don rinison where you basically think about how much it costs per unit of time say per week to not build the feature and there was a fabulous paper that came out last year at the agile conference by drosh arnoldon oslem yw chyr and they did some work at mesk which is the world's biggest shipping company on actually taking a number of requirements they actually had 3000 requirements in the backlog at mesk and they looked at all these different requirements and put together a team to actually put a dollar value on cost of delay for each of those features so how much would it cost the company per month to not deliver that feature and it took them a while to work out how to do it but they got a bunch of people and got a bunch of estimates for that and they plotted them on the graph and what they worked out was that there was three features in the backlog that had a cost of delay of more than one million dollars per week so by not delivering those features it was costing the company you know two million dollars for this one about 2.3 million dollars for this one and 2.7 million dollars for this one and then all the other thousands of features in the backlog will write down here somewhere unless you actually go through this exercise and it doesn't need to be very expensive because again we don't need much information to reduce uncertainty unless you're doing this exercise and actually looking at the value that your features deliver and thinking about it and trying to gather small amounts of data about it the prioritisation exercise is just going to be guided by whoever shouts the loudest and by politics and actually thought I did a survey of how people measure value the sea level and we actually worked the data showed that most people actually use that method and to see if I can quickly get the graph that actually because I think it's worth looking at yeah here we yeah here we go so this was the survey that we did please select the statement the most closely aligns with how your company decides which products are built so the big blue 47 percent is decision by committee the yellow one is financial modelling which is actually using kind of real data and thinking about things from a financial point of view 13 percent of people the opinion of the person with the highest salary wins out this is this is the hippo the highly paid person's opinion so that Ronnie Cahavi the guy who did the ab testing framework for amazon actually has little rubber hippos that he gives out to people so you know say you're the hippo here have a hippo but 13 percent this is this is large companies these are 14 500 companies 161 business decision makers said the 13 percent of them said the opinion of the person with the highest salary wins out and then this blue 7 percent here is no systematic approach and light gray is kind of product portfolio approach so 76 percent of the company's surveyed basically weren't using any kind of economic model for doing prioritization of products so the good news is there's a lot of room for improvement although it's somewhat depressing that we're in the situation in 2014 I want to end just by reiterating the takeaways when we spin up projects and spend all this time measuring cost and coming up with requirements and that takes by the way about 50 percent of our lead time to gather all this data which isn't actually that important for the investment decision that's a really great way to kill innovation and products in general are unsuitable for building products in conditions of uncertainty instead what we should do is use resource constraint and the experimental experimental methods to learn rapidly about whether what we plan to do will deliver value and in the normal case overwhelmingly they won't to work out how to pivot rapidly and come up with something that will deliver value for our customers the most important thing you can decide at the beginning of building a product is how you're going to measure the value for that product is supposed to deliver and trying to find cheap ways to gather that data as quickly as possible if you're working in an enterprise the most important thing you can do to improve the outcomes for your enterprise is actually come up with an economic model for deciding how you're going to measure the value of things and using that to make prioritization decisions and a great model for doing that is cost of delay on rhinocens model and then finally again an advert tomorrow for tomorrow innovation is something that needs to be pervasive throughout the company it's not something you can build in afterwards it's something that has to be there from the leadership to the people on the ground and the most important job of leaders and managers is to create conditions in which the people can do doing the work can work out how to measure value for themselves so we've got about seven minutes left what questions do you have yes so is it important for the development team to know how the product backlog is prioritized I actually would go further than that I think it's important for the product for the development team to actually be designing the experiments the job of leadership and management is to say what outcomes they want you know what metrics they're actually interested in and then is the job of the development team to actually come up with the experiments to try out different ideas in order to drive those metrics so you know this idea that you know deciding what features should be built is an upstream process that the developers aren't involved in I think is fundamentally flawed I think developers and analysts and product owners and UX people should be working together to design the experiments because actually experimental design is something that you need to understand software development in order to be able to do the idea that you can design experiments you can just have someone whose job is experimental design I mean you need people who understand about you know statistics and data but you also need people who understand about how to build experiments cheaply in other words from a development perspective what's going to be cheap to build so development teams are instrumental in actually designing the experiments and that again is very different from the traditional ideas about how we should do software development and even most agile ideas about how we should do software development which is the developers are passive receivers of stuff that other people have dreamed up I mean that's in my opinion fundamentally flawed yes how to measure the value of learning from failure well I really recommend reading how to measure anything because how to measure anything talks about I mean you you can estimate the the value of something and you'll normally get a range the value could be somewhere between here and here and in that in that book Hubbard talks about actually how you do estimation in this way and you get a range but then there's actually a mathematical way to calculate the reduction in uncertainty I mean this is what a measurement is defined as a measurement is defined as a reduction in uncertainty and so if you calculate the cost of the measurement and the amount of and the reduction in uncertainty you can actually calculate how much money you should invest or the upper limit to the amount of money you should invest in it running an experiment to reduce your estimation of uncertainty so using this kind of simple statistical and economic model you you can actually put real numbers on that data I'm doing kind of a poor job of explaining it because I haven't got it in front of me but it's all there in in how to measure anything I've actually got up here somewhere this should be much easier if I had this in front of me I'm not going to be able to get it up in time but I recommend how to measure anything because it talks about actually how much you can see how much money you should invest in the experiments and get the upper limit based on the reduction in uncertainty it's actually possible to calculate that mathematically yes so the question is how does strategy change the dynamics of the team greedy strategy I've still don't really understand the question yeah so I mean the strategy the role of strategy is basically to say what your your goal as a company is to make sure that everyone is aligned so um I mean that the strategy is something like this you know our strategy is to revolutionize space transport and make it possible for people to live on other planets so obviously you don't want things in your portfolio that don't contribute to that goal right but say you have a number of different things that contribute to that goal you need to work out how you're going to measure the value of those things in terms of contribution to that goal so yeah I mean you've got to have some end goal that you're aiming towards strategically and then you have a bunch of different ideas as to things that could move towards that goal and then you've got to prioritize them and you've got to work out the value and the value is going to be how we you know how far we move towards this goal so there's definitely an interaction there it's not that leadership is blind and it's just about numbers there's got to be something driving this in the first place absolutely no absolutely not value is not about dollars uh just about dollars um it's you can measure value in multiple different ways I mean for a non-profit um it might be I mean if it's vaccinations it might be something like how many people we can vaccinate in a certain period of time that might be this is called the overall evaluation criteria you need I mean that that's the role of strategy to divide the oh devise one of the goals of strategy is to devise the overall evaluation criteria by which you actually prioritize things um you know for I think for army logistics on rhinocene says it was cost per tonn mile that's why you're optimizing for so that's why I say you know at the beginning my third takeaway is workout deciding how you're going to measure value in what we actually going to use as our overall evaluation criteria in order to work out what to do okay has there been any more time two minutes one more question okay thanks very much everyone