 Product owner. I don't have any guidance and I made a bunch of mistakes and so this is kind of therapy for me decompressing up an experience And realizing a lot of the things that I did were kind of wrong I should say since I'm representing door exteriors that my comments today do not reflect the current state of the product Which I haven't worked on for two years and are in no way representative of door exteriors So my point I'm going to be making today is this I think the dominant paradigm product management is wrong and or poorly specified Lots of people think about innovation and product development in general as if we were building trust bridges which could be easily modeled and possess linear characteristics and where most of the thinking is in creating the specification rather than in actually building products and That's correct We need to think differently about the way we measure costs and in particular do less of it and we measure value and more of that and Also the way we manage portfolios of products And in particular think differently about culture and what kind of organizational culture is required for innovation So if none of this is interesting to you now would be a good time to go and see one of the other fine talks that are going on right now So I was made a product manager ago for works in 2008 and I Had no idea how to do product management and thought works attitudes Basically giving someone a job is to say you seem like you're not wildly in Compton. Go do this and That's it So you're I thought well, you know, I've had a distinct or scrum. Let's find out what that says about product management and Again, this is no way in representative of scrum is how I understood scrum at the time Basically, what it said is why can't what are you going to build? The big list of requirements prioritize it estimate it build it and then as a product manager you have to give feedback to the team when they build the stuff you told them to build off you go and So that's what I did and it turns out It's really easy to come up with requirements. I came up with loads and loads of requirements They come up with hundreds of those things and that's fabulous. So I had this great idea about how great the product was going so valuable software what is valuable software The agile manifesto doesn't really give any guidance on this and so valuable software What is value? So was everyone at Mary Poppindix talk this morning? Yeah, most of you. Okay, so she made this point about shareholder value and shareholder value is this theory US public companies are required to maximize shareholder value legally and Shareholders have taken companies to court for failing to maximize shareholder value and one. So it's kind of a serious thing And it came from these guys Jensen and Meckling who are academics who wrote a paper on this stuff and and for some reason it went into law really quickly So the only time the US government seems to adopt academic research is when it's likely to screw things up really badly As far as I can work out and this is one of the cases where this is really true Fiduciary duty means you can get sued if you don't do it or And so forth. So as Mary said, what shareholder value is done is presided over Decrease in shareholder value a decline in the return on both on equity investment and capital investment Accompanied by an eight-fold increase in CEO compensation from 1980 to 2000 So somebody has benefited from this but it turns out not to have been the shareholders So maximizing shareholder value turns out to be a very bad way or Focusing on maximizing shareholder value turns out to be a very bad way to actually maximize shareholder value So that's unfortunate There's seats kind of over here somewhere if you'd like to sit down if you prefer to stand that's obviously Some people consider that healthy. So I'm standing So shareholder value Focusing on this is a poor way to maximize shareholder value And so companies which aim to do this or set out to do this you should not invest in those companies So let's look at a couple of companies who have a slightly different take on value And I think have said some really cool things about value. So there's a company called space X anyone heard of space X Okay, one guy here anyone heard of Elon Musk Okay, so people Elon Musk was the founder of PayPal and one of the founders of PayPal and PayPal did quite well and Elon Musk got quite rich and Elon Musk Decided you know he had all this money and he didn't want to retire because that was boring. So he invested in Well, he started up two companies one was space X and Space X's mission statement. So space X built this thing Which is the Phoenix module and this docked recently with the International space station a couple of months ago Maybe three months ago, and it's the first privately created vehicle to have docked with the International space station So he in 10 years maybe 15 years He built a rocket ship and launched a space capsule that docked with the ISS Which is pretty cool What a guy And if you look at space X's mission statement, it's this The company was founded in 2002 by Elon Musk 11 years 11 years to revolutionize space Transportation and ultimately make it possible for people to live on other planets Now that's why I call value that's that's a mission statement and I mean this isn't that this Elon Musk has toned it down for the corporate website because Elon Musk stated goal is to retire on Mars so This is this is this is someone I kind of admire So this wasn't enough for Elon Musk Elon Musk decided that you know This is gonna keep me busy, but he had some other spare time and what was he gonna do with that? So he co-founded Tesla Motors anyone have Tesla Motors. Yeah, okay So Tesla Motors mission statement is this Tesla Motors was founded in 2003 by a group of intrepid Silicon Valley engineers who set out to prove that electric vehicles could be awesome Which which again is a mission statement I can get behind So Elon Musk a really cool guy and obviously isn't focused on maximizing shareholder value But he's rather focused on doing things that are really awesome and Will probably have enormous benefits to humanity as well, which is fabulous You know his personal life is not going so well He's just had his second divorce. So obviously there have been some sacrifices in the process of doing this But you know fabulous entrepreneur So the second person I want to introduce you to today is a guy called Jack Andraker who's I think 15 years old 15 year old guy from Maryland and he just won the 2012 Intel Science Fair Prize and what he made is a diagnostic tool for pancreatic cancer After his uncle died from pancreatic cancer. Does anyone heard of this guy? Okay, so He was he was into science and after his uncle died of pancreatic cancer he decided he was going to try and do something about it and so He got access to a lab and he started experimenting with antibodies and carbon nanotubes and He created this reproducible diagnostic test for pancreatic cancer using carbon nanotubes coated with antibodies And this sensor is a hundred times more selective than existing diagnostic tests. It's a hundred and sixty eight times faster And it's 26,000 times less expensive and 400 times more sensitive than the existing tests for diet for pancreatic cancer 15 years old Messing around with carbon nanotubes one day in the lab and he does this so that's pretty cool. I think and so awesome guy and I was kind of interested to see how he came to be able to do this kind of thing at 15 years old so there's an interview with him and he talks about this and This is what he has to say His parents he says never really answered any of the questions that they had Meaning he and his siblings don't forget to figure it out for yourself. They'd say I got really into the scientific method of developing a hypothesis and testing it and getting the result and going back to do it again so Who in this room is a parent who has children? Okay, many of you so I have two little girls and obviously I want them to be fabulously successful and Run the world and so I think a lot about how I'm going to help them become Fabulous people and you know I think a lot about how I'm going to educate them this kind of thing and so this is a bit of an eye-opener His parents he said never really answered any of the questions they had going back and up yourself. That's an interesting approach to parenting and One I hadn't really considered Saw this but obviously it's been quite successful for a jacket breaker So this idea of the scientific methods Turns out to be very important in product development and I'm going to talk a lot about the scientific methods So hold this thought So going back to shareholder value Mary Poppendick demolished that there's a guy called Jack Welch who used to be CEO of GE and Jack Welch says this about shareholder value Shareholder value is the dumbest idea in the world which is a high bar It's a result not a strategy your main constituencies are your employees your customers and your products and So if we want to create value, I think these are the three things we need to focus on employees Customers and products and those are the three things I'm going to be talking about for the rest of the session so Let's think about Customers and products you might think if you want to build something valuable for your customers Maybe you should ask your customers what they want Seems like a reasonable idea that he's tried asking customers what they want. How did that work out for you? Yeah, fast a horse, right? So that's the that's a great quote There's I don't think he actually said it I tried to find out if Henry Ford actually said this and I think it's apocryphal But Henry Ford said that if you if I'd asked my customers what they wanted they would have said faster horses So anyone who's actually spoken to customers will know that this great. I'm about to show you is true This is from Orson and Bells, but lean IT Customers don't know what they want. I mean so My first job after college was building a Product for some guy who had some money during the dot-com boom of the 2000s and he decided he was going to create a mail system and It was going to involve having a picture of a desktop Which actually looked like a desktop and having a dog that would run across the desktop graphically with your mail and show it to you And that was his USP for his products and it was a terrible idea Basically because he didn't know anything about technology now. I read about technology all the time It's my job to read the Internet's and find out what's going on in technology and every day I read things that blow my mind and I think wow, that's really awesome And that's my job. So if you think about our customers, you ask them what they want. They don't know what they want They've got no idea what they want. They don't know what's possible How could they know what they want? They can't and they don't and if you ask them they come up with something make something up because they want to make you happy You know and give you an answer But you know, they won't tell you It's hopeless. What customers are really good at doing is this Customers can't tell you what they want, but they seem insistent about what they don't want. Ah, I've lost the rest of the quote This is bad. I know it off my heart. They seem insistent about they about what they don't want once you've built it and shown it to them Yeah So this is what customers are good at once you've actually built something and you show it to them then they will tell you that that's not what they want customers are fabulous at doing that and So this is what we really need To actually focus on so the paradigm for product development where we come up with a list of requirements and we actually go and build the thing and Then we show our customers to get feedback for them That turns out to be wildly efficient as a way of delivering value because it involves actually building the thing Which is the slowest possible thing you could do to actually gather feedback so There's a guy called Eric Reese. Who's heard of every Reese? Yeah, right. So, uh, the lean starts up which everyone who hasn't read it should go and read it because it's brilliant and Basically what he says is is this You need to start having a vision for how you want to change the world. What are you going to do that's going to change the world? and make Business probably business canvas not a business model something lightweight not something that's going to take you weeks You can sketch out in you know an hour or two and get feedback on and then Build the minimum viable products and what's the minimum viable products so this turns out to be a contentious turn But what it is essentially is the least possible work you can possibly do to get feedback on your idea and That might be there's a great story from IBM when IBM invented the computer Well when they invented the commercial computer, I should say because there were academic computers When they mention the commercial computer what they did is they they had all this stuff And they had no idea what to do with it and so they thought some smart executive thought well Maybe you could use it for taking dictation that would be fabulous and He decided to test this idea and the way they tested this idea and this was in the 60s or something is Instead of actually building the software, which would have been very expensive they got an executive room with a microphone and a screen and in another room they got a Secretary with a keyboard and loudspeaker and They gave the exact to the microphone and said please dictate your letter and the executive started dictating the letter and saying dear sir, I Would like to Hang on. No, don't type on. No stop typing. No, don't type. No stop It quickly became clear that this was not going to work and it was a rubbish idea and here We are in 2013 and you know, maybe some of you have Siri and Siri still is not fabulous at doing this kind of thing. So MVP running an experiment the preferably doesn't involve building any software to try and gather feedback on whether the idea is any good So what's the smallest possible amount of work? We can do together feedback on the idea without building software and then do that as fast as possible and Keep going around that loop and when you reach a local maximum because iterative incremental development will get you somewhere But it may not get you to the stage where you think, okay This is good locally, but I need to change my business idea and come up with a different strategy to achieve my vision When you reach your local maximum, you may need to pivot and change your business idea change your strategy So he has Eric Reese's invention basically is this thing the bill measure learning you have an idea You build something you get feedback You learn what to do next and what we need to do in terms of project management for the project managers in the room We need to optimize our project management process for time around this cycle lead time and This actually changes project management in some very important ways. So in particular Managers driven by the financial people tend to optimize for utilization for everyone being busy So there is a school of management, which is look and see if everyone's busy And if everyone isn't busy then find out why and make those people busy It turns out you can prove from Q theory that if you optimize for utilization That's inversely proportional to lead time So if you optimize for 100% utilization you create the most inefficient process you possibly can in terms of lead time And you can see this by imagining checking in turn for a flight to an airport If you have 10 checking desks and there's people at all those checking desks you will get checked in quite quickly But those people are there all day. They are not 100% utilized. They have very low utilization across the day If you optimize for utilization, you might have one person behind the checking desk And that one person will be 100% utilized, but it's going to take you a very long time to get checked in So this is what happens in product development If you optimize for utilization, your product backlog will never ever I mean, you'll create the slowest process you possibly can by actually getting stuff shipped to your customers You need to optimize for lead time to learn as quickly as possible for your customers And that requires having slack in your system and not optimizing for utilization And if all the people developing your products are working overtime all the time There's something very badly wrong Because not only, I mean, a century of industrial studies show that working overtime makes you stupid And you've also created a very inefficient process for actually delivering value to your customers So it's important to consider that. This actually has huge implications for the way we do product development So this might sound very interesting and like a radical new idea Or it may sound like a load of crap And often that is the reaction to every research Like, well, this is all very theoretically interesting and you're a Silicon Valley entrepreneur But we actually have real customers and a large company and we need to do real work So if you could just go back to Silicon Valley and carry on writing books that will be fabulous and we'll carry on doing the real work So has anyone been to Barcelona and seen the Sagrada Familia, this church? Anyone? Okay, a couple of you So there's a guy called Gaudi who was an architect in Barcelona And he was a quite maladjusted child He was very nervous and he spent a lot of time in nature looking at flowers And then he became an architect, which seems to be a popular profession for maladjusted people And he ended up making a lot of changes to architecture So at the time when Gaudi became an architect, the popular style was perpendicular So a lot of things, churches and so forth that were built used a style called Gothic perpendicular And one of the things that Gaudi did is he invented new styles for architecture So instead of having things with right angles, right angles are very popular in Europe for a long time He came up with a parabolic style where he used parabolic structures And then he came up also with a hyperbolic design style where he used hyperbolic structures And this church, which is still not finished over 100 years after a work started on it Is a great example of either hyperbolic style and also just being incredibly Rococo aesthetically I mean, aesthetically it's amazing and it's totally nuts And if you go and visit and look inside it, you're like, wow, that's amazing And slightly gross, but incredible I mean, it's really a fabulous building But he used a completely different style, this hyperbolic style And architects and engineers will tell you that if you use a completely different style For building load bearing structures that millions of people will be inside, that is risky Because load bearing structures that collapse kill people and that's bad So when you change architectural style, it's incumbent on you to do a bunch of testing And so when you look at what Gaudi did He didn't create a large specification document And spend a lot of time on the specifications and then build it according to the specs This is not how he built the Sagrada Familia Instead, he spent a lot of time experimenting If you go down into the crypt, you can see he was always creating scale models And building things with plaster to try and test out the ideas he had And the hyperbolic style, this is not the first building he built with the hyperbolic style He built a small church using the hyperbolic style to test that it would actually work And before that he was building a load of models And there's this thing in the crypt that you can see He actually created an inverse model of his first church And these little sandbags represent the loads on the structure but using weights instead inversely So this is how he tested that the structure would support the loads that were put on it Which is really ingenious And what a smart guy to come up with this idea of testing the loads on the structure To make sure they would actually bear the weight So he was forever coming up with smart ways to test and build scale models And test and build things before he actually went and built the church So if you're building a trust bridge which has known characteristics and it's linear And we've built lots of trust bridges before And it's designed in such a way that it can be modeled analytically Then yes, you can build a requirements document and then go and execute it And be pretty sure that it'll come in on budget, on time And with the expected quality and that it won't collapse and kill people But if you're doing something that's a bit different, that's innovative Then you need to be thinking a lot about testing and getting feedback as part of the process Building the specifications, coming up with the ideas turns out not to be the constraint The constraint is actually working out if it will really work And if people will actually derive value from it So people who say, well, we need to make software development more like engineering What do you mean by engineering? Do you mean building trust bridges or do you mean building something like this? Engineers and architects who are building things that are a bit different Do a lot of testing and are very highly iterative about what they're doing And they know that the hard work is in actually building things and trying them out and iterating and experimenting Not only coming up with ideas for what to build and specifying them So a great example of a company which seems to just not iterate at all And launch things with extreme fanfare, apparently without doing any user testing Is Apple and Steve Jobs famously says, I'm not going to ask customers what they want I'm going to build something which I think is fabulous and screw my customers So that would seem to be the anti-lean start-up approach And so anyone who's got a fumble slab, that's the apex of anti-lean start-up, you would think So does anybody know what the first product developed by Apple is? Apple's very first product The PC I mean, it was a home computer Who said Apple one? Yes, specs, the Apple one So let's have a look at the Apple one It was very revolutionary for its time Came out in, I think, in 1976 or 78, something like that It had, well, I mean, the keyboard was an additional extra You had to have your own keyboards and you had to provide your own power supply But it had a display adapter so you could plug in a monitor All the personal computers before that had switches and lights and key punches But this actually had, you know, you could plug it into a television And you could connect a keyboard to it And so it was very radical for the time But it was still, you know, I mean, you could buy it in kit form and solder it yourself Or you could buy the pre-assembled circuit board And then build your own container and attach the keyboard It was quite MVP, let's say I mean, it was designed in, I guess, the garage of Steve Wozniak's parents' house And then they kind of showed it off at the local computer club So, you know, A, B, A, B Kind of different in its approach So actually Apple do a lot of testing and user testing But they just make the people who they do the user testing with Sign some very strict contracts about the fact that they've done it And if you look at the Apple Macintosh The Apple Macintosh was really the computer that led to mass market adoption of Apple's stuff Beyond computer hobbyists And the Apple II was very successful, but the main market was computer hobbyists The Macintosh was the first computer that professionals really bought And if you look at the team that built the Macintosh There's a great site called folklore.org And folklore.org is stories from the people who built the Macintosh And there's a story by one guy, Andy Hertzfeld, who says this Instead of arguing about new software ideas We actually tried them out by writing quick prototypes Keeping the ideas that work best and discarding the others And this is the important bit We always had something running that represented our best thinking at the time And they actually designed the computer with that in mind So all the logic was built using programmable logic arrays Where you can reflash the PLA every time you want to change the way the logic works And this was really revolutionary at the time And they made it so that if they didn't like the way the logic works Or they needed to change the logic, they could just flash it Plug in the PLA, reboot the computer and try again And then if it didn't quite work, well let's debug it And then let's flash a new PLA and plug it in So they could be iterative about the hardware design As well as the software design So they designed it with iterability and incremental design in mind And if you compare the Lisa, which was being built at the same time The Lisa used a very formal water-for-lea process And they had horrible scope creep And it came to market later than the Macintosh despite starting earlier And no one's heard of the Lisa because it didn't work It worked if you turn it on and stuff But it didn't work in terms of actually being a valuable product So the great thing about these people was that they were building stuff for themselves So how did they know if they liked it? How did they know if it was valuable? It was valuable because they liked it They were their own customers And then it's quite easy to develop products if it's to please you So how many people are working in products that they are the target audience for? Okay, a few of you So lucky you, that's a great position to be in Who is building a product for customers? Who is building a product that they would never buy for themselves? Okay, a few of you I'm not saying because these are horrible products Who is building a product that's fabulous and wonderful But probably isn't high on their priority list for things to buy Let's put it that way Okay, so no one additional If you're building products for people who are not you You need to think about the people you're building products for And so who's the personas? Okay, a few of you So this is pretty much everyone I prefer, instead of dog food, I prefer drinking your own champagne Because it turns out most people don't like dog food But yes, so drinking your own champagne, gone You have a question? Okay, Google does it So before they launch a test it out in the market They have a smaller, less risky audience They test it out with their own employees Yeah Sounds like a good idea Yeah, I totally agree So Google tests it out with their own employees Yeah, I absolutely agree If you can test it on yourself And on people close by you Absolutely, I think that's really the best way to do it And that's what we did with Go We used it ourselves and we found other people who wanted to use it We focused on getting people who would use it I think that's absolutely the best way to go But the problem is if you're building something that you can't I mean champagne that is not for you Then what do you do? So yeah, step one, drink your own champagne If you can do that, fabulous If you can't do that, then you have a problem Because if I'm building something which is an extreme example A pacemaker, I'm not going to dog-feed a pacemaker Because I'll die So I need to find some way to test that product So the problem is if you can dog-feed it, that's fabulous That's the best way to proceed If you can't, because the product is not for you If you're not the target audience for that What do you then do? And so this is where personas come in This is what we all do when we start building a product Is get a copy of Vogue, at least that's my excuse And cut out pictures of potentially nice looking people from it Occasionally one for diversity reasons And then put them up and describe what those people are like And those are your target audience kind of thing Which is great There's this guy from Menlo Parker Innovations called Richard Sheridan And he has this really nice exercise for personas Where he creates a target like this And you're allowed to put one persona in the middle Two personas in the second ring Three personas in the third ring and so forth And his point is you need to have one persona Who is your main target audience And he says this is the exercise that product owners find really difficult Choosing one persona to put right in the middle Because everyone says things like We want to dominate the sector and things like this And you want to have a wide target audience And no one wants to do this exercise, it's really hard He was working with someone who was building a product for somebody else And he went through this exercise and picked up a persona And he says I can't relate to this person And Richard Sheridan says to him How many of these widgets are you buying? And the guy says none And Richard Sheridan says why should I care what you think? And the guy says that's the area of your process I'm still having trouble with So it's hard to do this and it's hard to realise To actually put yourself really in the mind of the person who's buying your products Who is not you, empathy People become technologists because they don't like talking to other people That's why we become technologists We're not normally high on the list of skills that we're good at I'm speaking personally I have a friend who's like this So this is a difficult exercise but it's really really important And to think about that one person who you're actually going to He's going to buy your product and put yourself into their mind And then you need to find some way to measure the value to that person Of the thing that you're building So how do we measure value? One who's going to buy your product and one who's going to use your product Okay so that's an excellent question The person who's going to buy your products or the person who's going to use your products And those can be two different people So for example Facebook, the people who use the products Is you know the armadmi, the general person in the population Maybe not the armadmi, maybe the two armadmi But the person who's the customer is the advertisers Two different people conflicts the interest potentially So who are you actually measuring value for? And that's an important question The flip side of that is if people don't use Facebook Then there's no revenue for the advertisers So that's a difficult problem It's one I'm not going to address Because I'm going to run out of time when I have a bunch of stuff to get through But it's something you need to think about Are you actually selling to the person who's paying So in XP there were these two terms There was the gold owner and there was the gold donor The gold donor is the person who's paying And the gold donor is the person who's giving you the requirements And if those two people are not the same it creates conflicts So you really have to think about that And you have to represent both of those interests In that case the persona exercise becomes difficult Who do you put in the centre of that? Is that the advertiser or is it the user? Now I don't know what Facebook were going to do But if it was me I would focus initially on the user Because unless you have people who are using it You cannot generate revenue If Facebook went to advertisers and said We have a social networking site with no users Will you pay us money to advertise to our zero users? They would probably say no So I think you really need to focus on the people who are using Over and above the customers And especially with social networking sites And the networking effects Unless you actually leverage that Then you can't actually find money And cynically you might say Well this seems to be the case in Silicon Valley Lots of users no actual revenue And that's a problem as well But that's a second order problem I would suggest So how do we measure value? And this is really important because measuring value Is actually how you know that you've done a good job So there's a bunch of different ways to do it A, B testing This is a technique that was developed By the people who send you junk mail Now not everyone gets the same credit cards letters Offering you credit cards with high APRs Everyone gets different ones So what you do is they make those letters better Based on who actually applied to the credit cards Well that obviously is a better offer letter And it's more compelling So it doesn't have a great history As an idea you know from junk mailers And actually if you look at Nigerian scammers So you know the Nigerian emails My uncle has died And I have five billion dollars Those emails You may have noticed over the last five years That the grammar has been getting worse and worse Because of those emails They used to be very well written Now they're horribly written The reason for that Is because they were getting too many false positives Too many people were applying And then that's expensive Because you actually have to talk to those people So what they're doing By making the grammar and the spelling worse Is they're selecting for the most gullible people Because those are the people Who they have a high chance of being able to scam That's something And A.B. testing produces surprising results Such as in order to make money We should make letters more and more incomprehensible You wouldn't rationally intuitively Come to that conclusion And yet it turns out So A.B. testing is useful Because it comes to surprising results And if you're focusing As a Nigerian scammer is On making lots of money very quickly You really need to think about How we can produce surprising results And any web-based company Uses this extensively Amazon famously uses it for everything And in fact If I was launching a product today I would probably try as hard as I could To make it a web-based product Just so I could do A.B. testing Because it's such a powerful thing For actually getting real customer feedback Because you're experimenting on your customers You're not asking them what they want You're saying Would you like this or this And they don't even know They're being experimented on Which is the best possible scenario To be back from your customers Without knowing they're being experimented on Because then you can connect Directly to their subconscious Which is where you want to be connected to You know, system A In Kahneman's methodology Sorry, system one The intuitive first kind of Part of your brain That makes those decisions Without thinking about them So the second way The problem is You actually have to build stuff To do A.B. testing So is there a cheaper way to do it? Show your prototypes to real users So Steven Gary Blank Who came up with the idea of customer development In the lean startup He has a saying Get out of the building The answers are not in the building The answers are outside Get out of the building This is really important This is something we've done At ThoughtWorks on projects You know, we go out With some bits of paper With some prototypes on them To random people In coffee shops And in the street And show them stuff And say If you wanted to do this How would you do it And we watch them actually try And they're like Oh, maybe I'd press that button And then I'd do that And you're like No, that's totally not What you're supposed to do And that's really important Because then you know That your idea is rubbish And you need to fix it Or you go out with an iPhone With some lockups And get them to try And how would you come To this goal And then they actually try And do it And watching them Do a really horrible job Of using your product Is really important And you can do this In a cheap lo-fi way You don't need to be Amazon or Google to do this stuff You can just have bits of paper So go out of the building Show things to real users And the other thing Is measuring business metrics So it's really easy To get involved in vanity metrics The number of hits on the website Instead of business metrics How much revenue Are we actually making Revenue per customer Per unit time Actually focusing on growth In real business metrics Is really important And good AV testing It's not, you know Did more people Click through from this page Rather than this page What you measure When you're doing AV testing Is how many people Actually paid us money At the end of the workflow Having come in on this page So you always need To be measuring business metrics At the end Not vanity metrics Or more people clicked On this page And then you find out They clicked on that page Because they had an erroneous Assumption about the product That they then never bought Rather than Well, less people clicked On this page But more people who Actually clicked through Ended up buying the product Uh-huh What's it called again? How do you spell it? So that's Q-U-A-L-R-O-O So that's Richman's product recommendation Amazon offers Google web analytics Which is a free AV testing framework For doing this stuff What's that one called? Silverback Silverback Okay Silverback Okay, so there's Three products That you could try out So, thank you So I just want to talk a bit About what you do Once you've got feedback So there's this example Called Votizen Which was showcased at Eric Reese's Startup Lessons Learn conference And the vision was To get people more engaged In public policy And more engaged With politicians And I'm not going to Go through what they did I'm just going to say Version one of the site They built it in six weeks With $1,206 And They used these metrics This is called the R Framework Acquisition, activation, referrals, retention, revenue So it's sometimes called Pirate metrics Because it spells R So acquisition means Someone actually came to your site Activation means That when they signed up They clicked on them On the email To actually activate their account Referrals means That they sent somebody else to your site Retention means That they came back to your site And revenue means They paid you money So version one of the site They had 5% of people Clicking through from the front page 17% of the people who did that Then activated their account But no one No referrals No retention No revenue So they iterated a bit Until they got to a local maximum Which was 70% acquisition 90% activation Fabulous 4% referrals 5% retention But no revenue And they couldn't get any further By iterating on that Yes So they came up with They came up with an idea For how the site would work And that was one And then based on AB testing customer feedback They iterated And changed the layout And changed the links But didn't change the basic The basic user workflow So the user workflow Is kind of stayed the same But they just Did AB testing To make the site more You know More interesting But the basic business model Stayed the same Between one and 1.1 So they got as far as they could With this is the model For how we're going to get Customers engaged And I think it was something Like sending No, no, no Just to one to 1.1 And then And their basic business idea I think I can't remember exactly But it's something like Sending emails To your representative Something like that And so what they did is They realized they couldn't Get any further With this particular Strategy And so they pivoted And they came up with Some completely different idea For how people would engage With the political process And again I can't remember the exact details But they had to redo The whole site Completely different Set of ideas That they built from scratch Which was version 2 And All these match Basically the net result Was they actually started Making some money But 1% was not good enough In terms of actually Achieving their business goal And so they pivoted again And all the other metrics Look really, really Like they were improving Until they got to 0% revenue And that was bad So they pivoted again And finally they got to the stage Where they were getting 11% revenue But the important thing To bear in mind from this Is that You will have to change Your strategy It's People think Well, we're going to Execute this plan And then we're going to Iterate No Your first strategic plan For how you will Get Deliver value to your customers Will be wrong And your second one Will be wrong too And your third one Will probably be wrong as well And this is something That people Don't sufficiently account for The fact that their first And second and third ideas Will be wrong Some huge proportion Of startups fail And I think the ratio Of product ideas To working products Is something like 25 to 1 So that should be sobering People should not plan For their first idea Being successful They should plan For their 25th idea Being successful Which means that You need to manage The cost of your failures And do them as fast as Possible So This has an effect on your On the way you do analysis We're used to stories Everyone Who uses agile stories As a I want Hey Who's doing that? Okay, alright Good, congratulations So That's all very well But this whole idea Of requirements That we know what we want And we must do it And then we will get this Is basically wrong We have no guarantee That we're going to Deliver that value So A guy called Jeff Goffelf Is releasing a book Next month March Still February Called Lean UX And he's come up with this idea Of hypothesis driven delivery And the idea is That we shouldn't talk about Requirements We should talk about hypotheses And so instead of The story model He says, you know We believe that Building this feature For these people Will achieve this outcome We'll know when we're Successful When we see This signal from the market Which I think is a much better Way to frame The things we build What's the signal From the market What's the measurable Customer outcome That we expect to deliver And Frame all your requirements In this way And that forces you to A, to realize that it's a guess And B, to focus on How you're going to measure The value for that And so that's Like number one thing We need to do Once we have our idea And then we need to Think about what's done So I'm keen on saying That done means released In continuous delivery That's what I say But Grockit Which is a company That Eric Rees talks about In his book He says user stories Will not consider complete Until they lead to Validated learning So released is not enough You have to have metrics That tell you Whether the idea Was a good one In the first place Before you can say You're done with the feature Which I think is a radical idea You know We talked about What's the measurable outcome You can't say you're Done with the story Until you have that Measurable outcome Until you have Validated learning On your idea About the agile development process Instead of having requirements And then radically We're only done Until we release them to customers You know Death complete isn't enough That's actually not good enough We need to be thinking About hypotheses And we're not done Until we have Real metrics That demonstrate That our hypothesis Was either proven Or disproven And I think A really important thing To consider is Will people actually Use your stuff When you're thinking That's the question Isn't that a little bit That makes it Just Nothing clear to What you expect to see And then of course You test anyway And then of course You challenge it And in our Specters And our discussions And the project We always tell the developers If you think it's crap Bring it up And fight against it Yeah So But This language Why overload it Because First I want to do something I want to build something And if I start building The experiment And say Any single sentence And any single word I believe, I believe, I believe At some point It turns around And then you Don't do anything Yeah That's a little bit So you're concerned About analysis paralysis Basically the idea Will spend so much time Thinking about things We'll never actually Do anything If it comes to this point We'll get tested And then About our stories All like this And we would have No, no Yeah Okay, so that's fair enough And I think It's like Here's the template And we're going to write All our stories like this And we did this With given when then as well All our acceptance quotes Here needs to be given when then So I'm not I think that's a very good point I'm not proposing that I'm not proposing This is the format That everyone must use Religiously And everything to be written Like that So that's absolutely a trap And we don't want to fall Into that This is An idea For how you can Frame features And the point of it Is to Huh? Yeah At the beginning of the process Of feature development Which is So you're doing it all the time But it's at The beginning of the life Of a feature And the point of it Is to focus on What are we going to measure And the fact that We don't know Whether it's going to be valuable Yeah So yeah Don't focus on the form Focus on the function Right You know the tendency is I'm lazy So I'm It's already long enough I probably don't come to the point Where we actually The most important stuff Is the technical criteria That's at least my personality And I'm a little bit afraid of I can be turned around Yeah Which is fair So I'm going to cut you short Because I'm running out of time But yeah I mean You need to think about Actually getting something done And getting the Feedback That's the most important thing And what's the shortest part Of doing that Again we're focusing on lead time How can we get feedback As fast as possible So yeah It's not This is the answer to everything And you must use this format You know It's part of the toolbox And again you need to focus on Getting that lead as fast as Possible And getting measurable feedback As fast as possible Thank you So I think You know We focus a lot On measurement and learning The most important thing is Are people actually Going to use things And the problem is As a product owner Coming back to the product Only thing I always used to think About what I wanted to build Unless about The people using it And the thing to always keep At the back of your mind Is this This is written by a writer And an author And the title Of the blog post was No one wants to read your shit Which you know As an author I think Is something to bear in mind Using a product Is above all a transaction To use it to donate Her time and attention Which is a supremely valuable commodity In return you the creator Must give something worthy Of her gift to you So this is a high level thing You always need to be thinking Thinking about The fact that You're asking people to Give their time to you Your vision of what you want to do No one cares about that People care about Being able to accomplish Their goals And focusing on How people can Accomplish their goals Is the most important High level thing you need To do as a product owner So with that in mind I'm going to revisit The product manager role So what should we be doing As product managers? Number one We need to create And communicate with the team A shared vision Of how we want to change the world It's the measurable customer Outcome we want to achieve In terms of actually Changing the world And then work with the team To define the measurements And the single business metric Which measures how successful You are as a product company Or as a product developer And then you need to run experiments To learn a few ideas at any good Which is where the hypothesis Driven delivery stuff comes in And it's all about running experiments How do we run experiments As fast as possible? Optimize our process For cycle time To get feedback from our customers And from each other As fast as possible Because that's how you learn As fast as possible And really The only way to measure Productivity Is Not a lot of code Not a number of hours worked How many experiments can we run? How fast can we learn from our customers And each other? And I'm now out of time So I'm going to end it there