 You will find today that we do not have feedback forms for each of the sessions. We don't have, you know, paper forms in the rooms. It's not that we don't want feedback. It's just that we found that the paper forms aren't actually as effective as we'd like them to be. So I tell you what, if you are on the core team for today's theme, if you're one of the Agile Lifecycle core team members, can you stand up? That would be you. Okay. Stay standing, sorry. If you were involved in actually helping to select the sessions outside of core team members, can you stand up? Okay. All right. So we will have folks in each of the rooms. You can approach any of them. Also take a look at these folks here. You can approach them. And I will show you a slide deck with some of our other organizers on it. Any of them are happy to take your feedback. And we will, you know, we'll incorporate all of that. Okay. Thanks. Are we set? Without any further ado, I don't think there's a whole lot of introduction that's needed. I'm sure that most of you are very familiar with Ash and his work. As soon as we get up here, you know, a leader in the lean and agile space has written a couple of excellent books. If you have not read them, I do recommend that you do so. And as soon as we've got this set, I will have him. I've got an adapter. Okay. All right. Ash. You're the best. All right. Good morning, everyone. How are you all doing? So I usually like to start this by getting a sense of who's in the room. So how many of you are somewhat familiar or quite familiar? It doesn't matter with lean startup or running lean. Okay. So about, let's say about a third of you. That's good. So what I'm going to do today is really talk about, introduce some of the running lean concepts. I know a lot of you have agile backgrounds. So we'll kind of tie the two things together and see how one kind of stops at where the other one starts. But by way of introduction, I have kind of some books that I have written and I've written one book and I'm writing a second book, but I often like to introduce myself first and foremost as a practicing entrepreneur. So I build products every day. I do come from a technical background. I've written code for as long as I know and I still try to sneak it every now and then. It's harder these days, but that's really what I do. So I like to start with the sad reality. So this is the unfortunate statistics is that most products fail in general. So a lot of the lean startup really came out of the startup world, but we find the same level of uncertainty, the same level of failure rates in really any new product introduction. So this is not just limited or restricted to startups. Now there was a study that was done in Silicon Valley that looked at startups that did fail. And what they found is that the number one cause for failure was not the lack of the ability of talent or skill or the ability to build the product. All these companies were well funded. They had a business plan. They had a spec to go execute on and they all build a product to spec, yet they still failed. And so that number one reason for failure was not because they failed to build a product they set out to build, but because they wasted needless time, money and effort just building the wrong product entirely. They build a wrong product because they failed to find the right customers and markets for those products. And so if we kind of take a step back and see why this happens, the number one reason that I attribute for this failure is that we as entrepreneurs, as people building products, generally fall in love with the resolution. So when we get hit by an idea, we get this single clear idea that comes in our heads and this is a pyramid that I'm going to kind of build upon at the very slowest layer in that idea is a vision, something we want to go do, something we want to go achieve. Then there's the strategy of how we go out and roll out this particular product or feature and at the very top we have the product. The mistake a lot of us make is we rush up this pyramid, so we go right to the product and fall in love with the final solution and then fail to really understand what the right customers and markets are that we're essentially going after. The second reason that I attribute to this product development being hard or building the right products being hard is that product development itself gets in the way of learning. Now this one is a little counterintuitive, but if we go back to my earlier statement that the top reason why we fail is because we fail to find the right customers and markets. If we look at where we learn about customers and markets, they happen at the tail end of the product development cycle. So in the beginning we go out and gather some requirements and then when we release stuff out to them, that's when we learn whether what we built was good enough for it, it was meeting the requirement. But there's this big middle where we go away for a long time and I put very little learning, but I'll qualify that. We are learning a lot about product, we're learning about quality, we're building things per spec, but we're not really learning about customer interaction with those products. And that's one of the fundamental things that Lean tries to inject or say the Lean startup tries to inject is change this picture from being a product development life cycle and to being a continuous development life cycle with the customer feedback loop. The third thing that kind of an attribute to this failure is that we all know we need to listen to customers and we kind of pay lip service to this so we go and gather requirements and we try to get feedback from them, but you really have to know how to listen to them. And for those of you that have ever tried to go and ask customers for requirements, you know that they're typically not as visionary as you are. They typically do not do very well, I'm generalizing here but not every customer falls in that category, but oftentimes people will cite this code and say it's useless to talk to customers because they can't tell me what to go build. They're not as visionary or they don't know the product as well as we do because that's what we build. And so they cite this code, the Henry Ford code, which if he had gone out and talked to customers they would have told him not to go and build the automobile or mass produce the automobile but build a faster horse. What we fail to realize is that even in that code there is a gem of an information and that is that the customers aren't really telling us what to go build. That's not really their job and Steve Jobs said that really well but he said it's not the customer's job to know what they want. That's really our jobs as the people who build the product. So we have to figure that out. So even in this code there is a problem statement. What the customers are saying is that they are unhappy with their existing alternative which happens to be a horse and the attribute they're most unhappy about is that it's slow. So had Henry Ford gone and invested in genetic engineering and actually literally built a faster horse that would have been just as good of a solution. But he came up with the automobile and mass producing the automobile and that solved the problem as well. So that's something to keep in mind is that when we are talking to customers the goal isn't to go and run surveys and features and get features from them but really understand their current reality, really understand their current world view and then figure out the right solution to go build for them. And as we get deeper in this talk you will kind of see some techniques for doing that. So how do we build these products that people actually do want even though I just said you can't ask them if they want it or not. So I wouldn't be an entrepreneur if I didn't have at least one slide with my book on it. So this is my shameless promotion. So there is a book that I've written that goes into lots of how-to details for doing a lot of the things that I'll talk about here today but I am going to stay at a high level and really talk about three main topics. So we are going to take each of these layers and start at a very macro level and see how you go about taking the vision that you have and how do you break that down. How do you then strategically go out and launch your product and then we'll get down to some more feature development types of stuff. So I'll have a Kanban board and I'll walk you through the typical product development cycle. So the first step here when you are starting with the vision when you are starting with an idea is you have to document your idea down and for a lot of us when we look at the idea we only look at the solution we look at how we go build it but there is a lot more to that and the reason that this step is really important is that we are especially gifted and I use the word entrepreneur but I am using that for anyone that is building products in the general sense is that we are especially gifted at rationalizing anything in retrospect. So if we build something and it doesn't work well it's usually because of some external condition if we launched a product in the summer month and it doesn't sell as well or doesn't get used as well it's obviously because everyone is on vacation and then after the summer months when it's still not selling or still not getting the traction we want it's because well people are just coming back from vacation maybe their kids are going back to school then there is some festival or some holiday so there is always something that you can attribute to just the things not working and so we want to bring a more empirical sense to how we do things and so almost thinking like a scientist and a lot of this stuff is based in the scientific method is we want to bring some more objectivity so we want to declare outcomes and then measure ourselves against it so traditionally we have used business plans for this or a business case in a company setting so how many of you here have written a business plan before? be sure of hands, I know it's not very popular these days so keep your hands up if you enjoy that process actually one hand stays up so there is usually a few people like that kind of torture who keep their hands up the business planning process is a very painful exercise because a lot of what you're putting down there is you're trying to predict the future you're really writing more works of fiction than facts so it's really a lot of projections a lot of numbers you're playing with to make the case look good the sad reality with the business plan is that the people who make you write it don't read the whole thing themselves because they know it's not going to really work out that way but they do want you to go through the exercise because that gets you out of just thinking of the solution and other things so you can more critically explain what you're trying to do and even though it's not going to come through that way at least you've thought about it so that's what the point of that exercise is so you've done some planning so no planning is not a good alternative either so I'm kind of in the middle ground if you do too much planning or you do no planning you're just as bad so there's a new alternative and that is this idea of the one page business model so these investors don't these investors are people in the company the budgets don't want to read the whole plan but they want you to come in there and give them a condensed version of it so they want you to give them the one page executive summary or the 10 page slide deck and so why don't we just start with something smaller and that's where this idea of the one page business model really comes from it looks something like this we're not going to walk through it today but I just want to show it to you because it's got all the same elements that would go into a business plan but you can create this on a single page and put it down on a piece of paper when you show this to someone they can't help but have an opinion and that's the most important thing because the idea of business planning is to really have conversations with people other than yourself because we can always convince ourselves we are right but you want to have those conversations about what it is we're trying to build what markets we're trying to go after are we building the right features are we building the right products so this is that one page business model format this is a lean canvas format some of you may be familiar with now one of the things you'll notice here is that there is a solution box but it's very very small and I did that intentionally I wanted to add the solution box because as people who build products that's the thing we identify with the most it's also the thing we are most passionate about and the things we're best at but it's also not the reason I also made it very small is because that's the stuff we know how to do really well the other stuff we don't know as well and they're equally important but the epiphany that I had remember I was building products I had products that were successful products that weren't successful but the epiphany that I had is that it wasn't about the quality of the software that I was actually putting out there it was really that the product I was building wasn't just limited to the solution I began to look at the entire business model as the product and that was a big epiphany for me because it allowed me to apply the same techniques that I used for building products to building my business and so I looked at building something we would do if you're building a complex product you would start by some kind of architecture document or you would start with some kind of a blueprint if you're building a house you would start with an architecture blueprint and that's what that one page business model is really about it lets you model what you're going to do with this product without spending too much time planning the next principle also comes from the product development playbook if you had six months to build a complex product it's the simplest stuff first because we know that stuff we can do so what you really want to do is start with the more complex the more risky stuff first so that by month five you're not churning like crazy or thrashing like crazy but when we build our products we do the exact opposite we start with the simple stuff so we go and we build the features that we know and then we start tackling the more complex stuff which is in this case sometimes the market risk and the customer risk much much later we identify what's riskiest not necessarily what's easiest in this particular product when you're building it out and then in the lean startup vocabulary we have this concept of an experiment and that is this build measure learn loop I'll have a bigger picture here so you can see what it is but the idea here is that you take your plan identify what's riskiest it may be that you're building a new a new feature or a new product and you're requiring people to pay because people haven't paid for this so one of the things you may want to do is for example set up a landing page to see will people actually pay for this particular product and even forget payment you might even want to just test demand before you actually go and build this product out so those are some of the things you can do through experiments and as I said I'll get into a little bit more detail when we talk kind of talk more at the tactical level but this picture shows you that high level flow of starting with your vision kind of identifying what's riskiest in it and testing that before everything else through a series of experiments so if we talk about the next phase here which is how do we now go and roll out this product or what is the strategy for building it if we look at the the lean canvas or this one page business model here I talked about identifying what's riskiest on it and people usually ask me where is the risk box, how do you identify what's riskiest and the answer is that everything here is really a risk and everything here is mostly uncertain we think we know how our customers are we think we know what features they want but that is not entirely fact yet until we go validate it and so everything here is really a risk and we need to tackle them but tackling them all at once is not an easy thing either so we want to do this more systematically and so I break the product development launch into a set of stages so there's three stages that you a typical product life cycle goes through and we can take this from the macro level down to building features and releases which I'll talk about here in a second but let's talk at the very high level so when you have a new idea and you want to go implement it the first stage is all about testing whether you have a problem worth solving in the first place not whether your solution is going to really solve a problem but is this really a problem worth solving in the first place and the trick here is that we can do this without building anything we can do this without writing a single line of code and the way that we do this is if we realize that customers don't really care about the code, they don't really care about our solution they care about their problems so if we start with that insight we can build something that I call the offer and the offer is typically made of three things it's the unique value proposition so that's the benefit the customer is going to get after they use your product or after they use your feature so what is it that you're really solving for them are you helping them be more productive are you helping them make more money so what is that unique value proposition that you can tie this product or feature to then what you can actually do is show them a demo of what you're doing so without writing a single line of code you're going to have to do this work anyways you're going to have to design this feature out or this product out but if you go and build a demo you can articulate how you get the customer from their current reality which is their point A to where you want to get them which is the point B and we can do this again without writing any code and then if there's a pricing conversation that can also be had because at this point your customers solve that particular problem for them and you can do all these three things just through a casual conversation this can be a face to face conversation it can be done through tests online so you can if you look at the way we we do crowdfunding campaigns Kickstarter is an example of this where you take a landing page you put up a value proposition this is the product I want to go build here's a demo of it so it could be a video that runs that shows what this product is about and then there's a ask for what I want to go build give me money so it's the same idea there you can deliver this offer in many different ways it can be done with your customers face to face or it can be done online it can be done in other ways as well but the goal of this offer is not to go and again promise the world because if you promise the customers everything it's easy to go and promise people a lot of features I can go and ask how many people would like to take a vacation on Mars or on the moon and I might get a list of people interested to give me money but I can't go build that so you have to make sure that what you can build is only technically feasible but it can also fit within the smallest time frame possible because you want to accelerate that learning loop you don't want to have that big middle where we are building a lot of stuff for a long time so that's where this idea of the minimum viable product comes into play so during these conversations, these negotiations with customers you might start adding more features to your demo because that's how kind of scope creep comes in but there needs to be the reverse process where you start to take things away and you want to get to that minimal viable product which is the smallest solution that you can build that also delivers customer value so you don't want to compromise on that unique value proposition you're going to deliver to them but also you don't have to give them anything more than that so this is how we try to limit the scope and if you're launching a new product I oftentimes see and this is more in the startup world I see a lot of folks launching products and not charging for it because they don't want to ask for money they feel like this is an alpha or a beta version and I look at those things as cop outs because if you're going to do the research and you're going to build this right product for your customer and you're delivering value to them you should also go out and look to capture some of that value back which is just a fancy way of saying you need to get paid for what you're delivering so it's not a one way street you have to deliver value and you have to get some of that back otherwise you don't have a business model that works so the next stage after you've found the right product defined the next stage is to now go build this thing and then see if it's something that enough people want now the important point of the second stage the product market fifth stage is that we don't need lots and lots of users to test whether we have built something well and you will see this again when I start talking about features companies do this already where they do stage roll outs they will roll out to a small audience first see if the product or feature really works if it has the engagement the effect they want and then they'll roll it out to everyone so you don't need lots of users to learn you just need just a few good customers if you're rolling out a new product to your B2B clients you can do the same kind of thing you can roll it out to a subset of people first and if that product or feature works you can then roll it out to more people beyond that now once you have that that product working we then shift gears and go into the final stage which is the growth stage and so this is where we have got a product that's starting to work and we want to accelerate it at this point the focus begins to shift less about product and more about acquisition, customer acquisition and referral type strategies so let's go down to kind of the more lower level and talk about what is this new product development cycle look like so the first rule here is not to be a feature pusher so all of us have ideas all of us have backlogs that we just can't manage and the answer is not to build everything and just push them out but really apply a more customer based feedback learning approach to figure out what it is we should go build and while we're building a test whether it is the right thing that we're building and once it's launched we still want to do that kind of testing so this is how we traditionally do release planning so it's someone who has the product development or product manager role will go and plan out the next year's worth of releases so we say we've got all these features in our backlog we'll start to put them into these different buckets so we have a release 1.0, 2.0, 3.0 and we plan out the whole year but all that is predicated on the fact that we actually know what the customers will really want and so this is in many ways a dangerous place to be the more lean approach is to go out with that idea of the minimum viable product as quickly as possible and put that in the hands of customers because that's where the real learning is going to start that's where you're going to get people to start using your product and the rubber hits the road from there there's going to be customer poll so they're going to either ask for more features or they're going to ask you to change certain things that aren't working quite right and we use that feedback to then build or figure out what the next set of features or releases are going to be so I use the word MVF for a minimum viable feature and MVR for a minimum viable release which is just a collection of more than one feature now the reason I said I'm going down to the feature level is if you are practicing I know there was a workshop even yesterday on continuous delivery continuous deployment kind of is the same in the same realm but the idea there is that more and more companies are beginning to deliver individual features and even partial features out feature flags to do some of those rollouts that I was talking about so you can do partial rollouts full rollouts but if you are in that kind of a mode nothing stops you from delivering individual features to individual customers and testing whether those work and if they do then delivering the rest to everyone else so the big message here is one of going big on vision but then going small on your solution so you don't want to have this massive roadmap that you're just executing for a year in the hopes that it's going to be the right feature that you want to go and test along the way and remember it's not enough to just get requirements from customers because customers don't really know what they want again you have to figure out ways to understand what their underlying problems are and then build the right features and make sure they're using them and if they aren't you have to actually kill those features off otherwise you're just going to create more waste in your system so the second step in this in this product development life cycle is that every one of your releases is going to feature an experiment but every one of your release has to end in customer learning so this is the expanded build measure learn loop that Eric Ries codified almost maybe four years ago now and so it starts out with an idea of something that you have you go and build the product or you build an artifact of the product it could be even a screenshot or a demo it doesn't have to be the full product but you put that in front of customers and you begin to get some data back it could be quantitative through data analytics, things like that but all of that has to end in learning and that learning has to be customer-based learning so it's not enough to say we run an experiment, we build this great product and it was code complete and feature complete and we have deployed it so we are done you actually have to end in customer learning so you have to say we did all of that and we actually learned that customers are using this product or not so it has to end in that level of learning so this is where I'll kind of bring this down to I'm going to use a Kanban board to show a conceptual board that we started using and then I'll end with a final board that you can learn more about later that's a more updated board of this version but this Kanban board was how we typically started tracking our features and our backlog so we had things in progress and done a lot of you here should know what this says we ended by adding a fourth column here we started by adding a fourth column here to the learning column so this is when a feature is built we have to make sure that we actually learn something about customer behaviors customer acceptance, customer usage and so we added this fourth column as the first next step here the other thing that was very important was for us to build a continuous feedback loop during our product development cycle so it wasn't enough just to gather requirements and launch something and then see if customers accepted it we wanted to do something throughout the development cycle so we went back to this board and we kind of added some more states here which I'll walk through in slow motion so we'll go through each one in the steps but these green areas are all the customer validation areas so these are all the places while we are building a feature we actually talk to customers we either talk to them face to face or we measure something through data now when we what I'm going to do now is actually walk through the process of how a feature would go through this process so when we take something off of the backlog the first thing you want to do is prioritize for a problem worth solving so if you already have an existing product launched we want to state what is our key metric so this kind of goes more into some innovation accounting stuff that I'm not going to talk about today but sometimes if you are working with the product the things that you are trying to improve might be revenue, the things you're trying to improve might be engagement, things you're trying to improve might be activation, you may have a very low onboarding in your process so rather than trying to work on everything, you want to prioritize what are the right sets of metrics that you are trying to achieve so the lean start-up world I should say is very metrics driven and so we start by filtering out a lot of the backlog based on the key metrics that we are trying to improve so that's the first filter that we apply the second thing we do is we take things off the backlog and we want to really understand what's the underlying problem behind them so often times customers will ask for features but you want to dig a bit deeper and say if they got that feature what would that let them do what job is that really doing for them does it actually create business value and if not it may just be a nice to have feature so we go through every feature and we kind of prioritize based on that particular job and once we understand and sometimes this requires customer conversation so sometimes you can do some analysis with yourself and one of the techniques that we use a lot is a five wise root cause analysis so why is a customer asking for a feature and it might just be they want the print button but why do they want to print stuff and we go through a layer of five wise trying to get to the real root cause like what's the real reason behind their need to print out this thing which it's beyond just having it on a piece of paper like what's the business case behind that so by doing that kind of stuff you can answer that question but sometimes you have to actually go and validate that with customers so you may ask them just literally getting them on the phone and asking them what would this feature let you achieve and then do five wise with them on the phone but in a more casual manner you don't want to question them very aggressively five times but at the end of that process some of these features are going to be outright killed so it may be that the problem is just a nice to have and not the most important thing and so it gets outright killed or it moves into the next stage which is the demo or mock-up stage and so over here this is where I'm now kind of taking the ideas I shared early on at that macro level is that you don't really need code to test a vision you don't need to build out your solution to test whether this feature is going to work and so you can actually go out and build this mock-up of what you're going to go do and then get into this demo phase to your customers and so at this stage you are again talking to your customers and showing them what their current reality looks like where you're going to get them with this feature and see if you're solving the right problems and that's what we want to test here so the first stage was understanding the solution the second stage the goal is to define the solution sorry the first stage was understanding the problem the second stage here is to define the right solution for that problem and to kind of illustrate that I'll show you a product from one of our portfolio so we have a company that builds many products but this is one of them in there we wanted to take this screenshot which is obviously a rainbow screenshot very hard to see very hard to understand we wanted to improve on this and why do we want to improve on it because this is a dashboard we wanted to build a company-wide dashboard that anyone could really understand and we went through this process of doing these mock-ups so we didn't go and build any code we didn't try to build this feature out we instead worked with our designer and our team the developers at first to go through this process of just doing a bunch of series of mock-ups and we then would ask whether we could actually implement each one of them so we want to make sure that technical feasibility was possible because obviously designers can design things that cannot be easily implemented so when you're doing any kind of new innovation there are usually two leaps of faith one of them is can we build this so that's the technical feasibility question and the other leap of faith is will customers actually use this so will they buy it will they use it, will they engage with it and so we always start with technical feasibility make sure we can build what we are going to go show customers but then we do want to show it to them because we want to make sure we are building that right thing now the key point here is that this process took us days not weeks or months so we literally took days to build these mock-ups out we had things that we could then go demo and we started to put together some screenshots which looked very real when you're a customer sometimes you can show them things like wireframes but we like to go and show them something that's pretty real looking so this looks like a real web app there's actually graphs and there is real data on here that we took from our own system so we could actually tell a very compelling story and we would then give them a demo and essentially tell them what this dashboard what the goal is of this dashboard was I'm not going to get into those details but the point is that we could walk them through this a series of screenshots and show them how we solve a problem that they had asked us to solve and then we would test for acceptance based on this and once we had enough permission to do that we would move on to the next step now this was an iterative process we would go and show these two people and the initial dashboards were not as clear they were actually quite confusing and so we'd come back and just revise the mock-ups so we were iterating on our product but just through the mock-up process and the demo process now once we get enough customers to say yes if you actually build this thing I'm not going to solve my problem or I'm going to buy this even better we then move on to the next stage which is actually building this stuff out so this is where I'm going to go through a few states where we code this thing up and then we often will always do a partial rollout first so this is where we don't roll out a feature to everyone now if you only have a few customers you may just push things out to everyone if you're starting out with a startup or if you have a few B2B customers and they all want this feature but in instances where you've got lots and lots of users and lots and lots of customers doing a partial rollout makes a lot more sense because what we want to do here is make sure that we were actually able to build things that would deliver value so just showing someone a demo okay so just showing someone a demo doesn't guarantee that the feature is actually going to work because we have just shown them a screenshot show them a story of how this will work and then you want to go out and make sure that there's actual engagement with that feature and so this is the qualitative learning stage where I kind of stressed this one earlier you don't need lots of users to learn you can learn with just a few good customers now this stage also has a feedback loop because we go and show this to a few customers and we have them use this feature and based on their usage we either get data that is qualitative where they tell us this is working and do usability tests depending on the product of course you can bring them into your offices and have them use that part of the product give them a task if you're building a travel website have them book something with this new feature you've deployed and you can measure whether it's actually working better than the baseline before and that process itself is going to go back and forth so you're going to go back and improve the feature and come back till you've got something that's working at that small scale and it's a full rollout and even at this stage it's not enough to just be done so remember the done stage is once you've got this full rollout pushed out there you have to verify that customers actually are using the feature and accepting it now that can take a long time so because I kind of show this as a Kanban board we do limit our working in a progress queue so we're not building lots and lots of features at the same time we figure out how many features we can really handle and as we're pushing them out we are once things get into the done stage where they've got that initial validation with customers we then start working on the next set of features but we still leave that feature out so we can measure whether it's being used because if a feature gets accepted and the metrics validate that then it stays alive and in other cases that may actually be killed off it's deployed but we find out that the early signals we got it's rare for this to happen but it does happen where the early signals we got with the demo with people using it if it doesn't scale up if only a minority of people ask for this print function but it doesn't get used by everyone we may actually still go and kill that feature if it's just not the right feature to keep in that system so this is the more recent version of this one but it has a lot more going on with it it still has a Kanban board in the middle but it kind of moves up to a more macro level so it talks more about experiments and talks about how to run experiments in cross-functional teams it talks about doing metrics modeling and starting to do some of that modeling that I had in the board previously you can also see how the last set of experiments went so whether they passed or failed so it has a lot more stuff in here if you want to learn more, you can go to leanstack.com and look at that more updated board the feature Kanban board is also a blog post if you search for how we build features and probably put my name, you'll find that blog post so I didn't want to spend a lot of time talking because I like to have more interactive type conversations with the audience so I kind of open up for questions if you guys have any questions you can jump back to slides yeah sure is there a mic that we can but I'll repeat the question whether we have timelines if I understand right for each of the stages like how long does the process take sure I think in many ways it depends on the relationships you have with your customers so we try to we try to build our customers in the company that I run so that we can set these conversations up very very quickly so if we were trying to come up with a new feature idea it can literally take us days to get a mock-up done and customer validation and whether that mock-up is going to be accepted and then of course the complexity of what we are trying to build is going to dictate how long it takes us to build that initial minimum viable feature and then we go through that demo process but that usually we try to time box as much as so when I go into this particular board when we look at our experiments that we run we try to time box all our work in two-week increments so every two weeks we want to be having customer conversations and that sounds very aggressive for some people that's not feasible but it can be done so this looks all very linear and you know a happy path but very often what happens is you have a mock-up that people don't like you build a demo that is not acceptable a code that doesn't work and qualitative validation says that it's not valid so at every stage there's what we call as a possibility of a pivot right and where you throw this whole thing out and start all over again so I don't think this captures that part well so in some of these let me take an example of like this one you know there's places to go back so the idea here is that if you go through the demo stage and it's not talking to just one customer you're talking to a bunch of them you have passed that gate and you move on to the validate qualitative stage but if you're not getting the feedback if you're not getting the right signals there you are going back and reworking the product we saw that what I'm saying is that at some stage you might need to throw the whole backlog out and start all over again that's pivot the idea itself is right so I guess I just want to make the point that there is and also the other thing is that at any point in this feature life cycle a feature can just be killed I showed the killing at the end but at any point if you're not getting the right validation even at that qualitative stage you can abort that feature but to your point of if the backlog itself is wrong so we don't usually work off that backlog that backlog is really collecting things to play so if we do make a major pivot so we go through this process and we are trying to improve say revenue and we are just failing to do it consistently there's another set of there's another cadence of progress reporting that we have that says hey this product is just not delivering on revenue we need to make that bigger pivot so maybe we're going after the wrong customers and yes if that happens you would change your canvas up and you may come in here and like you said throw out stuff and the departments that just changes entirely but the idea of this so this is more of that tactical feature level kind of dashboard so provided your strategies intact so maybe just for the benefit of the audience a pivot is really a change in strategy and it's not something that should happen weekly or bi-weekly if it's happening that often you're doing something wrong it should be a major shift in your focus but when that pivot does happen then obviously a lot of things are going to have to be thrown out and kind of aborted questions yeah sure I'll repeat the questions how do you apply this to a legacy product so I would say it's almost the same so it's a great question people always ask if you're starting a fresh it's easier because you can start and do all this validation but even with the legacy product I would say it starts by baselining and it does require more work which is your current product so often times and I've been in this situation where before I started to apply these techniques I had products with lots and lots of features so you want a first baseline what's really getting used the best ways to go also interview your paying customers and see either through data or through conversations what are they really using your product for and there could be a whole bunch of stuff that you can just really essentially eliminate you can strip out and actually do more radical stuff where either accidentally they take out features or sometimes intentionally they take out features and they see if anyone really complains now you cannot do that with every type of customer but the point is that that would be step one is trying to figure out what is really getting used and what's not getting used and then for me as you start to add more things you can then inject this process in so it's scoping down and then trying to make that infrastructure set be a minimum viable release going forward yeah question following question to that is often the customer doesn't agree with the definition of the MVP that you have so how do you resolve that conflict sure and could you reset the first question again the first question is I wanted to understand how you can define the MVP as objectively as possible so that I'm in some sort of guideline which is very objective and the MVP in that framework sure so I guess for me if you're building a visual product like this one for me the best functional spec is not text or words it's really trying to get down to that user flow that user story and really getting to that job the customer wants done so if you can show them a demo and they get what you're trying to build of course they're going to be loose ends and they're going to be things that you have to you have to come and supplement the in many ways the best kind of spec for your MVP and the second question was when customers don't really agree with your MVP what do you do in those cases that's again where you have to dig deeper so it's again it's that process of conversation there are a lot of techniques out there some of you might be familiar with things like things from innovation games where you can play games with your customers to try to have them pick their features based on some currency that they play with but the the basic message there is that you want to get down to understanding why those features are really required and can they live without them because you're not essentially eliminating them you're not telling them we're not going to deliver this the whole promise here is that we want to deliver stuff faster to you and if you can get pretty sophisticated with this board you can almost get to a point where you can predict your feature life cycle so if you go to Disney World and that's the promise of Kanban of course if you go to Disney World you can tell how long the lines are just based on those markers you can similarly go into a conversation with your customers and tell them how long features are going to take to get done and really have them prioritize would they rather wait three months to get these three things or can they get the one thing initially so I think it is a conversation that you have to have with your customers but it gets down to that root cause analysis because many times customers will ask for things that they could really live without for a little while longer yeah I'm not sure if they actually started with mock-ups because for them it was trying to change the culture and behavior of the society right so they won't go to customer and ask for what are your requirements right so if there are certain products like that how do you go about I guess can you repeat the first part you said there were some products like Facebook and WhatsApp which probably are the drivers to change the society to change the behavior in the society it's not really that customer came with a problem certain folks saw that there is a need for that and they started with that so what do you do in those kind of cases do you really create mock-up and go to customers and validate that so even in those instances there is customer there's lots and lots of customer feedback so let's just take Facebook as an example so I kind of privy to some of the ways they develop stuff but when they run in a very decentralized type of an organization and anyone in the team can really propose a new feature idea so they have a vision for how they want to go build this and there's a goal behind it we want to improve engagement or we want to go and build this feature and it's going to cause this to happen so we have some kind of a case with it they go and build a team around it and then they do go through a lot of this mock-up process where they are testing for usability so they're not going and asking customers again so I want to drive that home at that point home again is that we never go and ask customers what do you really want what we want to really understand is what do they need by observing them or by interviewing them so in a B2B context you can go and interview people and the best questions to ask there but I would frame it as we are trying to solve this particular problem for you and people would say yes that's a big problem for us and then you want to ask them how do they solve that problem today so it's not having them tell you how you should solve it for them but asking them how they solve it today so by doing that you understand their workflow and then you can go back and come up with a much better solution now in the cases where you go and build so we can take Apple or we can take Facebook in those cases they have some intuition or some insight and Steve Jobs was known for being a great observer he would obsessively go on these long walks and just stare at people and see how they were using things and Apple even has there's a new book called Inside Apple and they talk about some of their design process so they actually go in they actually follow people home so if you buy an Apple computer at the store they get permission to go to your home introduce themselves as some VP of product and they go to your home but what they're trying to do there is learn through observation they're trying to learn where you're getting stuck and then they go back and design that right solution which they then test and then if it doesn't work as expected they keep going over it over that process one of the things that you said is start with the riskiest component first whereas we in the agile community talk about start with the components that you know instead of building the whole blueprint of what you don't know perhaps the best do you see a contrast in those two thinking paradigms? yeah so I would I would just say that I would say in going back to the going back to the underlying failure rates I would say that oftentimes the root cause of failure is building that wrong product and starting with that incorrect risk so Eric Rees likes to tell this story and he tells he talks about spending six months of his life building code perfect, unit tested great software and they learned that nobody wanted that software and so at first he was very depressed because he had to be taken and screaming because he spent six months of his life building this thing he didn't want to throw it away but then he convinced himself that well at least we learned something at least because we build the software we learned that nobody really wants it anything without writing a single line of code because the reason that nobody wanted that software was something they tested right on the landing page people were not even downloading the software so that in many ways was the riskiest thing so I would always just go back to if you kind of had more time I would go and show you the customer life cycle of how they adopt stuff but you have to first test customer demand or customer pull for your product and that often is the riskiest thing to go test if you can get over that that means you can start I wouldn't comment on product development so I keep that separate I'm talking more about prioritizing what's riskiest in the customer journey rather than what's in the product development path we are only talking about building the features how about purchasing or partnering with somebody so the question was partnering something for what sorry like instead of building from the scratch there's a similar thing what you want you know the first thing is what you really want then the solution is if something is available ready made which you can customize sure yeah so if I understood the question is if you can uncover what people want and there's a solution already out there can you just partner with someone instead of building it yeah so by all means I mean if that fits into your business model so one of the things we talk about there's three techniques that do just that so we use the more traditional MVP is what I showed here which is you take your vision of your product and you scope it down but we do a lot of really crazy stuff so that's something called the Wizard of Oz MVP which is just cobbling up stuff because we want to make sure is there really customer demand for what it is that we are building and if there's a faster way to go test that so when Zappos for example launched they didn't go out and build an e-commerce site so Zappos is online shoe store experience was very poor and by making the experience better they could sell, they could build a much better brand but they didn't go out and start building an e-commerce website and start stocking shoes they literally went to other retail stores and took pictures of shoes, took high end pictures of shoes, put them on a website and started taking orders that way and people would probably get freaked out and say well what happened next well they just went and bought those shoes from those same retail stores and shipped them and they build everything out and see does it work but they wanted to see if we actually did the riskiest stuff which is we build this high end experience would people buy those particular shoes and once they prove that out they then had to scale their own solution so they had to build their own products so in your case if it makes sense to partner and you can make the business model work that obviously makes sense but if it's a way to speed up learning that obviously makes sense as well the first stage is basically to figure out the customer requirement whether the customer needs it or not now let's say suppose you have figured out that there is a section of customer who needs this really and it's sort of verified learning now the next stage is the viability piece like whether is it profitable, is it like customers willing to pay for it those kind of stuff so think like whatsapp there are like 450 million users but you still don't know whether it is profitable what would be your approach I think that's a great question so the way you describe it the first stage is more about testing customer demand and then you test about business model viability I would say in some products where your users are your customers and you use whatsapp but if users are your customers I like to test viability with the demand so when I show that offer if I'm going to a B2B customer I wouldn't just show them a great product and say please use it I'd much rather go and test pricing right on the spot to say here's this product, we saw this problem for you and once their heads are nodding and they're excited I say this is going to be the price and then there's going to be pushback and we're going to negotiate so to me that has to happen early because price is one of the more riskier things in that business model now in cases like whatsapp, facebook, twitter all those are business models where there's a derivative form of currency so it's a multi-sided market you have your customers who are paying for your product and those are advertisers and then you have users that are creating value through either data or engagement and so I look at them as a derivative currency so there you're building this asset and in the beginning it's worth nothing because when you have 100 users today that's worth nothing in those types of products but once you have a sort of tipping point then you're worth a lot so there you're building, almost think of it as currency there's this exchange rate and you're playing with that the weird part with that is that exchange rate last year 25 million was the big thing and now it's... sorry 25 million users was big now it's 250 million users or whatever the number is alright, thank you very much I'll stick around for a few more questions I have just one question I think we're out of time but if you want I'll stick around we can talk for a few more minutes thank you