 So I'm just going to share a bit of background about myself and then talk about what I hope to achieve in this session. Okay, so I've been practicing Lean Startup for a couple of years now and I'm also part of a couple of different groups where I mentor a few of the company's startups. I'm now with Intuit, which is also a much larger company that still works as sort of small teams of startups using both design thinking and Lean Startup. And I want to sort of just share some real practical examples of how I've tried to use Lean Startup and some challenges that I've faced. And I don't want to be someone who is coming and trying to give a lot of gyan because I think there's enough gyan out there. So I'm hoping I can share some real examples because I think that's kind of where it's valuable and encourage more conversations within our community to talk about sort of what's working for you or what's not working for you. And how can we learn from each other? And I think to me if I can incubate some thoughts and get people together here, talking about it, figuring out how in the context of Agile and Lean things could work together, that'll be great. I'm not here to evangelize Lean Startup. I think there's some strengths here, there's certain areas where it hasn't worked for me. Ultimately, I think these things are horses for courses. Design thinking, customer development, I think you've sort of got to choose depending on what you're trying to achieve. But here I'll talk a little more specifically with respect to Lean Startup, how I've applied it and certain things that I've learned. So I'm a perpetual entrepreneur even if I'm a part of a big company, an intrapreneur in a big company. I've been a part of multiple startups in the past. I lived and worked in the Bay Area for a while as part of one startup that we took public. We grew it globally. I've had a number of different roles, some products to the market. I've been both in engineering leadership roles as well as in product leadership roles. And I just love working with small teams to foster innovation and help take ideas, you know, ground up. I've also been a mentor at Lean Startup Machine and that's sort of where I felt like, hey, it's good to sort of talk about all this but really sort of sharing examples is useful since we can then really learn a little more from each other. I'll also talk a bit about me as a senior product manager into it and the context of how I know many of you are from much larger companies and I think a lot of this still applies for you although the case study that I'm going to share is from a startup that I was a part of earlier. So this is a case study that ended up getting really popular. To popular at the point where it's almost a little embarrassing in some ways but I had people like Benjamin Southwick and Gary Toss, people who I really, really respect. I'm reaching out and saying, hey, this is how you guys should build product and I think it sort of inspired me to say, hey, you know, to look to share more with people, learn more from others too and I hope that we'll all have opportunities to do that. I did do a quick show of hands earlier. I kind of got a sense many of you here are exposed to Lean Startup in various ways. Some of you are actively practicing it. A few of you here are looking at design thinking too. But since there are people who are new, I'll spend a little bit of time talking about what Lean Startup is, just a little, and then I'll go on to cover up the case study a little more in detail, certain surprises that we learned, and a bit about just why I think Lean Startup is hard in practice and how I've struggled with it a little bit. And I welcome feedback and it puts from those who are practitioners as well. So I think the first thing I wanted to start off with is to say that, you know, Lean is used in a broad context in manufacturing where Lean is about efficiency. So, but I want to, for those coming from the manufacturing side of the world, I want you to sort of set that aside for a bit. And, you know, Lean Startup is about translating vision into product and it's not just about Lean from a cost perspective. So that's the first important thing when you think about it. It's not about cost savings or trying to run with a really lightweight team and get something done. It's not the cost saving piece, but really about a systematic way of translating vision into product. There's sort of three pieces to it. How many of you are familiar with the business model canvas? Okay, perfect. So I'll talk just a little bit about it and then I'll move on. But there's three pieces, business model canvas, customer development, and agile development. And these are sort of the three blocks that sort of work together and I think of them as tools that sort of bring Lean Startup together overall. I also think of Lean Startup with a methodology. It's important to know that it's not something that sort of set in stone is a process, it's a way of thinking about how we do things. And here's an example of a business model canvas which sort of tries to point out where the different pieces come together. The simplest way that I like to explain it is think of it as a business plan that's written in a certain way and it's a business plan that you write in a way in which it makes it easy for you to test out various assumptions that you're making. So for instance, if you're a startup and you're trying to map out various pieces of your business, the customers, your value proposition, acquisition channels, often I find that startups try to really test a value proposition side really high but they don't focus on testing the channel first. And I'll talk about why some of these pieces are important in the context of experiments, especially because you end up having multiple control variables which then become a little hard to manage. So think of this as just a business plan, right, which is a plan that you can test in various ways and gives you a way of thinking about things in a systematic fashion. The second piece here is customer development, which Steve Blank, who's definitely I think one of the people you should read, if you want to read more about customer development, I encourage you to read Steve Blank's blog. There's a manifesto that he put out and one fundamental important thing is, look, there are no facts inside your building. You could put a bunch of smart people together who could talk about things, you could argue and debate things from a logical perspective forever, but you've got to talk to real people. You've got to go out and talk to real people to find out the why behind what they do and there's a certain qualitative aspect to this that becomes really, really important that also has to be supplemented with the quantitative side of what you do. So ultimately the thing here is you're trying to... the thing is it's not about how do we do this, right, because we can always build things, but why should we build what we're doing, why are we doing what we're doing and trying to sort of figure out the why that becomes really, really important until it's almost at a deep philosophical level that helps you, gets you passionate about a broader vision for the change that you want to make in the world, right? So avoid surveys is sort of what I'd say. I mean, there's situations where surveys are useful, but I'd encourage you to avoid surveys, get out of the building, talk to real users. Now, I can talk a lot about customer development because there's this systematic approach to how you ask questions and to ensure that you don't ask questions which are leading questions, you don't ask questions which are close-ended and ensure that you're talking to people who won't hesitate to tell you that you're full of shit of what your idea, the idea that you have is bad and also ways of sort of going about asking questions without revealing your idea because people ultimately, inherently, just want to make you happy. So there's a classic, there's a book called The Mom Test which I'd recommend for those of you who'd like to look it up. Rob Fitzpatrick is outlined, I think, in a very simple language, different ways to think about how you go about doing customer development and asking questions that I think are very valuable to a lot of people and how maybe even internally in an organization when you interact with people about an idea, it's a good way to get feedback. Related to customer development, but different is design thinking and I won't delve into that, but I want to point out that ultimately it's sort of horses for courses. These are ultimately different ways of solving problems and you sort of have to understand these to choose which approach makes sense where, design thinking comes from a much more of a systems perspective broadly. I think the way I try to see customer development is sort of a more lighter-weight way of going about seeking early validation. I think a question that came up in the last session was around Lean and Agile and there's a slide from Steve Blank which, sorry, this is from Eric Rees which I think is useful. I don't think it completely defines how the two work together, but I think it's a good model to think about which is I see both as just loops, right? You've got one loop going with your customers at one end, you've got one loop internally and you're managing two different loops. One loop which sort of answers the question of what's the problem people have, why do they have the problem, why should we build, what should we build, right? And the other loop which is how should we solve this problem? Is there a better way to solve it and sort of maintaining a tight loop at both ends? And ultimately within the organization ensuring these loops are in sync, driving communication, fostering that culture of experimentation at both ends, to ensure there's sufficient customer focus, qualitative data coming in along with quantitative, combined with that internal cycle of enabling this becomes really important. So I think the two tie in really well. I don't think it's perfect quite yet, but I think this is an area where I feel that there's an opportunity for all of us here to see how we can embrace this more and we can talk more about that. One quick thing I want to highlight is remember that I think the focus is on learning, right? See, you can run through experiments and you may something might succeed, something might fail. But failure is failure to learn. If you fail to learn, that is failure. If you've learned something about needs along the way, that's still success. So this is an important thing. The focus is on learning, not just running through it. Go ahead. Should not be? Big failure which can doom you basically. Absolutely. It's a great point. I think for me, back in the day, 10, 15 years back when we would build products, we'd go through the whole big cycle of product out, go through a big bank, PR cycle to launch it and then for a tail end of the cycle that you really realize how far away you are from market needs and definitely the focus here is on failing fast in a certain sense. But I have some caveats around this and I'll talk a bit about that because failing fast has some downsides where I feel that you end up in certain local maxima. And I'll talk a bit about that too in a bit. So I want to ask you guys, what is an experiment? Anybody? Simulation of a reality that which is having a known outcome where the outcome could be either positive or negative. Anyone else? Trying something that you don't know. Great. Validating your assumptions. Trial and error. Trial and error. Interesting. Anyone else? Finding out something. Testing something in a laboratory. Testing something in a laboratory. A lot of good perspectives. I think different people will describe this in certain ways. In the context of Lean, I like to think of it as it's a series of tests which help answer a question. And the question is really sort of the hypothesis which a few people here have mentioned. And the reason this is important to understand this is we end up as a part of Lean startup structuring a number of experiments. Now these experiments all tie back to help answer certain questions. Now, what are these questions? What are the most important questions? How do you figure out what those most important questions are? Right. And how do you go about doing it? So in the Lean startup context, what this is often thought about is there's a leap of faith assumption which is a number of assumptions that you might be making is one that you would identify as being your leap of faith or sort of the most high-risk assumption. Right. And typically you would start off with your most high-risk assumption, craft a hypothesis or a question around that and then run an experiment to test it out. And I'll go on to show you a few examples. So we'll get into a little more detail. So one myth I want to debunk in case this is out there at all is that Lean startup is just for startups. Like I mentioned earlier we're a $10 billion market cap company and we run our product team in pretty small groups and we focus on design thinking, Lean startup experimentation. I'm personally involved in experiments focused on the long tail of small business owners and we drive innovation at a number of different levels trying to encourage entrepreneurs and keeping a tight loop between product management and product lab. So with that I'll go on to a case study which has been shared in the public domain of one of the previous startups that I was involved in called Levitim and it was called Fun with Sharads and these experiments were run a little while back November 2013 through Feb 2014 so sort of a four month cycle three to four month cycle where we went through with these and here's kind of where we started off we sort of had a certain vision which is hey, remember the days when it was so easy to meet people online you could connect with them and now we're all stuck with mainly products on Facebook and we meet people sort of in the real world but sort of level of online interaction for most young people has sort of gone down and we were trying to think of teens, college kids, yuppies, casual gamers as potentially being the target audience that we would look at and we felt that the problem was there's just sort of a lack of fun options for meeting more people you remember the days when you sort of had pen pals you would write to them back and forth I don't remember any of that but that's sort of gone in some ways and maybe it's a bit of a nostalgia that inspired this a little bit but the thought was what if we could create an online community as a solution where people would play charades over live video by either web or mobile over live video where someone would be acting out clues and the other people would be guessing so live video where people would access would get on to live video chat and people on the video audio turned off to the other people and they're guessing and people are responding so now you know as we sort of brainstorm and came up with all this we were thinking hey you know there's this popular game that got sold for like several millions of dollars and has like millions of people playing Pictionary online so if people play Pictionary online would people care to play charades and Damsharad is popular in India you know we think it's popular around the world it's fairly universal people understand how to play it so hey maybe it's interesting we were like super excited so but where do we begin there's tons of things we could go about where should we even start so you know we're practicing lean startups so we try to just sort of step back and say what are some of the high risk assumptions we're making here and we try to sort of just put a list together and then start biased towards action and try to go out and actually build something so I call these the big ifs this is not really a lean startup terminology thing but I try to encourage a team to sort of capture these big ifs what are the biggest ifs involved and have some sort of cadence around what these ifs are so this is sort of how we structured it and it's possible there are other ifs involved but this is what we thought of if people are interested in playing Sharad online if they are and if they're comfortable turning on video if they like the concept enough to invite friends or to join a public room because often you know these game rooms they either have private rooms or public rooms and if they enjoy the playing experience with others enough so that they will come back so your retention side of it will be ARRR metrics and if they invite enough people to sort of create a little viral product loop then we'll be on to something now we didn't know what that something would be and we didn't want to bother about monetization and all those fun questions until later but this is sort of how we framed what we thought of the fundamental pieces of it because once you get to a certain level of BA, UMA, U and active usage of product market fit some of those other options can sort of be worked through so these were our big ifs questions that we had in mind do people really care about these things or people would be comfortable turning on video and you know one challenge obviously within the team we've got some fantastic like 10x engineers who are like hey I can build this in a week let's just do it and we had to sort of resist that urge to sort of go through to say hey let's go about it systematically and see what we can do and we started with the very first one which is do people even care about playing Shiraz online and you know the experiment hypothesis was of a sampling set of people who are searching for Shiraz online right so people who are already Google searching for Shiraz online we felt that at least 25% of those people who are already searching would click through to come to our page and then would sign up now landing pages don't always tell you a whole lot mostly what they tell you is that people understand what you're doing so there's a bit of a caveat there but at least it's an initial signal with respect to the first piece of it so this is pretty much what we did we ran a Google ad which went to a fairly crappy page this is a mock up but the page looked worse than this it looked pretty bad so we ran an ad with like just a few hundred rupees and targeted people to play Shiraz online and we had more than 25 people coming and signing up on a pretty crappy page that had like nothing on it so we were like hey okay so at least there's an interest in playing Shiraz online among those who Google for it that's all we know you can't really infer more than that so one part is you've got to be honest with yourself you can't really extrapolate a whole lot here this is all you can learn from this so but from here we went on to the second hypothesis we had which was to it's still the first hypothesis but trying to collect a little more qualitative feedback so here is where we sort of built out the second MVP which is to sort of run another experiment where we spoke to people within the friends network to walk them through mock-ups of what the flow would be like and collect feedback and this is sort of an important one we essentially asked them the question about how disappointed would you be if we didn't build this if we drop the idea how disappointed would you be so the question to ask is not do you like it do you not like it let me know later we'll check it out never happens right so we actually asked them how disappointed would you be which is a fairly harsh test and the good thing was at least keep in mind there's a bit of bias here because these people are through the friends network there's a bit of bias that is induced but feedback was unanimously positive one little flag was a few people mentioned that they may be comfortable being maybe uncomfortable being on video online so there's a bit of a flag which ties into one of the other assumptions we had made about people being comfortable being on online so this is what the wireframes look like this is the initial MVP also keep in mind like I think Narish pointed out an MVP is almost always more minimal than you think it is you think it needs to be so you have to almost keep thinking hard to see what am I really trying to answer the question for and can I think of something that's an simpler, lighter, quicker way of getting that early sort of coarse granular validation keep in mind that the coarse granular validation when you expand out to a larger data set might change so this is the wireframe and feedback was largely positive like I said I think we spoke about 20-30 people and more than 85-90% of them felt they were very disappointed and they were like if I had it now I would right away start playing and inviting people so we noted as well that a few people had concerns about turning video on and they were like if I turn video on will that be something only my friends can see or will it be in public I'll talk a little more about this earlier but we grappled with this thing of for us to scale a business around this and get some real numbers for public game rooms because private game rooms have the ghost town challenge of like Google Wave you go in their private room and you don't have anyone to wave to you don't have anyone to play with and then there's abandonment so we wanted to start with public and find a core group of early adopters who really really cared about it and then grow it so we wanted to focus on public first now we went back to the people in the room and joined people and we tracked analytics through the course of this and as we went through this we had decent screens at this point you know a little more some amount of work and design putting a little flow together where people would come in and then they would need to turn on video and the little prompt comes up in the browser now so here's a surprise right we were hoping that at least 20% would respond but less than 5% turned on video in the public room now we could have followed up with other hypothesis like saying if the design was better that conversion rate would improve but then we really wanted a much stronger signal because sure design can improve it maybe a little bit 5 to 7, 7 to 10 but we're looking for a much stronger signal here so we kind of had to step back to the drawing board and dig into this a little more so we tested a few pieces out then we're like okay if there's concerns around being on video in public how would people feel about playing with friends in that context and so we ran a couple of tests and I'll walk you through them one part of it was friends of friends so one step further removed still some bias but a little less and we ran tests where we invited them into a sort of a setup where they were at our places and a few were out in like their own homes and trying to play from different locations and to see how they responded to the flow and how they started to go through and play we didn't actually build out the whole app at that point but video was there so people could be an online video together and there were instructions where with video they had to sort of figure out how to play and do it themselves so pretty much think of it like a Skype video call for all practical purposes now here's sort of the offline side of the validation which is running it live with various people and 80 to 90% of them felt very positive super excited wanted to come back and play again wanted to get in touch with all the other people they didn't know and connect with them the positive offline validation we had people in age group 20 to 30 as well as a bunch of like school and college kids who were like 15 to 20 keep in mind like India doesn't have copa laws so some things doing some of that was a little simpler as well for us to run through here now after we saw that people were comfortable with friends we were hitting a bit of a road block like we could have pursued the friends part but we hesitated so we step back to the thing of what can we do to get a core group of people who would be comfortable doing this in public and so we were thinking and scratching our heads really hard saying how do we find early adopters who are the people who really care who are the people who would be comfortable being on video online with strangers after thinking about it for a while getting a little creative we were like there are the sex chat groups online so let's look at the sex chat communities of people who see even I'll answer that quickly so as we interviewed the people who were comfortable friends of friends there was still a certain context that they felt comfortable in there were a few people who were still like yeah I might play with people but I'll have to see but we didn't get a strong signal the thing was still like I would prefer to play with friends and some of the guys were like yeah I don't mind being on video with strangers but overall the signal wasn't strong so we felt that it would still be a little hard and we wanted to test this out a little further the other thing too is we remember the past experiment we actually had data showing the people had come in and were not comfortable getting on to video with so we had data so we wanted to test this a little further to get a core group of people who cared so the chat rule and chat random community, sex chat community is the people who are comfortable being nude on video through most of the day with a whole bunch of different strangers so we ran Google Ads trying to acquire these guys and we offered a completely anonymous experience in a public room so they didn't have to sign up they could come in and play and we added a call room I don't know if you guys are familiar with call room but you know a little tool to try to gain insights on why they don't turn it on so the numbers went up so we had more people who were turning it on and we had people who were sharing it publicly too so these people were inviting people publicly saying hey come on in and join me to play Sharads so it was good but still the numbers to be fair were not high enough right so there was a huge bump in the numbers but the numbers weren't high enough so this is kind of where we grappled with this public-private question because the public rooms it's important because we want to get to the critical mass of core active users and then grow DA, UMAU from there but family and friends are not comfortable private rooms has a ghost town challenge similar to Google Wave and the real-time game also poses scheduling issues you have with the private room is I get on, I check this out, I invite you to say hey come join me you see it two hours later and by the time I'm gone so to figure out a time that works for different people who are friends is pretty hard because in real world we meet here we play Sharads together but to get that time sink was proving to be a challenge people we spoke to we got a sense that wouldn't work we considered an async flow via a mobile app and we tested a few experiments around it but there were experience issues which were causing friction and for the guy who's got to act it out where does he keep the phone is he comfortable putting it up there and doing it and there were a few other experience issues we were grappling with that made it a little hard so there were a few other surprises we learnt too and this is one which this is interesting but Sharads has just played very differently in India in India it's very technical it's very geeky, it's very competitive it's played in colleges people from their own clubs they've got all these alphabetical codes to do things but the data we got is at least in the US, Canada most of the parts of the world it's a very simple, much more simpler version of the game and it's very family oriented and it's played around the holidays around Thanksgiving, Christmas so just a very different context very different thought process very different approach altogether we discovered that there was potential for offline versions of Sharads so for people who are playing offline tools that will give them word suggestions tools that will give them ideas and how to coordinate it among a group of friends though we discovered this we didn't pursue it but what happened later is interestingly there's a popular app called Heads Up Sharads an iPhone app which you sort of hold up in front of your head and play which came after this which actually pursued the idea quite successfully popularized as well where a TV show which happened around then so we learned that there's interest in Sharads it's not a weak signal but it's not a strong signal either and we've got an ambivalent here because one of the classic problems with startups is you'll hear a lot of them talking about we're working on a growth hypothesis we're trying to grow from this stage and there are many of them stuck in what I think one of my friends Rajan he calls it the happy confused state we've got a certain number of active users some people are willing to pay you're unable to grow that number and often that hard question you've got to ask yourself this do I think if I keep spending time this will grow am I really onto something that's big enough is this a big enough problem for me to go after or am I just fooling myself into thinking like I can persevere on and that's hard should I pivot should I not is this sort of a broader pivot what sort of a pivot is it and it sort of is hard but at least at a certain level when certain things happen and we made a choice at that stage saying hey we'd like to pivot a way to try some other things there's certain things I think we did well but certain things which I don't think we did too well and there were good learnings there I think we captured a top process well let's set out the assumptions laid out hypothesis well I think we found small batches that we were able to go through and prove or disprove and run quick lightweight experiments on the agile side and I think we stayed relatively honest and objective we tracked metrics, we tracked cohorts we did the diligence going through it in a systematic objective way what we didn't do well and these were some learnings you try to build a product for a global audience it's perpetually a challenge doing it from India right because if your target audience is global and you're trying to look at a global audience and let me answer the question about why global see back then 4G hadn't come to India so data was still not great and a number of people with Wi-Fi in the target audience of young people were still low so we felt that the India play was too early and we had to achieve something in a global space before later coming back to India but getting qualitative feedback sitting here as a start-up was a little difficult because to reach out to people in the US if you're willing to get them to spend time now I'm at Intuit now at a bigger company with much more resources where we actually have at Intuit a team that's dedicated to recruiting people we can talk to but as a start-up we didn't have the liberty it was very hard so we had to find people we had access to which was local and these were not representative of our target audience second we pivoted away from our original vision and a lot of feedback that I got after the blog post was popular on Hacker News and a number of people reached out saying it was good why didn't you stay true to it because Sharad's need not have been the answer there could have been other solutions to have achieved it and I think that's something which we could have stayed true to in other ways third little thing you know we added inline surveys later to look at abandonment but we could have potentially done that sooner because one challenge is if we can't talk to people from here in India especially the anonymous users those inline surveys like Kualdru could have given us some insights on why are you leaving are you going to come back later is it because you want to invite a friend some of those questions where you just don't know why and how do you figure out the why when you're sitting here in India and trying to target a global user base it's a little hard so with this I'm going to touch upon I've got another 15 minutes I'll go through this a little quickly and maybe we can talk about this offline I want to talk a little bit about why I feel lean startup is a bit hard and practice and for certain types of things first off I think this experiment design is hard I've mentored a number of the startups a lean startup machine and other groups crafting a hypothesis is hard it takes a certain discipline to sort of just go through to figure out what your leap of faith assumptions are while this is hard this is not a bad thing it's hard but it's not a bad thing this is a good thing now second is we talk about experimentation and we feel good hey I'm like a scientist sure but scientists like many of you pointed out earlier run experiments in a controlled environment where they can manage control variables and you can run experiment batches with a certain level of rigor and persistence here it's out on the internet right different groups, different variables it's hard so to run these while understanding what are the control variables which are in flux and trying to manage them requires rigor and patience both rigor and patience are important see lean startup can be a little frustrating when working with an agile team you've got developers who want to quickly build something and go through cycles and you're like hey no no no we don't want to build it until this is really validated and sometimes it's a little frustrating while it's not necessarily orthogonal like I think someone has mentioned where is agile has been incorporated a number of these ideas in certain ways or the other in the past not completely new it's sort of a point of friction to just keep in mind which is a rigor and persistence and are both needed third is you know you need decent sampling sizes to really draw any inference you can talk to 10 people and infuse something you can have 100 people come to the site and infuse something can't always extrapolate that so the sampling sizes start to get important how do you know what the right sampling size is there are scientific approaches to doing it but in practice what you do it's a little hard you know I think our understanding of cognitive psychology I think is still evolving I'll be honest I think I'm still learning here and to design some of these experiments in a way which leverages some of those learnings becomes really important because you've got a small experiment and little things here completely change how people respond and I think this is a very important piece I'm still learning here I'm looking for more people to connect with so I can learn more about this I think the other challenge with Lean Startup is I feel that certain commercial opportunities may not emerge from clear problems and I kind of feel like if you take an example of how Flipboard came out when Google Reader was already there and there were a number of other popular apps that already solved the problem Flipboard was wildly popular but sometimes I struggle personally to see where the Lean Startup would help surface those sort of opportunities which could also be big so this is not to sort of say the Lean Startup is bad but to say that you sort of got to choose it for certain types of problems so I think the other part too is getting caught in local maxima and I think a number of people have spoken about this Peter Thiel you know 0 to 1 few other things but often with Lean Startup you're sort of iterating away in a certain focus area and sometimes you've got to sort of look at sort of a broader bigger vision which you want to go after and sometimes you get caught in the local maxima little caught in the weeds a little bit I think this is a valid concern I've heard it from a number of people I'd be open to ideas for those of you who have gone through this to see how you think about it and finally it's sort of this balance between bias towards action and bias towards reflection and I think both are so important because you know big company, small company you need to have enough space for reflection and reflection is important to draw insights to take the time to think but there's also a bias towards action to go through those cycles quickly and it's a tricky balance I personally struggle with this a little bit some of the entrepreneurs have spoken to have so this is just a quick summary of some of that, the points we just covered off so what happened there was after I made the blog post and I shared it in a couple of the lean startup groups that are the part of one of the guys I respect is named Sean Murphy one of the lean startup consultants he liked it, he picked it up and he shared it on Hacker News and I think it was like a Sunday morning I was going out somewhere with my wife and I was like damn all these people are commenting on my blog sending me tweets, I wonder what's happening later I looked at the analytics and I realized that I was on page one of Hacker News and one day I had like 20,000 visitors which is probably like 20 times how much any of the other posts in my blog have and I had people who I respect a lot like Benjamin Southworth and Gary who is a board member at Hootswit saying that hey I'm going to use this as an example of how you should do a startup and this is great to see someone practicing lean successfully and this is how you should practice lean and Rajan talking about how on this side of the world this is probably the best application of lean rigorous that I've seen and I'm like I just wish this had been something which we had had the way to sort of take it further so because we were sort of in the downslope of time we're invalidated, I'm sort of happy that we learned a lot out of it and then waste more time than we needed to and we failed fast if you will but it was straight off and it's always slightly confusing when you have multiple ways you could go about doing things so that's mostly what I had and if any of you are looking to apply lean startup to a new idea either in small company, big company like I mentioned I'm now senior product manager with Intuit where I'm a part of a team that's looking at long term innovation and running experiments I'd love to talk to you and happy to sort of share different things that we're doing so we can all learn from each other on Twitter and my blogs if you want to connect I think we have I'll just do a quick time check I think we've got time for questions yeah 10 minutes so sure who's next after this yeah got you so the question is about the example you mentioned about the flip board and missing commercial opportunities so basically I mean in what situation would you like to use lean startup and when not do you have a kind of ideas about that yeah I mean see this is my opinion at this point so I feel that I found among other entrepreneurs too that lean startup has worked well where we're able to find sort of a strong signal for a problem that exists but in cases where a strong signal for a problem may not there's no strong signal for a problem like you know like to me I thought of the flip board example because if I spoke to 20-30 people who use Google Reader or one of the other dozen couple of dozen apps out there I don't know where the lean startup would have helped surface that potentially design thinking might have because I think with design thinking just how you immerse yourself just the user, the behaviors think about from a systems perspective something might have emerged so I may think more design thinking there if you have the luxury to go through the cycle and have the resources to go through that completely so what you're trying to say the problem statement is very clear that you may try to go through the design thinking is that what you're saying? I mean see I don't think customers will ever tell you exactly what the problem is so even with lean startup you know there's a certain aspect of qualitative analysis where you still want to sort of understand what they say but also think about what they do, what they think, what they feel and all four are related but definitely if the signal is not strong with lean startup you're more likely in those experiments to sort of pivot away and move on because you're running very short cycles trying to go through it very quickly and you don't have a lot of data and tools and don't have the time to do all that diligence yeah I think the vision is still there so let's see I think you had a question what happens when the MVP fails so it's a success and failure are a part of it so you run an experiment and if it succeeds or fails you treat both the same important thing is what you learn from it now suppose you succeed then you move from the last leap of faith assumption you made to the next highest leap of faith assumption so this was your biggest high-risk assumption up here you start with an experiment there if this succeeded you move on to the next high-risk assumption to test that out now if you got invalidated that's sort of when you've got to sort of think about should I pivot away or should I perceive it wrong there's a lot more context here I'll maybe talk to you offline a little more about that so really the answer is as short as possible so literally think in terms of days not weeks and work in a really tight loop some of it comes down to sort of the channels that you have for acquisition and the ability to turn things around quickly to learn from those users so think days not weeks another question is the entire experiment surrounded around the solution rather than the problem statement if you look at the entire hypothesis questions surrounded on the carrots as a fun option playing the carrots as a fun option not on the problem statement you wanted to look at the fun options but when we went through as a solution first as the carrots then you went through all of your hypothesis with these carrots only where we surrounded the solution not at the problem statement this is where I was I agree with you so sort of in places where we apply lean start-up later we refined our approach where we tried to step away to not just look at the solution aspect of it but to look at the problem and the vision a little more closely so that was definitely a learning the part of the cycle to not lose sight of the vision and the problem that we were trying to solve I agree with you completely can you give an example of local maxima that kind of got you trapped in a similar kind of example I don't think I have an example here I'll have to think a bit about other public examples that I can share not from my experience but I can answer this offline I just want to give it a bit of thought to make sure I'm framing it right if the signal is not strong enough pivot on it not necessarily it will depend a little bit so the pivot question is a little hard so see like I said most of the experiments you saw here were testing the value hypothesis right now if you go back to the business model canvas there are other things to test as well so you might want to consider whether you want to test other channels other channels are acquiring users you might want to think about are you really going after the right personas of people what are the personas you have in mind so I think it kind of depends a little bit there's a lot of jargon that gets tossed around like people who talk about zoom in pivot customer pivot different variant terms all of it pretty much comes down to hey experiment might have failed in this context and a little bit of it is a judgment call still at this point it's a bit of a judgment call that you as a leader need to make saying hey this is in workout maybe a broader sampling size do I want to test that do I want to test with a completely different audience do I want to test it out with a different set of incentives and I'll give you some examples like right now some of the experiments are running are platform experiments you know ecosystem marketplace type experiments where I've got multiple types of users one set contributing and another one consuming so I have to run these experiments separately with different groups but behavioral characteristics are very very different and I'm looking at long tail where there are a lot of different variances involved so I have to run various experiments before I can sort of figure out if it is right to pivot or not and in certain long tail cases they may not be a strong signal it might be a moderate signal too so I think it kind of depends if you're looking at a utility tool for instance I think you may want to look for a slightly stronger signal and part of the reason to why you may want to look for a stronger signal is sometimes you want to start off with a group of early adopters it doesn't mean that you won't get to the point where those people with a more moderate signal may come on board but you need to start off with you definitely want to start off with your ideal customer think of who your ideal customer is what that persona is like where he lives, what he breathes, what he eats what he does, what his life is like what he cares about and figure out how to go and find them you might find that your hypothesis is wrong and then you need to think of who else maybe is the right one so there's a bit of subjectivity which also makes this whole process a little bit hard because we can say it's scientific and objective in my opinion so I think in the context this is a start up so it is a fairly small team like less than 10 people in my current role here we've got a larger team so literally there's like teams of 30-40 people around it now the team that might actually be involved in running the experiments was still small even in a big company so largely you look at you've got to think of the PD team the engineering team and people who are comfortable with this sort of experimentation right typically if you've got guys who want to spend a lot of time polishing things and getting things right it will be a little harder for them because they would want to take a lot more time to build it out and the opportunity is to sort of accrue your technical debt or engineering debt questions will start to also get in the way of their personal aspirations and career motivations so I think figuring out people who think who maybe want to be an entrepreneur someday who have such ideas and recognize that hey it's about working in small groups to try things out that there is a part of goals somewhere at the end of that rainbow but chances are low and we are doing a number of things that are very experimental very innovative chances of failure are high I feel that that's the piece of it sorry at my current I'm talking about the audience who we experimented the team who built it largely through people we knew mostly that definitely drove greater collaboration but we had one person who was not connected at all I don't know if that helps maybe I can take it offline and the problem first and then the solution then you would still hypothesize the problem statement so both have their own importance you would definitely hypothesize the problem part first and then go on to run your experiments on the solution part so in this particular thing was enough importance done or already paid to hypothesize the vision that you had or the problem statement the problem statement yes at a broad level mostly qualitative data are talking to a number of people we spoke to like tons like more than 50 so people agreed that that's the problem and then it began to hypothesize the solution but what we didn't do which I think is a valid point is tie things back to the problem tying it back to the we didn't do that later since embrace that a lot more so probably always validating the vision with each of the hypothesis that solution you had to keep it in mind and to keep in mind when you're pivoting that you could pivot away to something else or works towards that you know but yeah start with the problem and then on to the solution there's a question there who's next I'll let it finish to park the problem see there wasn't competition as much as there was another player so really Zynga has draw something Zynga has draw something Zynga had acquired a company called OMGPOP which had built it for like 300 or 400 million dollars and draw something wildly popular and they had both a web sort of live dictionary flow as well as an async model where people could play with the app so there were a few reference point where similar things had succeeded now this wasn't necessarily we didn't see I tend to think about it I don't worry about competition I feel that distracts you yeah no it is but I try to I try to validate it by looking for customer pain and if that's there I go from it now I think it's useful to understand why others may not have succeeded or why no one else has done it people have tried it and failed but for me personally I feel that it's important to not get too bothered by that like for instance you could build something someone could be copying you how much should you bother about them I'd rather focus on innovation from my perspective on that I think you said one thing which is did I answer your question fully did I answer your question sorry so essentially we had we went through a period of four months and mostly it was two people working on it most of the period was two people working on it in between there were others who were involved pretty much two people so I pretty much built most of the web app myself I'm quite hands on I actually built iOS apps, Android apps and web so I built most of it myself with getting help from another team member to build out some of the other pieces so yeah, fairly light small team going through most of the pieces you might not be so you might be comfortable with the idea of sharing the product with your friends or friends or something but do you really suggest that people should validate the idea with the community that they know because the idea can get stolen as well yeah so this is a common question that I think I hear from entrepreneurs about protecting your idea see at least the current thinking that most people have is I'm sure your idea is good I'm sure your idea is valuable now if someone else if you really are on to something very very very unique then maybe people want to do it in some cover secrecy for certain reasons but in most cases what I've seen with entrepreneurs is those are ideas which are good but it's still not something most of the people would care about I know this is hard to accept I know it's hard to accept but most of the other bigger companies out there but still most likely not either have the time or focus to go after that if you think about it most often that's the case I'll give you a little example for you to just chew on how much is Google innovated in the last 5 years pretty much every innovation you think of even though you think of Google as an innovative company has come through an acquisition YouTube came through an acquisition Android came through an acquisition self-driving cars came through an acquisition so don't worry about the big companies at least in the space if you are a big company then you've really got to think of innovation because you want to get ahead of the small guys who are able to innovate much better than you do what about your peers who are also into the market and they also want to build something else I think we can take this offline I have certain views on competition so your philosophy to fund it or your philosophy to approach for funding Amir does Lean startup also have some way to do that or that's generic across any approach that you take I'm not aware of the thought process on that I can check and get back to you from others I'm not the business model of Canvas has things that you can test out with respect to monetization, revenue and some of those pieces I'm not aware I'll have to ask and find out I don't have an answer I don't know if anyone else does but please examples I think given this to me was a good example where I think I'll give you an example in the early 2000s I was a part of the valley couple of different startups where we built products and if we had gone about doing this and even back then we were quite agile I mean we worked through small cycles but the sort of refinement of thinking of why are we building this do people really need it who needs it was not there we sort of had the engineering side of it which is oh yeah to build it to go through the quick cycles to build it out but that refinement wasn't there fairly successful and it really saw the saw it today so such an example where it became a huge success yeah so I don't think there are good examples to be completely honest with you I don't think there are good examples people will tell you that there are examples I don't completely believe them yet I think this whole space and this thought process around how do we do this is still maturing we're part of you know fortunate we're part of a growing community that's trying to solve for this it's still not great that people have succeeded and then said oh yeah I'm usually in startup but I'm not sure yeah I'm not as sure there are some examples of people but I wouldn't quote them quite yet I'm not so as an example I mean buffer Joelle a buffer has blogged about things he has done and I think they have achieved you know success but I don't quite know sort of what happened when because sometimes people do completely candidate people will latch on to it and say hey I did lean startup this is a bit of a buzzword almost now so we've got to move beyond those buzzwords share what we do openly between each other and think of this as ways where we can constructively help each other to refine how we think about why we do what we do and to build good products through that we have a great opportunity sitting here in India to drive such innovation hopefully we'll all be a part of that sorry last question what are the challenges when it comes to execution or what traits we should have in the execution that you know it becomes successful I think an open honest mind be as unassuming as possible try to recognize your biases and within your team try to play off against the biases so that you look out for each other's biases without getting into groupthink with groupthink 2 is not good I think that's one so really a learning mindset being willing to learn on the other hand that bias towards action versus bias towards reflection it's important you're a startup you're a small innovation team you've got to be scrappy you've got to do things in a fairly rapid way to try to get to the crux of some of these problems because you can't run a research team that's going to go through a couple of years in a very academic way the answer is not to move to a very academic model so to still things run things in a fairly light somewhat scrappy way but to still have a slightly more refined thought process around why you're doing it so that you're learning along the way so to me it's that willingness to sort of question why are we doing this to challenge some of those assumptions and to have a little bit of creativity in crafting lightweight experiments and to have the discipline and rigor to test out things by managing some of those control variables so for instance there's no point if you run one experiment with like you're doing an A B test with two completely groups of people where the characteristics are so different you're comparing apples and oranges so is it validated or not validated how do you know so with an experiment you've got a test with multiple batches and play around with those control variables a little bit so it's a combination of few different things like it's a little hard to pin down but it's a little honest with yourself trying to stay objective learn from each other and having a tight rapid loop is sort of a must I think we're done I'm around and you guys are my contacts thank you