 So I'm so excited to be here speaking at a developer conference because a lot of times lately when I give talks to business people, you guys are kind of my people because I am a developer. I got my start with startups which is kind of what I know and love. I have also done development for some big companies like Oracle and Microsoft and helping them with how to be more lean and agile. But really what gets me excited is working with people who are really pushing the edge on what's possible. And so for the past few years I've managed to work with literally hundreds of early stage tech startups and just really focused on helping hackers and entrepreneurs who are building really innovative disruptive products with how to be successful at market. So that's what I'm going to share with you guys today, whether you're building a new product for a big company that wants to enter a new space or whether you've got your big idea for the next great thing and want to go do your own startup. So in thinking about innovation, the first rule of innovation is there are no rules of innovation because if you're doing it right, then you're breaking more rules than you're following. And if you're going to be innovating, then you're going to be trying a lot of different things. Thomas Edison famously said, I have not failed. I've just found 10,000 ways that won't work. But don't worry, it's not a complete black hole. So what I wanted to do today is share some patterns that I've seen innovators who are successful do time and time again and kind of walk through these and give you some examples of how innovators have done them. So the first one is eliminate waste, which if anybody does anything with lean, this is not going to be anything new to you. There's actually this lean startup methodology which has tried to take the principles of lean and apply them to how to do a startup. And Eric Ries, who's the creator of the lean startup methodology, says the number one most important thing for a startup is learning to tell the difference between value and waste, which is kind of weird, right? Why would you not know the difference between what adds value and what is wasteful? But of course it actually comes from lean from the Japanese term muda, which doesn't just mean something that's never going to add value, but eliminating anything that isn't explicitly adding value today. And when you're innovating, think of Thomas Edison, you're going to be trying a lot of things and finding out what doesn't work. And if you are trying to do this stuff that you think you're going to need tomorrow, you don't know what tomorrow is going to hold yet. And so if you wind up going over here and you build all this stuff over here, then that is wasteful. Or for any XP, Extreme Programming folks in the room, just think Yagney, you ain't going to need it. So the truth is a lot of companies think that they're going to do one thing, and then they want to do things very different. So to give some examples, PayPal started off as a way to beam payments between PDAs. But it turned out that the world was not yet ready for mobile payments in 1999. And Flickr actually started as a massively multiplayer online role-playing game. They actually built out the game, and then it turned out that the most fun feature of the game was sharing photos. And Instagram started as a gamified Foursquare, and again, they built out the entire iPhone app. I don't even know if they put it in front of anybody, though. They just looked at it and said, oh, my God, there's too much shit going on here. They threw the entire thing out and just focused on photos, which they thought was going to be the most effective. So these are examples of companies that are successful today. But when you look at the stats, it's pretty bleak. The stats are that nine out of 10 new products fail, which is pretty abysmal. Specifically, when you look at tech products, they tend to fail not because the technology doesn't work, but because people have created something that there isn't a real market for. A great example of this, or a scary example of this, depending on how you want to look at it, was this company Actuality Systems. They were actually here in Boston, so some of you guys might have heard of them. They created a holographic 3D display. Okay, that's pretty badass, right? They actually created it. They got it working. And then they spent the next 10 years trying to find a customer for it that would let them have a sustainable business. And in the end, they finally had to shut down. And all that they could do was really sell off the license for the technology. And so the question is, were they successful? I mean, in innovation, they created this thing that had never been created before that was absolutely amazing. And so from that perspective, absolutely yes. But then they weren't able to actually make a company out of it and they spent 10 years of their life trying and not succeeding. And so I'd like to see technologies like this do a little bit better. So it's kind of interesting, there was a survey done into what is the single biggest predictor of failure amongst startups? And what was found sticking to the initial business plan? So this is kind of scary, right? Because when you're trying, when you're doing any new venture and you're trying to figure out if you're on track or not, even that word on track is like how are you tracking to plan? And so if staying on plan actually means you're going to fail, it gets kind of confusing, right? And so that brings us to pattern number two, which is to start small. Which is usually the exact opposite of what I see people do when they're all excited about they've got this next big thing they wanna go off and build. And so hackers are like, I wanna build this end-all device that's going to do everything, right? And entrepreneurs say, I'm gonna build the next Facebook and get a million users and make a billion gazillion dollars, right? Go big or go home, baby. But the question is, how do you go from this big vision that you've got? We have nothing else. But your big vision to a million billion dollars or users or devices or whatever it is. I mean, Facebook today has over a billion users. How would you even create something that from day one would have enough features to be appealing to a billion different users? And the other thing is that would you guys wanna get on the next Facebook, if nobody else you knew was using it? Most people wanna wait and see, right? And so when I think about kind of this quest for how do we come up with an innovative product that's gonna be successful, I think of it as a search for the intersection between our big vision of what we wanna do and what reality can actually accommodate today. So a lot of companies wind up starting small in order to do some experiments. A great example, Microsoft, not doing so great with their phone, but I think everyone would agree they're pretty big doing okay. They actually started with basic for the Altair. Now, I don't know how many Altair computers were ever made, but I'm thinking like a few thousand, it wasn't very big, right? Or even Facebook, which is sort of our quintessential example of going big. They started at Harvard and there's only 20,000 students at Harvard. And so it feels very unintuitive, but this is the way that you wanna be thinking about things. Start from your big vision, you wanna have that big vision, that's great. But then go small, figure out a way to dominate a small market. And then you can build on that success to go bigger. There's several reasons that you wanna do this. One is, of course, what we already said that if you're innovating, you're gonna find 10,000 ways that won't work. And so you don't want to go big and spend a year or 10 years building something out and then it doesn't work and you're like, okay, that was all the time and money and patience that I have for that. So small experiments are good. But even beyond that, by going small, you can go after this very niche market. And all you have to do then is find these people that really want something bad. And find that one thing that they're dying for, and then just build that one thing. You don't need to build a billion features to be exciting to them. So Altair users really wanted a way to program their computers. I mean, I think all they had were like blink and lights and toggle switches, right? And then Harvard students really wanted a single centralized student directory, so Facebook was able to build that out without building all these features that they have today. So this is how you want to be thinking about things like Microsoft, like Facebook, like actually most successful companies actually start by going small with a very small market and taking that to billion gazillion. So of course, in order to be able to dominate the small market and understand that one thing that they're dying to have, you really need to be able to deeply understand your customers. And I would even go so far as to say, if you're entering a space where there's like a bunch of competitors as there almost always tends to be, this is the number one way to dominate that space. Because if there's a bunch of different people entering the same space at the same time, he who best understands his customers is going to be able to create the solution that best resonates with them and likely going to win. A really great example of this is Dropbox. This is a bit of a conversation that Drew Houston had with some BCs when he was starting Dropbox. And they said, I don't even understand why you're entering this space. There's already a million billion cloud storage startups out there. And Drew said, yeah, but do you use any of them? So what Drew did that all the other competitors that were trying to enter the space didn't, is he actually understood who his customer was. I think because he was one of them, and he built it specifically for them. So he understood that his customer was, or at least his initial customer, that small market, was the hardcore techies who have a bunch of devices that have the problem that you need to transfer files between them. He understood what they wanted so well that he put up this web page, very exciting web page here. You can see this was built in 2007. It looks a little bit like 1990s web page, right? He actually, he built out the tech, he did a proof of concept on the technology because he was doing tech that hadn't existed before. He had to make sure he could get it working. But once he figured that out, he didn't go build the full product. He put together this mock-up prototype, and he did this really scrappy three minute video. I don't know if anyone has seen this, you can still find it on the net. Three minute screencast of him walking through this mock-up with him as a voiceover, nothing flashy, nothing exciting. He posted it to Hacker News, so one place. And overnight, he got 75,000 sign-ups, right? Now you think, okay, well today, if Dropbox put up something and got 75,000 sign-ups, that would be kind of impressive. But they've got like 300 million users, they can do that. When he put this out, nobody knew what Dropbox was because it didn't exist yet. And he put it one place because he knew exactly who his market was. And overnight, 75,000 sign-ups, so it's pretty astounding. So I feel like there is this myth around innovation, especially in the last couple of years with so much hype around startups. Everybody wants to do a startup. And there's this, these stories that get told that make it look like, if we build it, they will come. But when you really dig deep into what's going on, what you find, or these stories of these founders, the founders who are really successful, have actually gone to extraordinary lengths to really understand their customers. So Airbnb, I guess having some legal problems lately, but still doing pretty good. One of their co-founders did not rent or own his own home. He just went around and stayed in Airbnbs. I can't even imagine, I guess he lived out of a suitcase. But how's that for understanding your market? Or Ben Silverman, the founder of Pinterest. He actually personally reached out to the first 5000 users and invited them to coffee and gave him his cell phone number. But can you even imagine? And so the thing that you want to be thinking about as you're going off and you're talking to all these people and you're proving at your technology, is that you always need to be learning. In fact, a lot of this comes from sort of lean startup if people are familiar with that, but they say for innovation, what you want to be thinking of is, how can you be scientific about this? So I'm not a scientist. I don't know a lot about science, what they do. But as I understand, when they want to prove something, they come up with a hypothesis. And then they do a test to either validate or invalidate the hypotheses. And when you are trying to innovate, when you're trying to come up with this new product that you want to be successful in the market, you have nothing but a bunch of guesses or hypotheses, if you want to be fancy about it. And so the question is, what can you do to test if your guesses are right or not without going and building the entire product and then possibly failing, right? What are the small tests that you can be doing to be constantly learning what works and what doesn't? So initially, you can do some of these tests by talking to people. You know, the holographic display, I think they would have done really well if they just, I assume that their customers were commercial businesses. If they'd just gone out and talked to people in different industries and understood better, what is your budget? What are some problems that you have today that maybe could be dealt with with a 3D display? Drew Hauston from Dropbox could have, and I'm sure that he did go out and talk to people and say, okay, what devices are you using today? And what are you doing today to transfer files between those devices? And that's really good information to have because carrying around at USB sticks sucks, right? And so all you need to do is come up with a solution that's better than that. It doesn't have to be super elegant. It just has to be better than what people are doing today. But here's the thing, and this is a mistake I see a lot of people make, is talk to people absolutely to understand sort of what problems they're having and what they've done in the past. But don't say, hey, I've got this great idea, isn't it a great idea? Because people get caught up in your excitement, in your passion. And so of course they're going to say, yes, that's a fantastic idea. My god, you're building a holographic display, yes, I want one. And you say, would you buy it? And they want to encourage you. And they say, yes. And then you hand them a device, and I mean, I don't know the details, but say it's $50,000 and you have to actually write this real-time 3D rendering code to get anything to output to it. And they're like, oh, are you serious about that? So you don't want to your validation for if people will use a product or if they'll buy it, should not be asking them. It should be actually putting something in front of them. So I like this quote from Linus Travals who says, talk is cheap, show me the code. In startup world, we have this phrase which I kind of love and hate because I feel like it's overused like a buzzword. It's called minimum viable product or MVP for short. And the idea here is don't build out the entire product, but figure out what that minimum viable thing is that you can get in front of that user to actually view what their behavior is. And usually you want to put a lot of analytics on that or some things so that you can actually see what they do with it. So we've already seen one example of it. Kind of the quintessential MVP is no code required at all. It's something called a landing page. That's what Dropbox did. So you put up a description of your product or maybe a video on it and then a place for people to enter their email address. And you put some analytics on it so you can see how many people you're able to get to the site and how many people are signing up. To give you kind of walk through a little bit more elaborate of an example of this, there's a company called Buffer. They do social media sharing and so say you like to tweet a lot and you go and you, like I like to read all my news at like 2 AM or something on Sunday night cuz I'm procrastinating. I don't like to go to bed. And so I'll find these ten awesome stories to share and I can tweet them all out right now but like nobody but me is up. And the people who are up are probably like why are you spamming me right now Abby with all of this stuff. So I love them because I can put them in this queue and they'll kind of buffer or post them out over the week, which is really nice. So this is how they look today. When the founder just had an idea for whether this would work or not, he put up a landing page. It was a simple concept, so he didn't need a video. He just described what the initial version would do, which would be just for Twitter. If people are interested, they can click the plans and pricing button. And when they click it, you get a hello, you caught us before we're ready but leave us your email address. Tweeted that out, people clicked through, they left their email address, he said okay, that's pretty good validation. But not enough for me to go build it because I wanna know if I can actually make a sustainable viable business out of this. So I wanna know if people will actually pay for it. So all he did was add a middle page to his landing page. When they clicked plans and pricing, they got three plans. One was free, but two were paid. People kept clicking through, most people clicked free. But a couple clicked paid and he said, you know what, this is good enough not for me to quit my day job and spend six months heads down building it, but good enough for me to put out that first really minimalistic version and see if people actually use it. And so that's what he did and very quickly, it took him a week and a half, I think, to get the initial version. It was very minimal, but very quickly he had 500 users using it. He even had some revenue coming in, so now he was able to actually evolve the product along with seeing how users were actually using it. Another obviously great example, especially if you're doing any physical products like hardware, smartwatches are all the rage these days is to do a crowdfunding campaign like Kickstarter and actually get people to pay for it before you go manufacture it. Now again, whenever you're doing something that's technologically new, you want to make sure that you can actually get the tech to work. But you don't need to build out the final end all piece. You can do a prototype, make sure it's working. And then if there is enough interest, because you get these pre-orders, then you actually have the money to go manufacture it. So as we like to say in the startup world, these are all great ideas, but ideas are kind of a dime a dozen. It's all about execution. So how do you focus and get shit done? So I want to spend the second half of this talk talking about what are some development practices that we can use to make sure that we're focusing on the right thing. So make sure that we're focusing on things that add value, not waste, and allow us to move very quickly while continually learning and adapting. So the first thing is just get into this mindset, and this is really painful, but of minimum viable everything. Like you're really excited about the idea. You want to go build this huge thing, but stop and wait. And what is the minimum thing that you can do to kind of get to the next stage? So we've already seen examples of this. If my assumption or hypothesis is if I build this product, people will want it, great. Do a landing page. That's a great way to test it. If your assumption is if I sell this product, people will buy it, you can add a pricing to your landing page, or you can do a Kickstarter for it. Now, if you want to build the product, I think the next thing you're probably assuming that you want to test out is, OK, I have identified who my customer is and what that one thing is they're dying for. And so if I build this particular solution, I'm thinking of that people will use it. So here, what you want to do is think about what Buffer did and think about, OK, what's kind of the absolute bare bones minimal feature set that I can put out? And they kind of say, if you're not embarrassed by your first release, then you waited too long to put it out there. So I don't actually know all the details, but I took some guesses at what Buffer's initial minimum feature set might have been. Obviously, people need to be able to create a user account, link it to their social media networks, add posts to the Buffer, and then edit and delete them, add what your posting schedule is when you want the tweets to go out. Buffer needs to then automatically be able to post from your Buffer to the schedule, and then just let you view a history of the post. So that felt pretty minimal. The thing that you always want to kind of do is go look at, OK, I think this is my best absolute minimal thing. Ask yourself one more time, like, what is it that I'm trying to prove? And do I really need all of these things for it? And so again, I'm making guesses, but I think that what Buffer initially wanted to see is, OK, people will download the app, but there's a big issue with people download apps all the time and never use them. Will they actually post something into the Buffer? And if that's what they're trying to prove, then some of this stuff can go away, at least initially. So if you haven't had anybody post the Buffer yet, you don't need to allow them to view a history of their post, or even edit or delete their posts, and that's painful. But if you see people are using it, then you can kind of very quickly add some of these simple features. Similarly, if you're just trying to address the issue that they don't want to post all of their tweets at 2 a.m. on Sunday, you don't necessarily need to let them set up their own schedule. You could hard code something in and say, this is based on when we found, or the most popular times for people to look at Twitter, people are going to be happy with that. Again, you're just trying to see if they're going to post or not. And then, of course, you don't really need to add Facebook and LinkedIn. Just start with one, whichever one you think is going to be most important. See if that works or not. So that's sort of how the process looks. Now we've gotten down from what we thought was the absolute minimal to only four of those items. And you can get that out pretty fast and see if people are going to be using it. So, OK, you know what features you want to build in now. Now what is kind of your minimum viable software development process to use for this? I'm kind of an agile girl. That's my background. I remember when Lean Startup first came out, I was asked to present at a Scrum Club on that Lean Startup. And I was really excited about it. And I was at, actually, a Lean Startup Meetup. And I was bragging about it. I was like, oh, I'm so awesome. I'm going to go present about Lean Startup to Scrum Club. And one of the organizers of the Lean Startup Meetup looks at me horrified and goes, Scrum? Oh my god. I hate Scrum. It's so heavyweight. And I felt like a really big dummy. I was really embarrassed. But it was a big learning moment for me. And I've noticed that many of my big learning moments have come when someone's kind of screamed at me and may be really embarrassed. I don't know. Maybe that's how I learned things. But I realized who is really right. I mean, when you're innovating on something, you are going at such a different pace and at such a different scale than we're used to in software development that Scrum just isn't even built for this. It doesn't even make sense for this. And if you try to do something like Scrum, it's going to slow you down. And I don't necessarily mean all aspects of Scrum. I think there's still a lot of good theories from this. But in particular, the time boxes that Scrum sets up. In Scrum, you have these sprints that are one week or they're two weeks. And it's really arbitrary. And honestly, two weeks can be a lifetime in the beginning. It's way too long. So I just want to see if anyone's interested. I just want to put up a landing page that takes me. There are tools that you can do this without even coding. You can do it in an hour, right? I just want to do it and put it up there in an hour. Why would I wait two weeks or even a week for the sprint to end? It doesn't make sense. And then to be constantly looking at how our users responding and be able to adapt to that immediately. And so I put up the landing page and I'm looking at the stats and I notice that people are coming to it. But no one's leaving their email address. I say, OK, I should try to put a different message on the page. Maybe that'll get more email address signups. Again, I can just do that in 10 minutes. I don't need to wait for a time box. Even when I get to the point where, OK, I figured out that first version of the product, that's those four features and I'm going to build them, well, say it takes me two days or a week and a half, whatever it takes me, once it's done, I should just be able to post that up. It doesn't make any sense to put it in a time box. Time boxes make sense when you're kind of building towards this actual list of known features. They don't make sense with when you're at this stage and you're trying to figure out those 10,000 things that don't work. So what I like instead is Kanban. So you can start with sort of a simple task board. I'm sure you guys have all seen this before. What's our backlog? What are we working on? What do we have? What have we finished? A problem that I see with startups just do this is that they're in progress. Column starts to look like that very quickly, which is particularly bad when you realize there's like one person working on it. And it's tough because you have, especially if you're trying to actually make a business out of this, even if it's just you're doing it for yourself, you're excited. You have all these different things that you want to be able to build into it. And then you talk to people, and they have ideas. You talk to investors, and they're like, if you build in X, Y, and Z, it'll totally be fundable. And you're like, oh my god, we better stop everything and do that. And if you talk to customers, or potential customers, you're like, oh, if you just built these features, then we would buy it. And they're like, oh my god, money. I should build in these. And pretty soon you have something that looks like this and nothing is getting done. And so what we're missing from Scrum that was really nice was Scrum had these nice sprints where you'd actually have to finish everything in the sprint before you can then go work on the next items from the backlog. And that was really nice. So we want to still have that constraint in place, but we don't want it time-based. We want it more task-based based on our capacity. So if there's just one of you or maybe there's just one developer on it, you put a work in progress limit on that column of one, meaning everything else has to wait. If there's two of you, then the work in progress limit would be two. So this way you make sure that you're actually completing those things before you can go work on anything else. One thing that might strike you here is obviously I just talked about how you have to be able to move really fast and be really flexible. If that one item is like a two week or three month or whatever long item, you obviously can't wait that long to do anything else. So I would keep these really short. It's a little different for everybody. But I usually recommend that places start off with one day, keeping the tasks less than or equal to one day. The beautiful thing about that is then every single day something completed. And it has this momentum that's going forward. And usually no matter what's going on, it can wait until the next day to pull in the next thing. And while you're working on that, the to-do, unlike Scrum, this to-do backlog can be moved around. Priority order can be changed to whatever you want. It just matters. Once you start working on it, it just needs to stay the same until it's done. But I'm looking at that to-do column. And that still looks pretty overwhelming to me, right? I mean, would you guys be like, oh my God, there's so much to do? And I feel like when you get to do columns this long, first off, if you're really early and you're still trying to figure out what the right features are for the customer, how do you even know what all that stuff is? That's like a bunch of guesses, right? And you should be just getting little things out in front of the customer to kind of figure out what the next thing is. So first off, a lot of that could possibly change. Second off, like if I'm the developer and I'm working on A and I see all these different things, I'm like, oh my God, when I'm doing A, I need to think about how B and C and D and E and F and G and blah are gonna like work into this. And you get overwhelmed and it slows you down. And so I say, you know, if you want to write all these ideas down, great, put them in a spreadsheet, put them in an op-up or whatever, but put a limit on what you can put in that to do column as well. And I would put, you know, at a maximum, how much you can do in like a week or two. Get rid of everything else, that's a maximum, not a minimum, so it can be less. And then you are hyper-focused on just what are we trying to do this week, right? And it's helpful if it's just you, it's helpful if you have a team of people, everybody's kind of focusing on the same thing. So okay, now you're gonna go start developing it. You wanna think about what's our minimum viable design, which of course we would really call emergent design. I actually, for anyone who has been here for Katrina, has talked very nicely about how to refactor, which is great. You know, this is another thing. If you keep thinking Yagnit, you ain't gonna need it, you don't know what's gonna come tomorrow, so don't try to build out this architecture that's gonna scale to a million users when you have zero users. The truth is, a lot of startups build a product, it's totally wrong, they throw it out, they build a product, throw it out, build a product, throw it out. So you don't need this built up architecture, you're just wasting time, and that's sort of what Eric Rees was talking about, when he thinks about the difference between value and waste. Similarly, you don't need to go design, for we like to follow all these principles, like okay, I'm gonna hook Buffer up to Twitter, but I know other social networks are coming in, so I better do this abstraction layer so that it can then easily be replaced with Facebook or LinkedIn, but in order to do that, I kinda have to research a little bit about the different social networks we might use and the differences between them and how you interact with them, and then I have to figure out the abstraction layer, which is really hard to do, until I built one and oh my god, just do it for Twitter. And then, if later on you determine, okay, we're gonna need to add another social network in, great. Refactor it, figure out what the best design is for it at that point. Just keep thinking Yagney, like very aggressively, much more than you might on a normal project. So okay, you're developing the code, you're moving really fast. I know that this looks a little bit water folly here, but these are kind of generally the steps that you're gonna be doing when you're doing development. And if we think about okay, if a big success factor, a big indicator of success, on whether or not our products are gonna be accepted in market or not, is understanding whether the customer is gonna like them or not, then in kind of learning that, then we wanna look at this and say, okay, where is our learning occurring? And so, there's some learning that occurs when we do requirements, because hopefully we've gone out and we've talked to customers, we kind of deeply understand what their problem is, and we're trying to come up with something to solve for that. But really, most of the learning about our product happens when you release something, put it in their hands and see how they actually use it. And unfortunately, the place where we spend the majority of our time, and as developers is often the most fun, there's very little learning that happens here. And so, the question is, how do we maximize our chances for success? How can we maximize this learning that's going on? So another way to look at that is to say kind of how do we shorten the timeframe between when learning occurs? And so instead of doing a whole bunch of features and batching them up and putting them out, a great thing to be doing is continuous deployment. I put a little smiley face, because I'm very happy that there's like a whole continuous deployment track here today. So I won't go into a whole bunch of details on that, but I did want to kind of talk about how some startups are using it. So Etsy is fantastic. If anybody's doing continuous deployment, they've got a terrific blog that talks about what they're doing. They're currently deploying 50 times a day. Actually, I got that stat about a year ago, so it might even be more now. But anyway, in 2011, they deployed 10,000 times to production amongst 100 engineers, which is pretty extraordinary. And what they said is, ever since we started deploying so many times, ever since we started deploying 10,000 times, our system is so much more stable because we've already done this 9,999 times. Like, we got this. And also, when you think about you want to be in this mode of a scientist where you're doing experiments, Etsy also said, you know, this allows them to experiment so much more. So before, when they were only releasing once every few weeks or maybe once a month, they were constantly having to pare down what actually went out because they were like frantically trying to make the release. Now, once they started doing like over 30 times a day or something crazy, they're like, this allows us to do things like throw a feature out there in like a minimum version, see if people actually use it or not. If they do, we can either pull it back and kind of build it out or we can just quickly add features to it. It just gives us a lot more flexibility. Another example that I like a lot is Ash Moria. He has done a lot in the lean startup space. He was actually one of the really early lean startup practitioners when there was just theory and he's like, okay, I'm actually trying to do this for a startup. How does this really work? And he had this great blog which he's since turned into a book about kind of his experiences. And so I feel like, so his company at the time was called Wired Reach. I feel like they were one of the early practitioners of continuous deployment and he just kind of compared his before and after experience with it. And what he said is, before they were doing two week release cycles, after they were releasing multiple times a day. Before releases were all day events, like I know I've been there before when you're like, oh my God, and it's frantic and the whole team's like trying to make sure it works right. Now that they release all the time, releases are non-events. Before releases were 100 lines of code, now releases are under 25 lines of code. And before they had a lot more emergency releases because they'd only put something out every couple of weeks. If something didn't work, like it was such a big deal to have to go release something again because it was like an all day thing to release it and they had just deployed such a huge amount of code that it was really hard to find where the error was. Now, it was no big deal. Like if they pushed out a release, it was like a couple lines of code. So A, they could either just pull it back in or go look at those couple lines of code and find the bug in like two seconds and do another release. So it was no big deal. So I really liked kind of that before and after and how much easier it makes things, which I don't know, I feel like it's unintuitive. You feel like it's gonna make things harder and it actually makes things easier. So I kind of like, I feel like I've heard that a lot in Agile. Like if you're not good at something, just do it a lot, right? And then you get really good at it. And then one other thing that Ash Moria does, which I really like and I feel like continuous deployment makes this a lot easier, is he actually adds a validated learning column to his Kanban board. And the idea is that a feature isn't actually considered done until they've put it in front of a user and actually watched how that user behaves with it, either in person or through analytics that are going on. And I really like this because now it actually builds learning explicitly into the feedback cycle. And the idea is hopefully that learning that you get is now going to feed, figuring out what your next features are. So that makes a lot of sense. So I just got the 10 minute warning, but I guess I'm a little bit ahead of time, so we'll have time for questions. These are the patterns that I have seen time and again for successful innovators. And thank you for your time. So yes, so questions, yeah. I'm sorry. You didn't want my cool hacker check up there? This is kind of just to add on to your Kanban. We use Kanban, I love it. I think it's definitely better than Scrum. One thing that you mentioned the learning column, we have sort of a review column. And I find that really helpful because it's like this pause in the development where people think that they're done, but it's nice to have a time for reviewing the pull request, making sure the PO's are on board, making sure QE's on board. And I think that's really valuable. And with Kanban, you can kind of add in whatever you want. Yeah, absolutely. So how do you determine when something is ready to move out of the review column? When something's ready, it's when the developer thinks that they are finished with the feature and they put it in review and that's when everybody basically reviews the pull requests and comments on it. And only after then, actually our QE department then is the one that's responsible for merging it into the release because they're also kind of watching the pull request and they can say, okay, this looks good, everybody's happy with it. And it kind of gives them the control over what goes into the release as well. Nice, thank you. Something I've seen people do is say basically, well, if it's not perfect when I put it out there, people will get the wrong impression about what it is and not try it later. How can you combat that when people say that? Because I worked with some people who were doing a startup for like two years and they never put anything in front of anybody and I was trying to convince them to actually put something in front of somebody and they're like, well, but then they might not use it later. Right, so I have several different answers to that. So I'll just run through a few of them. One is actually a personal pet peeve of mine. I said I kind of hate the term MVP, but it's like a love-pate relationship. I feel like people have taken the MVP, the minimum viable product concept, a little bit too far and I feel like people started creating MVP with crappy product and I don't actually advocate putting out a crappy product. So I just want to be clear on that. I feel like you could do a very minimal feature set but not have it a crappy experience. Another thing that I would say is, and it also, you know, it depends so much on who your user is. Different users have different tolerance levels for this so you have to be sensitive to that. But another thing that I would say is I hear sometimes startups say this concern when they have no users and I'm like, so who are you going to offend? You don't have any users today. There's no one to piss off. Maybe you'll actually get a couple by doing this. If it's that absolutely terrible, you had no users, you can create a new company name, right? I mean, that's extreme but you don't have any users yet, right? The third way I would answer this, however, and actually it's funny because I did a talk here for constant contact and I had a great slide but it's not here anymore. I have this great example of TripAdvisor doing something very similar to this. So TripAdvisor, actually I don't even know, they have a lot of users, right? Millions of users, I guess. They sometimes have ideas for new features but you know, they like this lean startup principle and they don't wanna go take the time to go build a feature until they know if people want it or not. And so with their hundreds of thousands or millions of users, they will do something. Sometimes they gave me an example, they're like, sometimes we think maybe what we should do is actually make hotel recommendations, right? You can go on TripAdvisor and you can see all the reviews but wouldn't it be nice if TripAdvisor just recommended one for you? They're like, I don't know, would people want that or not? And so they actually like just put a dialogue box up as soon as you loaded the TripAdvisor page, like you didn't even like click it into a hotel or anything like just boom, right in your face, super obnoxious. Do you want us to recommend a hotel for you? Yes, no. If they said yes, it actually gave them an error message saying system unavailable. Like it was totally ghetto, right? It wasn't even elegant. But the thing is they have so many users coming in that they were able to put that up for like two hours and get, I don't know, like 100,000 tests from that. And they were like, that's really good. And we pissed people off for two hours but we got this great test. And no one's, we're not losing customers over this, right? So those are my three very different ways of answering that. Does that help? Hey, Abby. I know Dropbox got lucky with posting in Hacker News and gaining 70,000 users just overnight. What are some other great ways of acquiring users? So what you really, I don't know that I would totally say that they got lucky. They did get lucky in the fact that they had a very easy way to reach their audience but I think they knew who their audience was and their audience were these heckies who happened to hang out. And so that is sort of the answer is figure out who your people are and where they hang out. If you don't know where they are, there's actually another story of Pinterest. Apparently when Pinterest first released, they had like 3,000 users very quickly because the team was very well connected. Like I think they'd been through iCombinator and stuff but I think like all of their users were actually like them and so like other young like 20-something techies who probably lived in the valley and I don't know, Pinterest's target market is probably like moms in the Midwest or something. I don't know. It was not the right target and so they had these users and nobody was using it and they're like, oh my God, I don't know how to reach my users. And so they tried to think of who would be the people who would use this type of thing and they realized that people create these, I don't know what they're called, vision boards or something, like physical boards where they do collages and they put up pictures that they like and things that inspire them and so they actually created pin it forward meetups. I think it was called. So physical meetups to bring people together who were interested in this stuff and what they said is not a lot of people came but the people who came, I guess they let them do the vision board and then let them do it online. They were so excited about it and that was kind of the momentum that they needed is to start reaching these passionate people who could then spread the word. So long answer but you need to just figure out who your user is and where they hang out. Yeah, what's a good way to control for false positives and validations? For example, say I built an app, I put it up on Facebook, I get a thousand likes, I feel great and then I check the next day and have zero signups. What do you think is a good way to control for false positives? Yeah, I think you just have to be careful about what you're actually proving because it is, I don't remember the stats but the stats are really abysmal like in the number of mobile apps say they get downloaded and never use and so that's almost a false positive too. And so kind of be very deliberate about what is this actually going to test? Is it going to actually test that people are just curious and will download it or can I actually put a feature in there and see if they use it or not like Buffer did with that first like week and a half release? Thanks very much. Thank you.