 2010's most recently an awful lot of focus on mobile as just at a company in Norway they do web-based selling online and they've been tracking the amount of browsing that comes in on mobile not the amount of sales but the amount of browsing that comes in on mobile and it has the same kind of trend that they saw almost a decade ago when their newspapers which were their want ads were stopping getting the same kind of traffic and stuff just started dribbling in over the internet the CEO of this company said if you're not heading towards mobile you're crazy yeah it looks kind of young today but it's where the future is going to happen so if we look at that we have to think about this concept of platforms so once upon a time when there was a marketplace it was somewhere in the center of a village and that's where stuff was brought and stuff was sold and then some more formal structures were put together so we saw we saw this is in Melbourne big central marketplace buildings where people brought their wares and sold them to each other and that sort of predates when we had platforms now we have platforms that are a little bit more interesting where people can exchange a lot more than just goods and services so we saw that the Amazon Web Services and other cloud services become platforms in I don't know early maybe five years ago maybe four years ago and these days there are companies that that have never seen anything but cloud services how many here use Dropbox okay they actually have no IT infrastructure they just plain use Amazon Web Services many new startup companies do that many companies moving to continuous deployment find it's much easier because of platforms like cloud services and so those IT departments that have these big server firms are pretty rapidly becoming obsolete and then we have the store platforms the things that are replacing the village square and the and the marketplace square we have app stores and Android stores and stuff like that so another place you need to be if you want to think about making money is in a platform environment there isn't much better way to make money than to have a platform with people selling on top of your platform you make just a little bit of money or maybe you make money from advertisements it's very hard to create a successful platform but if you happen to be lucky enough to create a successful platform you've almost created a revenue stream for a long time so where you want to think about is how can I get into this platform business now let's just go on and talk about IT because we're going to go back to that corporate computing world which a bunch of you may be in so here's a guy his name is Jeffrey Moore he wrote the book Crossing the chasm it's the classic marketing book about how technology gets adopted your early adopters and then there's his chasm and then there's later adopters and that sort of thing and he's written a bunch of books his last book is escape velocity it's kind of how to escape the pull of the past he spent most his entire career being a consultant to corporate enterprise level IT departments and he says if you're in a corporate IT department or supporting a corporate IT department you have to understand something IT is going away from the time in the last decades when we wrote systems of record so systems of record there are these highly structured knowledge stacks that where we keep track of stuff we keep track of transactions we keep track of customers we keep track of relationships we keep track of core businesses we keep track of all of that sort of stuff and we keep track of it with some pretty old technology but the people who are using the technology whether they're traders in a financial institution or buyers in a corporation when they go home what do they have they have all those platforms that we were talking about they have smartphones they have web access they have their kids have tablets they have all sorts of stuff and then when they get to work they expect the same easy communication to their peers and colleagues and works in their business environment as they have in their home environment and so anymore if you are in corporate IT he says you really have to be thinking about the experience of the people that are using the system because they're no longer going to be tolerating the kind of very clumsy overnight batch systems that they've had to live within the past not when they have such easy to use technology at home so you need to have some more loosely structured knowledge flows some some some stuff that is much easier for people to use I was I was working at a bank a couple weeks ago and they said that they discovered the people who were doing various types of analysis for trading no longer wanted the old spreadsheet kind of stuff they wanted stuff that looked just like the stuff that they had on their tablets at home they were no longer happy living with the old kind of experience that they had in the past so what we have is some drivers that are forcing technology and not just consumer technology also corporate technology towards a much more experience orientated environment if you think you can get by with very clumsy experiences think twice because there's so many competitors out there that can provide a better experience and I just want to talk about a certain company this is a consulting firm in Washington DC area their name is sphere of influence they they do mostly stuff for you know federal government Washington DC US federal government as most of their customers and they decided to do a survey and the survey was for let me see which of our customers like our products so they surveyed the customers and they found that the customers were satisfied but they said just one problem the clients were just satisfied that wasn't good enough agile's intense focus on customer priorities amplified by the grind of just-in-time delivery created an environment where we didn't have any time to create engaging experiences like ability even innovation into the things that we made we didn't have time we were just supposed to grind out one feature at a time and so our software kind of came across as if it was done one feature at a time whatever the customer happened to ask for which guess what is exactly the way agile more or less tells us to behave and exactly the way their software was being produced and they decided that that wasn't good enough that didn't give really likeable stuff so they chose to become a design firm rather than a software firm even though they were all hardcore software engineers so they started studying design what is design all about they read various books they went to different kinds of conferences and they came up with the idea that design is something that creates a product that is likable and in order to be that it has to have a strong center okay strong center so here we have a quote from fatty says engineers make things that work designers make things people like we have a soul of engineers but the heart of designers we're hardcore software engineers who are who find elegance and beauty and you do too elegance and beauty and things that are engineered and not just assembled and we've been just assembling software and pondering this we realize we weren't design people we were not engineering people we were product people we created products so once you make that meaningful shift from engineer to product person you see you don't have users anymore you have an audience now audiences don't care what you do they want to like what you do it's not about if you go to a theater is the production technically correct it's do I like it so you don't look for things features details in product you look for something that you like that you enjoy that's a great experience so they decided that's what they were going to do now interestingly enough that meant they had to tell their customers they were going to operate with them in a different way we realized our job was not to defer design to the client and transfer transform user stories and requirements into programming syntax we have a responsibility to invent technology and design engaging software by making decisions that clients and their users would never make that's our expertise we are software engineers and we understand how to design an experience that customers can't describe and so we no longer allowed features from customers if they inhibited the formation of a strong center they said no they said let's help you do something different but that's not something you want in the software because it will make it hard to use confusing it will mess up the elegance of the product we found ways to fill the needs without clearing up a product now that's a pretty bold move and they find today they don't know how to deal with all of their work because these days customers aren't just satisfied they are delighted with the products that this company makes now that's a challenge I'm laying down in front of you it's not just about doing what customers want it's about designing something that is really well designed I come from a family of architects and an occasional construction engineer and one thing that I know about architects is that they care about the way a building looks and feels in addition to the flow of people through the building in addition to the construction details all in one person and so if we go on from there I want to give you a couple of new definitions the first definition is the definition of design so when you think of design are you thinking of like UML and how do I design the code yes let's let's go to a college which teaches design they teach industrial design or they teach people how to design for advertisement that's not the idea that design brings to other people's mind why we get the idea that code design is design is interesting but it's not product design so let's talk about design as the purpose to fit a product to the humans who use it who own it and service it see the strong center okay that's design and design is what makes a product likeable and appreciated by the people who experience it that's a well-designed product do people like it do they enjoy it do they like let me give you an example one of my favorite websites recently and this was a couple years ago I still use it all time though is hip monk have you heard of hip monk it's it schedules airplanes and when it first came out it needed absolutely no advertising because its experience is so engaging and every time I help one of my relatives get an airplane flight I direct them to hip monk and they always say well I'm never going to use any other airplane reservation system again at least not to start with really well design concept design comes before we start pouring features into software design is the thing that makes our products likeable to other people that's design my definition actually not my definition that definition puts us in sync with the way the rest of the world thinks about good design requirements hmm let's talk about requirements that shear says requirements exist to constrain design elements they describe the critical few results that define value requirements are never a long list of things those are specifications requirements are the few things that constrain the design they are not descriptions of what to build so what are requirements let me give you an example so as I said that's me I did process control systems for many years and I didn't ever have things like requirement you know list of things to do or user stories or anything like that I'll show you what a requirement for a process control system and my company consisted of five things it is was a process control system for a new process made by the engineers upstairs and when the process was installed it had to make good product number one and by the way it had to be on time because in a manufacturing company having a control system for a new plant come late is very expensive my boss would not have kept his job if we were going to come in late so it had to be on time the operators that were going to use this thing to control the process had to find a convenient and by the way it was going to be maintained by the plant engineer some you know two or three thousand kilometers away not by us and so it had to be turned over to somebody who probably was good enough to understand basic and that was it but this guy was a plant engineer dealt with all kinds of stuff in the plant not just a control system for my product and by the way it should contain the latest technology because they always wanted to have the newest and greatest thing which is why I as a junior engineer was there because I could put some software in it and that was kind of a new concept at the time now everything else is a specification those are requirements and those things you call requirements I propose to you are decisions on how to make things but their decisions on specifications they are not requirements they requirements frame the design and put constraints on the design and define the value to customers so requirements are not designed this is a book by Tom Gilbert was written in 1988 it has gone through 20 printings very popular all about software engineering management and he's written several other books too and he's what the first person that discussed the whole concept of incremental software development in 1983 and he's been writing about it ever since so you can imagine that when this whole concept of agile came along he thought it was a pretty good deal people are listening to what I've been talking about for many years except he doesn't actually like it because there's a problem and his problem is that the whole concept of figuring out what the system is supposed to do used to be part of software develop part of software engineering back when I did software Tom did software we didn't have somebody come in and tell us how to what to do and set priorities and give us little detailed stories we were expected to engineer solutions to technical problems and so he says the worst scenario I can imagine is when we allow real customers users and our own sales people to dictate functions and features to the developers carefully disguised as customer requirements maybe even conveyed by our product owners if you go slightly below the surface of these false requirements you will find that they're not really requirements they're really bad amateur design for the real requirements so I call those specifications not requirements and I propose that if you have a long long list of specifications whether in the form of user stories or whether in the form of written things that other people might call requirements somebody else has figured out engineered the technical solution to the problem not the people who have the background and capability to figure out what a good technical design for that solution is and to be giving teams that level of detail of what they ought to be doing means they no longer are expected to be designing the solution to the problem and you know I agree with Tom that used to be something that software engineers always did and it still should be something that software engineers always do so another guy that's been around for a while his name is Fred Brooks he wrote the book the mythical man month it's classic software engineering book written over 30 years ago and Fred Brooks was the lead architect on the IBM 360 operating system and the mythical man months had much influence on software engineering for many years in about approximately three years ago he published the book called the design of design it was a bunch of his writings he became a professor and he had been writing and speaking in this area for many years and I want to show you three principles he has in this book one is solve the right problem deciding what to design is the hardest part of the design task secondly go beyond analysis and have some insight design isn't just to satisfy requirements bits to uncover requirements design isn't selecting from alternatives it's realizing alternatives exist and don't divorce design from implementation which we've done a long ways towards doing even in agile agile has a huge distance oftentimes between somebody designing the product and somebody implementing that design one of the most striking 20th century developments in the design disciplines has been the progressive divorce of the designer from both the user and the implementer and as a result we get a whole bunch of big disasters this is not design sorry this is just a bunch of thinking little story cards but that's not the design of a good product this is designed this is a proposal by sphere of influence to a government organization to put together a set of software analytical tools for containing insider threat and you see it has a strong center you can see what it's about and the concept of the picture and the proposal and also the experience of the software is going to be that focused centered on what's important and not a whole bunch of arbitrary stuff that people choose to put in so let's go and let's look at these three things from solve the right problem have insight and don't the voice design from implementation so solving the right problem here's some great solutions to the wrong problem how about that one okay or this one this is actually in 1896 out of the lo and fall who in did a whole bunch of glider flights in Germany he wrote the book called something like aviation as bird flight or something like that and he specified all kinds of aerodynamic numbers which basically created the tables of how lift was going to work unfortunately he was killed in an airplane clash in 1896 no a glider crash crash and the problem was that he didn't have a way to shift they controlled the gliders with shifting weight which didn't do such a good job every so often here's some more great solutions this is back when people were trying to decide what a horseless carriage was going to be this is it's going to be a bicycle maybe it's going to be a carriage maybe it's going to look like a train and have a steam engine and this interestingly enough is an electric car designed around 1900 by Mercedes Benz so nobody really knew what the car was going to be until Henry Ford said guess what it's going to be a model T Ford and that became the basis of what a car was going to be and by the way Henry Ford designed that first because he raced cars and he designed a car that he could win a race in let's go back to a really interesting case of how design works if you look at an area of study called lean product development not lean software development but lean product development they're focused very much on don't build until you understand those problems that you are don't design the details until you understand the problem you are trying to solve make sure you got the solutions figured out by testing before you create any detailed design so we in software development have said whoa big design up front is bad I guess what in lean product development they say whoa no design up front is bad okay there we're both right no design up front is just about as bad as big design up front maybe even worse so they're fighting a different problem and back in the 1890s after out of Lillinville was killed in the crash there were many people trying to invent heavier-than-air flight and they all built planes that would go up for three minutes and crash and destroyed and then they'd have to build another one and it would take a few years and so the Wright brothers said you know there's a couple things here one is we need more time in the air we we can't we had a test all sorts of things we can't be crashing all the time and the second thing they said was we don't actually care to die either so probably the biggest problem is how to control a plane once it's up there it's not getting it there in the air it's controlling it and so this is a picture from a patent application and because the Wright brothers in Dayton Ohio were bicycle mechanics that's what they were they the think of a bicycle tube and it comes in a like an oblong box and they took an oblong bicycle tube box same 110 years ago as it is now and Wilbur Wright he kind of bent it and he said you know if I make wings of an airplane like this box and I bend them I'll bet I could control the way the plane flies in the air and that concept changing the shape of the wings to control flight is the way all heavier than air airplane have worked ever since so they got that concept of control right and then they tested it out they built a kite they tried it with you know pulling strings to see if it would actually do some control and they were reasonably happy that they got the idea down so now having comfort with the fact that they could control the plane once it got up there they were a little bit unhappy with how long the plane was staying up they could make it land gracefully they could keep control of it the direction it went how it turned but they still only got a few minutes and they had done a ton of research and they could not understand why they couldn't go up in the strong winds off the ocean they went to North Carolina and stay flying for hours and they got like you know a few minutes so they tried to figure out what was wrong with their calculations and they built next year another one okay and here they are trying to make this thing stay in the air having got the when they built this they changed the shape because some of their friends said oh you probably didn't follow out of will until shape well enough and so they tried another one and while it was pretty controllable it wouldn't stay in the air it was like worse than the first one so they said okay either we're really dumb or somebody's got all those aerodynamic tables wrong and they proved to themselves with some quick experiments that actually the all the constants that had been published were wrong and then they spent a winter recreating all the known constants for aerodynamic lift by creating a wind tunnel and then you can see they have little tiny wing shapes and they put it on balances they gathered data and by the end of that winter they went from October to spring they knew more about how to keep an airplane in the air and what kinds of shape the wings should have than anybody in the world and they figured that out in this small wind tunnel that they built in their bicycle shop and the next year they built a plane and it really worked it stayed up it stayed up and it behaved the way that they calculated it should you can see there's a vertical tail there because they learned that they had to control the yaw and the role of the plane that was a new concept but they had enough time in the air they started discovering those kinds of concepts and so now having solved the control problem and having solved the lift problem then they went to propulsion rather than you know doing that too fast so here again they went to the wind tunnel and they tried two things what kind of an engine and the engine they figured out how many horsepower and how much it could weigh and they couldn't get anybody to build it because it was too light for the horsepower so the car companies weren't interested but you know these guys are bicycle mechanics they had a mechanic in their bicycle shop and the three of them put their heads together and they built a very simple gravity-fed engine for four cylinders not much to it except it was built the first engine ever built of aluminum it was light and had more horsepower than they needed and the other thing was that they they did in their wind tunnel design of propellers and put together the most efficient propeller that had been designed despite the fact that chips have been using propellers for a long time so here they are on their first flight in took them approximately three years people had been spending tons more money than they had for many many years but they had one breakthrough a year for three solid years and after that flight just to go off in fact by 1916 the first plane had crossed the Atlantic Ocean and it's not because they hurried up and built an airplane or even incremental it's because they understood the core problems and tested a solution to it before they built the entire airplane and in lean product development that is the classic example of how you really should go about designing an engineering solution to a problem it's not lots of little features put in iteratively it's understanding the underlying problems and figuring out the solution to the problems before you put the whole design together so the last one I want to talk about is don't divorce design from implementation now let's get back to software the first thing we should be doing when we're developing software is that both designers and by designers I mean people designing the experience and by the way I'm not a designer most developers aren't I can tell that most developers aren't because when I go to websites I see some very poor design anybody that wants me to put my birth year in with a hundred items in a drop-down list ought to be shot there is no excuse for the kind of data entry that I have to do to fill out purchasing forms you know whatever forms are on the web and you can always tell then the you know 85 to 90 percent of those forms were put together by a software developer picking a widget a terrible widget not a designer understanding the experience so the first thing you want is a true designer that thinks about the audience the experience and the implementers who are going to implement it getting in direct contact with the people whose problem they're going to solve and understanding it and possibly even doing it so for example when Philippe Pritchin was leading the air traffic control system for Canada he had all of the software developers and all of the designers sit in the air traffic control top next to the air traffic controllers for a good long time so they truly understood the problem before they tried to design a solution the next thing to think about is design outlines the themes the broad concepts of a product okay not the details and then implementers are intimately involved with this design process because you do not want somebody who doesn't understand the technical issues doing a design that's like never going to be implementable but neither do you want implementers who do not understand design trying to figure out how to do design so you want people who understand both the technical issues and the design issues and perhaps a third person who understands the product selling issues and those three people a product person a design person and a technical lead or architect or you know senior technical person are the core of developing an interesting and a good experience then then you want to take that high level design concept which is all you really should need and incrementally create a specification out of it not all at once but in steps steps that make sense so you elaborate those product features iteratively incrementally to the specification descriptions specifications are descriptions that define this is a feature and it's correct when it does these things a specification by example for example is a really good way to do this and then you want the whole team to have ongoing feedback as they take and put these various types of features and capabilities into play so that you don't usually need to put the whole product in play at the same time remember the Wright brothers they did three annual steps not the whole product at once same thing that you might want to think about so great designs start with something I'm going to call empathy so you have people with different backgrounds some people are designers and they don't really get software some people are developers and they don't really understand design some people understand how to understand customers and think in their shoes some people have I don't know maybe legal or regulatory or background or whatever is necessary and various people with different perspectives have a problem in the market that they're going to look at and they go out and they spend time truly trying to understand what's going on figuring out how does this customer problem work what is the issue what is the problem and they come back and they talk to each other and create a journey map now I don't know if you've heard about journey maps I don't know if Jeff talked about them but a journey map has nothing to do with the product you're creating it is the map of the customer experience when you have right now a problem so that's a map of what they saw going on in the customer environment what they're looking for is pain points problems issues things they can make better and after a journey map is when you start designing the product after you have true empathy with what the customer issues are and you want all kinds of different people involved in this including at least key people from the implementing team not just the designers not just the product people for sure if you just have designers and product people with no technical breaks on their imagination their imagination will run away with them we already know that so we want to have a balanced team of different people the key people and they should all be at the same level so they can have arguments and fights are the product the design and the technical person we should not have a product person telling a team what to do why not because the team in that environment is junior to the product person they have to do whatever the product person prioritizes where is the person to have the argument with the product guy or the design person saying sorry but technically that's just not gonna work why don't we try something else so at the same level as the other people designing the product you also want some technical input somebody that is same level so you can have fair fights fair arguments good trade-offs so only way you're gonna get good trade-offs mate so this is not the way to do it despite what you might have heard in a few different rule books that's not the idea here is the idea the team goes to the market it talks about what they saw draws a journey map creates a story map of how the solution might look then you might be ready to start coding and incrementally putting together pieces with feedback from each piece as you put it together that's a whole balanced team working together that's the whole process and when we skip the first one two three four steps of that and just have somebody bring priorities to the team we tend to get pretty unbalanced products we tend not to have stuff that's going to really engage customers and their experience now I'm going to tell you a story about somebody here in India actually Deepa Baku did I say that right you guys can pronounce better I can she lived in the US and worked for a company called Intuit probably haven't heard much of Intuit but in the US it's a very interesting company that created financial home financial software back in the PC remember the PC era and I use it for my home finances for my business finances but it's not really very popular in other countries because financial laws and tax laws and stuff differ from one country to the other and it's clearly optimized for the US but you know they were still trying to sell it in other countries including India but Scott Cook who was the founder he decided that there should be a lot more experimentation and innovation and so he chartered all of the vice presidents of every region including here to create new experiments with new ideas that will improve the financial lives of people in for example here in India and it can be anything as long as it's aimed at improving the financial situation of ordinary people which is what Intuit does in the US so Deepa was in the US and came back to India to live and the vice president at Intuit here was looking for somebody to come and help him out figure out how to do this new innovation he got funding for he said that maybe two or three people so I'm supposed to change the lives of financial lives of people in India with two or three people hmm how am I gonna do that so anyway she was hired and chartered with about two other people to figure out some solution and since she grew up in a farming community she said I think we'll look at farmers I know farmers okay I grew up with them and her charter was how can I use technology to help the places I grew up in so she went to a village maybe 35 kilometers from here and talked to people she knew and she said so what's your biggest financial problem and they said well we grow vegetables and we have an awful time when we try to sell them because we can go to like any of three different markets this way that way or the other way and once we're there we're at the mercy of the buyer we really can't state our price we don't know which one to go to get the best price it's really a problem so she said hmm so then she went and talked to the buyers and they said well you know we have some orders and once we get our orders filled then we don't need any more vegetables so we can't offer good prices if we don't have a market to sell them and so she was thinking the one piece of technology that everybody had even though they didn't even speak the same language was that so she said hmm what if we could find a way to put those folks together so she did an experiment she had she and her colleagues went to the three nearby markets she took about a dozen farmers with a single vegetable went to the nearby markets every morning got the prices of that vegetable and then sent a text message to the dozen farmers to say here's the best here's the prices for the three and the results were promising so this is a very simple experiment took about you know two or three days and the results were promising the farmers were taking the advice and saying hey you know this has some potential so then she come back here and the three of them spent 60 days putting this system together it's called fossil and right now the idea is that a farmer that's interested calls into a toll-free number and is asked where do you live what vegetables do you grow so they know the date when they're going to be harvested and where do you live so they know the markets around and what's your cell phone number and what language you speak and then the people in the markets are asked to send in their prices now at first the question was why would they do that but turns out that they like having some pre-ordered stuff so they know they're going to get the amount of vegetables that they need so they would after a little bit of encouragement and asking they send in text in their prices and then an SMS goes out to the local farmers who might be interested in those prices and in the end typically the farmer will call the buyer and negotiate a deal and know which market to go to so very very interesting solution they put together a test in 60 days and it was quite successful and since then they have been running small tests and expanding the system ever since a word of mouth has gotten around and farmers have discovered that the interesting thing about the system is that they get something like 15 or 20% increase in income by knowing where to sell their vegetables and by the way at the end of last year they had something like over a million farmers and they were getting about 20,000 a week now this is a platform remember and they're beginning actually into it finance this for four years when it was making no money and costing just a little and now they're starting to get some revenue with ads so it takes a long time and a lot of luck to build a successful platform but this was done through the kind of really empathize with the customers really understand their experience really do something useful to solve their problem and it was the development team as a group that went out and solved that problem just because into it said this is a good idea why don't we have our people explore ideas and see if they can come up with interesting solutions so we have a learning cycle in my opinion I'm going to tell you the answer to this question the biggest waste in software development is well what do you think what do you think I think too many features too much junk in the code so of course you never want to measure code quantity or feature quantity or function point quantity or story quantity because actually all that stuff is bad any junk you put in the code is usually more stuff than you need so if building the wrong thing is the biggest waste in software development the second biggest waste is all the complexity you're causing in your code base because of all those extra things really this is not news this has been true in our world of software for as long as I can remember it's always having too much junk in the code and we need processes by the way that strongly discourage putting stuff in a code base that doesn't really have to be there but how do you do that well the concept in the lean startup movement by Eric Reese which you heard about in the keynote this morning I think is you start out by building a minimum viable product okay and you try it with real customers this is exactly what Deepavaku did you measure their response she had all kinds of measurements and you learn and you do it again and you don't necessarily you know your minimum viable product might be various things but you want to have a learning cycle note the entire team is involved in this learning cycle this is not one person doing the learning and telling the team what to do this is everybody understanding the experience and improving on it so one of the interesting things is today people are looking at software development processes and if you happen to be in a world that does software as a service anyway anything that's online anything that's an app anything that software as a service that is it you're not delivering software to another company you're either delivering it inside of your company or you're delivering it to the web where you can update your own software instead of waiting for somebody else to accept your software this is the process to be heading for if you are involved in any web-based software as a service thing every company we know that really is making headway with good market stuff is adopting this as this is the process that we need to think about this is the process we need to do this book was published I mean just someone's here and he's doing some some talks and this was published in 2010 Tom and I do a lot of classes and workshops and starting about 8 to 10 months after the book was published people in our classes started saying yep we're doing that and I would say oh really what did it take for you to get there and they would always say about a year they wouldn't say how hard it was they wouldn't say how complicated it was they would say about a year and people have told me that it becomes a lot easier than it used to be for anybody in a cloud environment because the ops part of this becomes a lot easier when all of the work has been done by your cloud provider anyway you start with a design and by this I mean a product design not so for design how is the experience going to look and then there is a development stage and by the way I'm going to be posting these on the corp on the conference website so if you want to see any of these things all the footnotes and stuff will be available somewhere so then you write some code and you have good source control but also environment control so you have scripts for your deployment as much as you have scripts for your keeping track of your code itself and you check it in you immediately fire off some tests and those tests happen right away let you know if something's wrong the results go down to a repository and by the way should you happen to be in any kind of regulated environment the regulators love this thing all you have to do is put password protection on anybody who enters either tests or code and you have a very traceable system your tests are either TDD tests or specification by example tests and you run when you as soon as you can somewhere maybe up to every day I call continuous delivery anything that's shy of a week you know twice a week once a week every day are very common cadences for continuous delivery Amazon commits every 11 seconds I hear so you know it can be pretty fast and you run it through an acceptance test testers can then do a push button deploy to a good solid useful test environment I have a friend at some who does Gmail he's the engineering manager for Gmail's name is Mark Strybeck but before that he was working on creating the test environment for and encouraging everybody in Google to be serious about testing and he said that when they got to the point of wanting to do integration testing not just app testing but integration testing they needed an environment as big and as as you know fast as their production environment so he went to the top of the company and said I hate to tell you this but in order to run these tests I need a flat-out production environment and he told me lucky we were founded by people who understood technology because they just said yes sure you got it and no oh none of this oh it's too expensive and that sort of stuff you've got it and so you configure the proper test environment and it needs to be a robust and healthy test environment not just a weak one you should be doing capacity testing automatically and when it's appropriate and as I say we're talking about cadences of a week or less the operations people will deploy so this is a process it's a very disciplined process and when you're doing this most of the other agile things aren't happening anymore you're surely not doing iterations because they're too slow you're not estimating you actually aren't doing that much planning because your your cadence of decisions on what to do pretty much match your cadence of delivery so the feedback cycle is driving a whole lot of your decisions you develop to the trunk you don't branch that doesn't mean that a feature takes just a few hours it means that a feature has a toggle and it can be turned off until it's done and then you turn it on when it's ready and it might take a few days and this concept here has been is as I said something that if you are in the consumer experience world you need to be heading here because you can't run the kind of test that the lean start up wants you to run if you're not able to deploy very frequently so you're trying to validate that what you're doing is the right thing and I think if you were at Jeff Patton's thing he said actually that's really hard so in this test that Microsoft wrote about it this is knowledge knowledge discovery and data mining in 2009 and what they did in Microsoft is they ran a test with their product people the ones who decide what's going to go into a Microsoft product and the the product folks were supposed to say here's a feature and here's what I expect from it and then they actually did a B tests because a B tests are good for cause and effect and they discovered that something like 50% of the time the product managers were wrong and 50% of the time they were right so you may as well flip a coin and so no matter how expert a person is they don't necessarily know the right thing to develop so you want to test early you want to test often and you want to run a bunch of them because most of them aren't going to tell you something but some tests will be really useful let me give you an example this is a case study from Amazon where Greg Linden worked and he had this idea that when people put stuff in their shopping cart if he would present them with what other people bought that had the same stuff in their shopping cart they might buy it and the vice president of marketing said oh no Greg there's something you do not understand about marketing when somebody is ready to buy you close you do not distract them with stuff like here's what other people bought no just forget about it but Greg was not a very obedient guy and so he set up some tests anyway to test what would happen they were quite robust tests and so at the point in time when they were ready he needed permission to run them and the vice president of marketing was furious design comes before we start pouring features into software design is the thing that makes our products likeable to other people that's design my definition actually not my definition that definition puts us in sync with the what way the rest of the world thinks about good design requirements hmm let's talk about requirements that here says requirements exist to constrain design elements they describe the critical few results that define value requirements are never a long list of things those are specifications requirements are the few things that constrain the design they're not descriptions of what to build so what are requirements let me give you an example so as I said that's me I did process control systems for many years and I didn't ever have things like requirement you know list of things to do or user stories or anything like that I'll show you what a requirement for a process control system and my company consisted of five things it is was a process control system for a new process made by the engineers upstairs and when the process was installed it had to make good product number one and by the way it had to be on time because in a manufacturing company having a control system for a new plant come late is very expensive my boss would not have kept his job if we were going to come in late so it had to be on time the operators that were going to use this thing to control the process had to find a convenient and by the way it was going to be maintained by the plant engineer some you know two or three thousand kilometers away not by us and so it had to be turned over to somebody who probably was good enough to understand basic and that was it but this guy was a plant engineer dealt with all kinds of stuff in the plant not just a control system for my product and by the way it should contain the latest technology because they always wanted to have the newest and greatest thing which is why I as a junior engineer was there because I could put some software in it and that was kind of a new concept at the time now everything else is a specification those are requirements and those things you call requirements I propose to you are decisions on how to make things but their decisions on specifications they are not requirements they requirements frame the design and put constraints on the design and define the value to customers so requirements are not designed this is a book by Tom Gilbert was written in 1988 it has gone through 20 printings very popular all about software engineering management and he's written several other books too and he's what the first person that discussed the whole concept of incremental software development in 1983 and he's been writing about it ever since so you can imagine that when this whole concept of agile came along he thought it was a pretty good deal people are listening to what I've been talking about for many years except he doesn't actually like it because there's a problem and the problem is that the whole concept of figuring out what the system is supposed to do used to be part of software develop part of software engineering back when I did software Tom did software we didn't have somebody come in and tell us how to what to do and set priorities and give us little detailed stories we were expected to engineer solutions to technical problems and so he says the worst scenario I can imagine is when we allow real customers users and our own sales people to dictate functions and features to the developers carefully disguised as customer requirements maybe even conveyed by our product owners if you go slightly below the surface of these false requirements you will find that they're not really requirements they're really bad amateur design for the real requirements so I call those specifications not requirements and I propose that if you have a long long list of specifications whether in the form of user stories or whether in the form of written things that other people might call requirements somebody else has figured out engineered the technical solution to the problem not the people who have the background and capability to figure out what a good technical design for that solution is and to be giving teams that level of detail of what they ought to be doing means they no longer are expected to be designing the solution to the problem and you know I agree with Tom that used to be something that software engineers always did and it still should be something that software engineers always do so another guy that's been around for a while his name is Fred Brooks he wrote the book the mythical man month it's classic software engineering book written over 30 years ago and Fred Brooks was the lead architect on the IBM 360 operating system and the mythical man months had much influence on software engineering for many years in about approximately three years ago he published the book called the design of design it was a bunch of his writings he became a professor and he had been writing and speaking in this area for many years and I want to show you three principles he has in this book one is solve the right problem deciding what to design is the hardest part of the design task secondly go beyond analysis and have some insight design isn't just to satisfy requirements but uncover requirements design isn't selecting from alternatives it's realizing alternatives exist and don't divorce design from implementation which we've done a gone a long ways towards doing even in agile agile has a huge distance oftentimes between somebody designing the product and somebody implementing that design one of the most striking 20th century developments in the design disciplines has been the progressive divorce of the designer from both the user and the implementer and as a result we get a whole bunch of big disasters this is not design sorry this is just a bunch of dinky little story cards but that's not the design of a good product this is designed this is a proposal by sphere of influence to a government organization to put together a set of software analytical tools for containing insider threat and you see it has a strong center you can see what it's about and the concept of the picture and the proposal and also the experience of the software is going to be that focused centered on what's important and not a whole bunch of arbitrary stuff that people choose to put in so let's go and let's look at these three things from solve the right problem have insight and don't divorce design from implementation so solving the right problem here's some great solutions to the wrong problem how about that one okay or this one this is actually in 1896 out of the lo and fall who in did a whole bunch of glider flights in Germany he wrote the book called something like aviation as bird flight or something like that and he specified all kinds of aerodynamic numbers which basically created the tables of how lift was going to work unfortunately he was killed in an airplane clash in 1896 no a glider crash crash and the problem was that he didn't have a way to shift if they controlled the gliders with shifting weight which didn't do such a good job every so often here's some more great solutions this is back when people were trying to decide what a horseless carriage was going to be this is it's going to be a bicycle maybe it's going to be a carriage maybe it's going to look like a train and have a steam engine and this interestingly enough is an electric car designed around 1900 by Mercedes Benz so nobody really knew what the car was going to be until Henry Ford said guess what it's going to be a Model T Ford and that became the basis of what a car was going to be and by the way Henry Ford designed that first because he raced cars and he designed a car that he could win the race in let's go back to a really interesting case of how design works if you look at an area of study called lean product development now lean software development but lean product development they are focused very much on don't build until you understand those problems that you are don't design the details until you understand the problem you are trying to solve make sure you got the solutions figured out by testing before you create any detailed design so we in software development have said whoa big design up front is bad but guess what in lean product development they say whoa no design up front is bad okay there we're both right no design up front is just about as bad as big design up front maybe even worse so they're fighting a different problem and back in the 1890s after out of Lilinville was killed in the crash there were many people trying to invent heavier than air flight and they all built planes that would go up for three minutes and crash and they destroyed and then they'd have to build another one and it would take a few years and so the Wright brothers said you know there's a couple things here one is we need more time in the air we we can't we had to test all sorts of things we can't be crashing all the time and the second thing they said was we don't actually care to die either so probably the biggest problem is how to control a plane once it's up there it's not getting it there in the air it's controlling it and so this is a picture from a patent application and because the Wright brothers in Dayton Ohio were bicycle mechanics that's what they were they think of a bicycle tube and it comes in like an oblong box and they took an oblong bicycle tube box same 110 years ago as it is now and Wilbur Wright he kind of bent it and he said you know if I make wings of an airplane like this box and I bend them I'll bet I could control the way the plane flies in the air and that concept changing the shape of the wings to control flight is the way all heavier than air airplane have worked ever since so they got that concept of control right and then they tested it out they built a kite they tried it with you know pulling strings to see if it would actually do some control and they were reasonably happy that they got the idea down so now having comfort with the fact that they could control the plane once it got up there they were a little bit unhappy with how long the plane was staying up they could make it land gracefully they could keep control of it the direction it went how it turned but they still only got a few minutes and they had done a ton of research and they could not understand why they couldn't go up in the strong winds off the ocean they went to North Carolina and stay flying for hours and they got like you know a few minutes so they tried to figure out what was wrong with their calculations and they built next year another one okay and here they are trying to make this thing stay in the air having got the when they built this they changed the shape because some of their friends said oh you probably didn't follow out of will until shape well enough and so they tried another one and while it was pretty controllable it wouldn't stay in the air it was like worse than the first one so they said okay either we're really dumb or somebody's got all those aerodynamic tables wrong and they proved to themselves with some quick experiments that actually the all the constants that had been published were wrong and then they spent a winter recreating all the known constants for aerodynamic lift by creating a wind tunnel and then you can see they have little tiny wing shapes and they put it on balances they gathered data and by the end of that winter they went from October to spring they knew more about how to keep an airplane in the air and what kinds of shape the wing should have than anybody in the world and they figured that out in this small wind tunnel that they built in their bicycle shop and the next year they built a plane and it really worked it stayed up it stayed up and it behaved the way that they calculated it should you can see there's a vertical tail there because they learned that they had to control the yaw and the roll of the plane that was a new concept but they had enough time in the air they started discovering those kinds of concepts and so now having solved the control problem and having solved the lift problem then they went to propulsion rather than you know doing that too fast so here again they went to the wind tunnel and they tried two things what kind of an engine and the engine they figured out how many horsepower and how much it could weigh and they couldn't get anybody to build it because it was too light for the horsepower so the car companies weren't interested but you know these guys are bicycle mechanics they had a mechanic in their bicycle shop and the three of them put their heads together and they built a very simple gravity-fed engine for four cylinders not much to it except it was built the first engine ever built of aluminum it was light and had more horsepower than they needed and the other thing was that they did in their wind tunnel design a propeller and put together the most efficient propeller that had been designed despite the fact that chips have been using propellers for a long time so here they are on their first flight in took them approximately three years people had been spending tons more money than they had for many many years but they had one breakthrough a year for three solid years and after that flight just took off in fact by 1916 the first plane had crossed the Atlantic Ocean and it's not because they hurryed up and built an airplane or even incremental it's because they understood the core problems and tested a solution to it before they built the entire airplane and in lean product development that is the classic example of how you really should go about designing an engineering solution to a problem it's not lots of little features put in iteratively it's understanding the underlying problems and figuring out the solution to the problems before you put the whole design together so the last one i want to talk about is don't divorce design from implementation now let's get back to software the first thing we should be doing when we're developing software is that both designers and by designers i mean people designing the experience and by the way i'm not a designer most developers aren't i can tell that most developers aren't because when i go to websites i see some very poor design anybody that wants me to put my birth year in with a hundred items in a drop-down list ought to be shot there is no excuse for the kind of data entry that i have to do to fill out purchasing forms you know whatever forms are on the web and you can always tell then the you know 85 to 90 percent of those forms were put together by a software developer picking a widget a terrible widget not a designer understanding the experience so the first thing you want is a true designer that thinks about the audience the experience and the implementers who are going to implement it getting in direct contact with the people whose problem they're going to solve and understanding it and possibly even doing it so for example when philippe pritchin was leading the air traffic control system for canada he had all of the software developers and all of the designers sit in the air traffic control top next to the air traffic controllers for a good long time so they truly understood the problem before they tried to design a solution the next thing to think about is design outlines the themes the broad concepts of a product okay not the details and then implementers are intimately involved with this design process because you do not want somebody who doesn't understand the technical issues doing a design that's like never going to be implementable but neither do you want implementers who do not understand design trying to figure out how to do design so you want people who understand both the technical issues and the design issues and perhaps a third person who understands the product selling issues and those three people a product person a design person and a technical lead or architect or you know senior technical person are the core of developing an interesting and a good experience then then you want to take that high level design concept which is all you really should need and incrementally create a specification out of it not all at once but in steps steps that make sense so you elaborate those product features iteratively incrementally to the specification descriptions specifications are descriptions that define this is a feature and it's correct when it does these things specification by example for example is a really good way to do this and then you want the whole team to have ongoing feedback as they take and put these various types of features and capabilities into play so that you don't usually need to put the whole product in play at the same time remember the Wright brothers they did three annual steps not the whole product at once same thing that you might want to think about so great designs start with something i'm going to call empathy so you have people with different backgrounds some people are designers and they don't really get software some people are developers and they don't really understand design some people understand how to understand customers and think in their shoes some people have i don't know maybe legal or regulatory or background or whatever is necessary and various people with different perspectives have a problem in the market that they're going to look at and they go out and they spend time truly trying to understand what's going on figuring out how does this customer problem work what is the issue what is the problem and they come back and they talk to each other and create a journey map now i don't know if you've heard about journey maps i don't know if jeff talked about them but a journey map has nothing to do with the product you're creating it is the map of the customer experience when you have right now a problem so that's a map of what they saw going on in the customer environment what they're looking for is pain points problems issues things they can make better and after a journey map is when you start designing the product after you have true empathy with what the customer issues are and you want all kinds of different people involved in this including at least key people from the implementing team not just the designers not just the product people for sure if you just have designers and product people with no technical breaks on their imagination their imagination will run away with them we already know that so we want to have a balanced team of different people the key people and they should all be at the same level so they can have arguments and fights are the product the design and the technical person we should not have a product person telling a team what to do why not because the team in that environment is junior to the product person they have to do whatever the product person prioritizes where is the person to have the argument with the product guy or the design person saying sorry but technically that's just not going to work why don't we try something else so at the same level as the other people designing the product you also want some technical input somebody that is same level so you can have fair fights fair arguments good trade-offs it's the only way you're going to get good trade-offs mate so this is not the way to do it despite what you might have heard in a few different rule books that's not the idea here is the idea the team goes to the market it talks about what they saw draws a journey map creates a story map of how the solution might look then you might be ready to start coding and incrementally putting together pieces with feedback from each piece as you put it together that's a whole balanced team working together that's the whole process and when we skip the first one two three four steps of that and just have somebody bring priorities to the team we tend to get pretty unbalanced products we tend not to have stuff that's going to really engage customers and their experience now i'm going to tell you a story about somebody here in india actually Deepa Baku did i say that right you guys can pronounce better she lived in the u.s and worked for a company called Intuit probably haven't heard much of Intuit but in the u.s it's a very interesting company that created financial home financial software back in the pc remember the pc era and i use it for my home finances for my business finances but it's not really very popular in other countries because financial laws and tax laws and stuff differ from one country to the other and it's clearly optimized for the u.s but you know they were still trying to sell it in other countries including india but Scott cook who was the founder he decided that there should be a lot more experimentation and innovation and so he chartered all of the vice presidents of every region including here to create new experiments with new ideas that will improve the financial lives of people in for example here in india and it can be anything as long as it's aimed at improving the financial situation of ordinary people which is what Intuit does in the u.s so Deepa was in the u.s and came back to india to live and the vice president at Intuit here was looking for somebody to come and help him out figure out how to do this new innovation he got funding for he said yeah maybe two or three people so i'm supposed to change the lives of financial lives of people in india with two or three people hmm how am i going to do that so anyway she was hired and chartered with about two other people to figure out some solution and since she grew up in a farming community she said ah i think we'll look at farmers i know farmers okay i grew up with them and her charter was how can i use technology to help the places i grew up in so she went to a village maybe 35 kilometers from here and uh talked to people she knew and she said so what's your biggest financial problem and they said well we grow vegetables and we have an awful time when we try to sell them because we can go to like any of three different markets this way that way or the other way and once we're there we're at the mercy of the buyer we really can't state our price we don't know which one to go to to get the best price it's really a problem so she said hmm so then she went and talked to the buyers and they said well you know we have some orders and once we get our orders filled then we don't need any more vegetables so uh we can't offer good prices if we don't have a market to sell them and so she was thinking the one piece of technology that everybody had even though they didn't even speak the same language was that so so she said hmm what if we could find a way to put those folks together so she did an experiment um she had um she and her colleagues went to the three nearby markets she took about a dozen farmers with a single vegetable went to the nearby markets every morning got the prices of that vegetable and then sent a text message to the dozen farmers to say here's the best here's the prices for the three and the results were promising so this is a very simple experiment took about you know two or three days and the results were promising the farmers were taking the advice and saying hey you know this has some potential so then she came back here and the three of them spent 60 days putting this system together it's called fossil and right now the idea is that a farmer that's interested calls into a toll-free number and is asked where do you live what vegetables do you grow so they know the the date when they're going to be harvested and where do you live so they know the markets around and what's your cell phone number and what language you speak and then the people in the markets are asked to send in their prices now at first the question was why would they do that but turns out that they like having some some pre-ordered stuff so they know they're going to get the amount of vegetables that they need so they would after a little bit of encouragement and asking they send in text in their prices and then an SMS goes out to the local farmers who might be interested in those prices and in the end typically the farmer will call the buyer and negotiate a deal and know which market to go to so very very interesting solution they put together a test in 60 days and it was quite successful and since then they have been running small tests and expanding the system ever since a word of mouth has gotten around and farmers have discovered that the interesting thing about the system is that they get something like a 15 or 20 increase in income by knowing where to sell their vegetables and by the way at the end of last year they had something like over a million farmers and they were gaining about 20,000 a week now this is a platform remember and they're beginning actually into it financed this for four years when it was making no money and costing just a little and now they're starting to get some revenue with ads so it takes a long time and a lot of luck to build a successful platform but this was done through the kind of really empathize with the customers really understand their experience really do something useful to solve their problem and it was the development team as a group that went out and solved that problem just because into it said this is a good idea why don't we have our people explore ideas and see if they can come up with interesting solutions so we have a learning cycle um in my opinion i'm going to tell you the answer to this question the biggest waste in software development is well what do you think what do you think i think too many features too much junk in the code oh so of course you never want to measure code quantity or feature quantity or function point quantity or story quantity because actually all that stuff is bad any junk you put in the code is usually more stuff than you need so if building the wrong thing is the biggest waste in software development the second biggest waste is all the complexity you're causing in your code base because of all those extra things really this is not news this has been true in our world of software for as long as i can remember it's always having too much junk in the code and we need processes by the way that strongly discourage putting stuff in a code base that doesn't really have to be there but how do you do that well um the concept in the lean startup movement by Eric Reese which you've heard about in the keynote this morning i think is you start out by building a minimum viable product okay and you try it with real customers this is exactly what Deepavaku did you measure their response she had all kinds of measurements and you learn and you do it again and you don't necessarily you know your minimum viable product might be various things but you want to have a learning cycle note the entire team is involved in this learning cycle this is not one person doing the learning and telling the team what to do this is everybody understanding the experience and improving on it so one of the interesting things is today people are looking at software development processes and if you happen to be in a world that does software as a service anywhere um anything that's online anything that's an app anything that's software as a service that is you're not delivering software to another company you're either delivering it inside of your company or you're delivering it to the web where you can update your own software um instead of waiting for somebody else to accept your software this is the process to be heading for if you are involved in any web-based software as a service thing every company we know that really is making headway with good market stuff is adapting this as this is the process that we need to think about this is the process we need to do this book was published i mean just someone's here and he's doing some some talks and this was published in 2010 Tom and I do a lot of classes and workshops and starting about eight to ten months after the book was published people in our classes started saying yep we're doing that and i would say oh really what did it take for you to get there and they would always say about a year they wouldn't say how hard it was they wouldn't say how complicated it was they would say about a year and people have told me that that becomes a lot easier than it used to be for anybody in a cloud environment because the ops part of this becomes a lot easier when all of the work has been done by your cloud provider anyway you start with a design and by this i mean a product design not software design how is the experience going to look and then there is a development stage and by the way i'm going to be posting these on the corp on the conference website so if you want to see any of these things all the footnotes and stuff will be available somewhere so then you write some code and you have good source control but also environment control so you have scripts for your deployment as much as you have scripts for your keeping track of your code itself and you check it in you immediately fire off some tests and those tests happen right away let you know if something's wrong the results go down to a repository and by the way should you happen to be in any kind of regulated environment the regulators love this thing all you have to do is put password protection on anybody who enters either tests or code and you have a very traceable system your tests are either tdd tests or specification by example tests and you run when you as soon as you can somewhere maybe up to every day i call continuous delivery anything that's shy of a week you know twice a week once a week every day are very common cadences for continuous delivery um amazon commits every 11 seconds i hear so you know it can be pretty fast and you run it through an acceptance test testers can then do a push button deploy to a good solid useful test environment i have a friend at um who does gmail he's the engineering manager for gmail his name is mark stribeck but before that he was working on creating the test environment for and encouraging everybody in google to be serious about testing and he said that when they got to the point of wanting to do integration testing not just app testing but integration testing they needed an environment as big and as as uh you know fast as their production environment so he went to the top of the company and said i hate to tell you this but in order to run these tests i need a flat out production environment and he told me lucky we uh were founded by people who understood technology because they just said yes sure you got it and no oh none of this oh it's too expensive and that sort of stuff you've got it and so you configure the proper test environment and it needs to be a robust and healthy test environment not just a weak one um you should be doing capacity testing automatically and when it's appropriate and as i say we're talking about cadences of a week or less the operations people will deploy so this is a process it's a very disciplined process and when you're doing this most of the other agile things aren't happening anymore you're surely not doing iterations because they're too slow you're not estimating you actually aren't doing that much planning because your your cadence of decisions on what to do pretty much match your cadence of delivery so the feedback cycle is driving a whole lot of your decisions you develop to the trunk you don't branch um that doesn't mean that a feature takes just a few hours it means that a feature has a toggle and it can be turned off until it's done and then you turn it on when it's ready and it might take a few days and this concept here um has been um and is as i said something that if you are in the consumer experience world you need to be heading here because you can't run the kind of test that the lean startup wants you to run if you're not able to deploy very frequently so um you're trying to validate that what you're doing is the right thing and i think if you were at jeff patin's thing he said actually that's really hard so in this test that microsoft wrote about it this is knowledge um knowledge discovery and data mining in 2009 and what they did in microsoft is they ran a test with their product people the ones who decide what's going to go into a microsoft product and the the the product folks who are supposed to say here's a feature and here's what i expect from it and then they actually did ab tests because ab tests are good for cause and effect and they discovered that something like 50 percent of the time the product managers were wrong and 50 percent of the time they were right so you may as well flip a coin um and so no matter how expert a person is they don't necessarily know the right thing to develop so you want to test early you want to test often and you want to run a bunch of them because most of them aren't going to tell you something but some tests will be really useful let me give you an example this is a case study from amazon where greg linden worked and he had this idea that when people put stuff in their shopping cart if he would present them with what other people bought that had the same stuff in their shopping cart they might buy it and the vice president of marketing said oh no greg there's something you do not understand about marketing when somebody is ready to buy you close you do not distract them with stuff like here's what other people bought no just forget about it but greg was not a very obedient guy and so he set up some tests anyway to test what would happen they were quite robust tests and so at the point in time when they were ready he needed permission to run them and the vice president of marketing was furious but jeff bezal said brag that anybody if they can bring test data can get their way and so he had to run that he had to be allowed to run the test and the fact is that every single test showed that his idea generated three percent additional revenue all the time reliably and now you get to see that when you they have already defined as part of their narrative a minimum viable product the smallest thing that they would be proud to send to give to customers so they go to a stage called build it and then the three get a squad a bunch more developers to put the minimum viable product together and once they get it again they show it to themselves if they're proud of it they show it to the management team and if they say yes this is something we would be proud to show our customers and this is the right time timing is correct to add a new to try a new feature then they go into a phase which they call shipping now shipping is not a single step it's a long process looks just like the lean startup process this is this is the spot where lean startup works at this point they take a few percent of their customers just a few and they release it to that few and they start making measurements they start doing ab tests they start tweaking it they start improving it making it better and better until finally that everybody decides this is really good the time is right let's release this to all of our users and then at that point this is where the most of the work in the company happens the product is then grown tweaked made full fledged not just minimum viable product over time by that same squad without at that point too much interruption from management because it's it's out there and they're just supposed to make it better and generate more revenue from it so that concept of let's think about design let's think about making sure that it's really a great experience before we start sending it out to customers but also once we send it let's keep measuring how we're doing let's run some tests to make sure it's what customers really want is a concept that's a sort of a whole team whole product concept which i think you might think about so with that guess what it is 530 you guys are probably really ready to go home um i could i suppose take questions but i don't know if anybody actually wants to stay so i'll just say thank you and then we could do questions if you feel like it thank you