 My name is Raj, I am with Lightspeed, we are a small boutique agile transformation coaching and training service company based out of Washington DC. I'm happy to be here and this afternoon I'm going to share with you some of our experiences at Lightspeed in terms of product development and how we stumbled into the whole lean startup movement and talk about some of the tools specifically that we've used some of the learnings that we've had and how we are applying some of those learnings to the larger enterprise because we do a lot of coaching services in the enterprise space but we definitely begin to see some aspects of the lean startup ideas and tools that we can bring to bear in the larger enterprise. How many of you work for large enterprises? Okay most of you and startup type come is relative yeah but so what's your definition of large? Okay oh wow that's mega large but anybody in the startup space here yeah okay all right just one okay so let me just quickly start with by asking you guys a question how many of you think some of the items listed here are risks with current agile implementations? backlog items not validated against customer needs then we have this godlike product owner who knows everything about the needs the customer backlog the acceptance criteria and so forth you know there's lack of clear advice on how to change direction agility is about responsiveness to change but increasingly I noticed that I don't really see that manifest in a lot of implementations and also you know we're all technologists at heart we do what we do best which is to build build build so does this sort of resonate with at least some of your experiences it definitely has happened with me both in the small and large scale it seems like what we are really doing in the name of agile is building incrementally and I try to draw a distinction and hopefully I'll share that with you as we go through the slides is there's a difference between incremental time box development and iterative development you know with with incremental development you already have a well-formed idea and you're really using the incremental model the time box model to incrementally build out that fully realized idea in the minds of our stakeholders does this happen with a lot of you guys or is it truly iterative where you are refining idea of sprint after sprint so one one hand up right so but how about for the rest of you is this really the reality of your agile implementations the idea is sort of concrete and all we're doing is building it out on incremental passion yeah sometimes okay okay yeah so that's not truly being agile it helps but it really is not you're not really the full benefits of being agile because fundamentally most initial business models are these ideas that stakeholders think I have a brilliant idea is going to sell them do they pan out of most famous examples of course is a pdm encryption software system led to the creation of PayPal right another famous one is there was a game on online massive online multi-user game called game never ending g&e that was the genesis that that led to the creation of flicker right g&e lasted for a year and a half or something in 2004 2005 so the point I'm trying to make is most most of these ideas that we think are brilliant ideas are probably brilliant in our own minds or our stakeholders mind of course in these cases these are fantastic great examples where they've made a ton of money but unfortunately that wasn't our story so so at life speed we had this brilliant idea on this is not it's just gonna yeah so let me just talk through this okay because I know there's another session following right after mine so the idea was a few years ago we said you know we want to build an electronic tool to facilitate continuous improvement items the most obvious example of that would be retrospectives right how many of you all do retrospective what went well what didn't go well so we wanted to increasingly as we started to see a lot of distributed teams we begin to realize and like with most agile adoptions the ceremony that's dropped most is the retrospective and we believe it's an important of practice and and with the proliferation of distributed teams we thought that thing was totally you know falling by the wayside what could we do to sort of mitigate that problem so we said okay we have a brilliant idea we're gonna build this tool for for facilitating near real-time retrospectives so we built a prototype in code not a paper prototype and at the time Ruby on rails was bang thing to do so we hired some hot shot Ruby on rails developers we started building this prototype and we took it on the road to all the big conferences and everybody said this is awesome and by everyone I mean just our friends of course they're gonna tell you it's awesome the question we should have asked is will you pay for it right which we didn't realize they said okay we were stoked by everybody's enthusiasm there wasn't a tool such as the tools called Sensei it's still out there and they say it was awesome so we said okay we'll continue doing what we know best which is continue to add more capabilities and we continue to do that we lost our team because Ruby on rails developers were you know in demand maybe there still are anyway the point is a continued building building building you know thinking that build and they will come someday paying customers that is so we spent about six months doing that six to eight months and then we almost I mean we were funding this ourselves not that it was a big expensive tool but still was coming out of our pockets and we had a decent solution in place but very few customers right and at this point it was still in beta we weren't really charging anybody by customers I mean people who were just using the system but we didn't know where to go right we were running out of our runway money and the tools there advertised some customers didn't know what to do right so just about that time 2011-2012 we attended this conference the lean startup conference and every crease was there and that was our introduction to the whole lean startup movement how many of you you're familiar with the lean startup movement I assume right yeah yeah and sort of Eric Reese came out with this book but it was a revelation to us in terms of his ideas again not all of his ideas are totally original it was based off of ideas by Steve Blank talked about customer development the nurturing customers is key when you think about product development and so forth but again this has become ubiquitous now right the build measure alone cycle I'm not going to sort of talk a lot about that because these days everybody knows about the bill measure one cycle but in a nutshell what we got out of that was so we were at this sort of crossroads right we build this product we didn't know what to do with it you don't have money what the lesson learned for us was really what we should have done was we should have described our business case as this a set of assumptions instead of essentially thinking that this is going to be a sure shot talking to early adopters we had done some of that but not really real paying customers and the way we were interviewing people was obviously totally incorrect so create prototypes but maybe test assumptions very cheaply pivot which is another word you'll see a lot and deliver often with high quality software using agile methods so what we knew really was just this part right just because we were good at agile delivery doesn't necessarily make as good product development folks and that's what was the realization that dawned upon us so there's this whole aspect of discovery that we were completely missed when we built our product so to me so this picture again is the courtesy of Jeff Patton by the way for those of you who know Jeff Patton's story mapping and all of that work it's been popularized by him the so the the real agility is about making sure that you have ways to think of your idea as just that it's just an idea and then you need to continuously iterate on it validate the idea that it's actually going to be return there's going to be a return on investment before you start to build build build right the question is looks good on paper how the heck do you really do that right so next I'm going to get into what we've tried and some of the tools right again it's our experience and there's probably a whole lot more tools out there that'll serve your needs but let me share what we've done and how we were playing that in the larger enterprise now okay so what the lean startup conference gave us was really the idea that you may be great at delivery but that's not the whole story that there's an equally important part which is discovery right and this isn't this isn't relevant just in the startup space right this is relevant even in the enterprise space and when I when we talk about this entire value it talks about this right gram xb kanban whatever you want it's about delivery but the discovery aspects precedes that and there's also an aspect that's afterwards right identifying the real customer needs the product market fit all of those aspects as well as once you're iterating on the solutions how do you really validate that you're meeting the meeting those needs and this is where to us a lean startup movement gives us and fills out that larger picture right so any questions so far pretty straightforward right so what we begin to believe in and this illustration depicts that is there's there's tools there's techniques to be good about discovering your customer needs your product product needs and that should be balanced with your agile delivery mechanisms your scrum xb what whatnot and and then also your entire product increment should be validated by some sort of data metrics and what I'm going to talk next is really the flow of how this happens and what kind of tools that you can bring to bear as you sort of go through the discovery process you know your product backlog and so forth so let's start with holistic discovery so this is this is the part preceding before you even get to the product backlog right you have an idea and you're trying to sort of target customer needs and and market fit and again just don't think of this as true just being about startup companies and startup ideas because I'll share with you examples of how this is being done in big companies marriott.com is an example of how hotel chain that does this pretty well so I'll share some ideas there so in terms of the discovery part the first part of it what we're really trying to see is identify and discover the customers the market risk and maybe to some degree even a technical if there is some sort of unknown there and in order to do that you really to discover customer needs a couple of techniques interviewing users that's easier said than done if you guys are in the space where you want to sort of do this in very intentional customer discovery it's not enough to just say yeah I'm going to go outside and talk to talk to somebody right the first aspect really is about what in the lean start of what they call it goofing you heard of that getting out of the building g o o b right the first step is to get out of the building and talk to people and there's lots of questionnaires and samples that you can take a look at I particularly like the one that I have listed at the bottom and by the way I'll share all of these slides with you guys if there's interest I'll post it on slide share definitely worth checking out if you're thinking about being very intentional about your discovery aspects and if you're more if your mode of interaction is going to be just these conversations with with potential customers or even people on the street right so observing users in their native environments how many of you here how many here developers testers coaches developers uh testers coaches and the rest of you managers yeah um so what I wanted to say is uh how many of you actually go and observe uh I'm talking about let's say the adult teams to develop those testers b a's um how many of you actually observe your your potential end users in their environment as they're working how many probably nobody right so right now I'm a coach in a large large transformation in the US federal space homeland security you would think this kind of stuff is not going to fly in a large enterprise especially a federal government space but what they're beginning to do now is they once every month every two uh every two sprints teams will actually spend time with the actual field agents and see how they're working how they're interacting with the software and actually uh and come back and report those findings to those product owners sometimes the product owners go there just so that the developers in the testers have a real sense of how end users are are are interacting with their systems okay um and another technique so interviews sitting in in in the native environments concierge simulating the system and I'll show you some examples in fact there's a great example right here outside of Bangalore that Intuit ran and I'll share that experience in a second uh we'll talk about concierge uh concierge prototyping rapid prototypes you guys how many of you use like uh balsamic and such to run a paper prototype before you actually build things out anybody no okay okay uh so all different kinds of bays of discovering and seeding your product backlog okay a tool because we are talking about tools here uh a great tool that we've used and I recommend you check it out it's called the validation board and it's it's a tool that lets you write down your uh hypothesis you know what customer segment are you going after what is the problem hypothesis what problem is your particular feature or your product solving it could be as small as a feature right but you don't go out and crank out the code and build it out but you essentially be very intentional about understanding who am I targeting what is the problem uh before you get to the solutioning and what um are some of the uh the experiments if you will that you can run to validate before you actually get to the solution a simple example could be narration is running this conference right now his hypothesis uh his problem hypothesis might be yeah there's not enough agile conferences to serve or the development community right that that's a problem hypothesis uh and the customer hypothesis would be he says if I do this I think developers, testers, agilists will attend the conference what's the risk case assumption that they'll actually show up right so again it's a very simple board it's a visualization technique you can do it for product development for whatever you want right so um and then you get out and you start interviewing the questions observing them and so forth and you're starting to validate the actual need the problem right you're discovering the problem need uh the customer segment that you may be targeting is is it the right customer segment and so forth very simple tool very effective tool and if you if you if you realize that hey it's not the right product need it's not the right solution whatever the case maybe you start making adjustments try different experiments maybe it's not the agilist maybe they need some other type of a customer segment right so um so that's part of the discovery process again right before you have written a single line of quotes once you have narrowed down on um at least that's what we did once we came to the point where we like we don't have enough money to continue working on our product what do we do we started we stopped we started actually using the validation board before we added new features to make sure we were adding the right kind of things um and another great tool the guy who's going to talk next has used it right so the lean canvas tool is a great a way to sort of depict it's a one pager to depict your business case and your business need once you have validated that a particular idea is resonating with a certain customer segment you make your business case single pager nothing fancy nothing big right um and I'll if only the slide would come up I could show this here it's freely available and again I have the links to all of these canvases and models so you can free to download and try it out right so I'm just going to keep moving on because the lean canvas again there's a certain intentional way you go about filling it but the idea that it's still a single page uh single page business business model not like your typical business case documents that you put out um I was going to show you an example of how we created the business uh the lean canvas for our product the retrospective tool um but maybe I'll come back to it now this is a recurring problem huh all right so this is the lean canvas it's broken up into a little different segments I'm not going to necessarily walk through each of them I'd encourage you to sort of go to lean stack.com um and Ash Moria wrote a great book called running lean in the references section at the end I um list out good reads and good templates that you can potentially use but fabulous little tool to capture your uh your business case so so what we did this is essentially an example of what we tried to do a bit sensei our retrospective tool because we weren't really being very intentional about exactly who we're targeting what sort of metrics are we trying to capture none of that but once we sort of came back from our conference we started to be more intentional about it we tried to look at our customer segments who we're really targeting distributed teams coaches scrum masters uh what kind of metrics do we need how many people are signing up well that was a lousy metric and those are called vanity metrics and if we were getting excited about the fact that oh hundreds of people are signed on but the question is are they using it right that's the deeper question to answer so metrics and we'll talk a little bit more about metrics if they need to be a little more than just pure vanity metrics saying hundred people came from a website but what did they do with it right how long did they stay on it did they at least finish an entire use case those are all important so those are uh metrics that you come up with um so once so moving on to the product backlog you have an idea you captured your business case perhaps now you're going to start to sort of fill out your product backlog what sort of things could you do there and what kind of tools could you sort of bring to bear there um how many of you have heard of the term MVP right what does it mean sir and what does that mean basic flow yeah but just okay right and there's different kinds of things that you could bit sometimes it's about just learning you know the MVP could be just about trying to get some information and oftentimes you would see people just putting these facades of of organizations where behind there was no meat zappos which is a big who's heard of zappos it's a big shoe retailer in the u.s the way they launched it all they did was created one page and behind there was really nothing there was no real store right it was essentially whenever they got orders they actually would go to other stores and buy the shoes and ship they were really testing an idea with in the cheapest way possible before they built out an entire inventory of system so now that's a real party there I think so and oftentimes perhaps we're in the space where we want to essentially learn a little bit more MVP could be about prototype building prototypes cheaply and we look at some examples of how we we might do that so the idea really is early releases have to focus on still validating your idea in the cheapest way possible before you build some big cathedral okay that's really the general idea of an mvp so if you are in this phase where you need some sort of creating nice planning just to see how people sort of jump to your idea or not that's a great tool unbalanced need to slide to talk to the next one how many of you use story maps yeah all available yeah yeah yeah not in that one not in the slides I'll upload the latest slide yeah absolutely I'll do that so so maybe maybe beyond just landing pages maybe that's not your business case perhaps what you need to do is still define your mvp and oftentimes we find using how many of you use story maps to visualize your product backlog in a two dimensional format yeah it's a great little tool I would strongly encourage you guys whether you use use the other tools or not in order to visualize your one dimensional product backlog a story map is a great way to add a second dimension to your product backlog gives you much needed clarity as to how stories relate to each other and so forth and helping you fill gaps and we could use story maps here to build out your mvp again these are visualization techniques and tools very simple tools you don't need an electronic version you have a whiteboard and you can do this the idea of a story map is you have your backbone of the major themes and you build out your stories under those epics if you will in in in in terms of the most important things and then the nice to have to the bottom right and then you just create your releases based on your essential things and not the nice to have again simple idea so so this essentially is is showing you a lean kit which is another tool that we use where once we've discovered the the the fact that we we are not just going to just blindly build features without expressly validating them we essentially made sure that before we even get to the uh the to-do doing done column we have a host of things uh from our backlog the validation aspect making sure we actually talk to people before any of these features actually make it to the uh to the doing column we're very very intentional about throttling the amount of things that we are feeding into the dev team again it's a very simple idea but if you want to be uh make little changes I would say just making sure that you add a single column like validate before you get in could pay rich dividends so this is part of our discovery validation and to the right would be our development okay so now that you have this risk driven product backlog now you actually get into the sprinting aspect and I don't have to define sprints to you guys but what I want to mention here is what some of the organizations that we are working with are doing so in exploratory sprints I talk about team intercepting users team helping design experiments so in fact marriott uh does this as I mentioned quite well what they do is the the development team itself the agile agile team within the sprint actually takes some time to craft lightweight prototypes using balsamic or easel or some of these tools before they make changes to the hotel reservation system things that you see online right now they actually intercept users in the field and they have a capricet of early adopters if you will and they show them the stuff before they actually make any change to the website okay and they do that every team does that and the team is it actually helps design those experiments right it's not somewhere some usability expert doing it for them it's the actual team so balsamic you guys have heard of balsamic again a very popular tool I think not all of them are open source you know once you start looking for some of these prototyping tools maybe you'll find some open source this is a great one for creating this very low fidelity type prototypes paper screens and so forth this is what marriott would use they use paper prototypes or maybe draw things on paper and they would actually go out talk to users and before they actually implement anything easel is an example of another great tool which actually is a little more sophisticated we can actually create clickable prototypes it feels like a real website but it really is it's just all smoke and mirrors there's nothing behind so another good tool so so those are the previous examples were about creating prototypes so you can take it to users they can touch and feel concierge prototyping is another interesting example I wanted to bring this example because this was done by intuit intuit the software company intuit global services a few years ago right outside of Bangalore I forget the name of the little village what they did was they wanted to essentially help farmers get better price for their crops okay so that was their mission and how did they do it essentially like building a little stock market for farmers because otherwise they were sort of being gouged by landowners it's unfair prices they didn't go out and build some sophisticated system they talked to the farming community right here I think it was 100 kilometers from here from out there of Bangalore and in the end what the the initial cut of the solution they built was actually having the the farmers register their mobile numbers with intuit and intuit actually would just send them sms matching matching potential customers to the farm it was that's an example of a concierge prototyping you're not really building a platform but it's as simple as registering and sending sms messages right so the the you can read up more about it it's called fusillus the name of the the initiative that intuit launched so hugely popular I think now there's I don't know what the current state is but now there are tens of thousands of farmers registered so an example of a little thing can make a big difference all right any questions that's fine so we've talked about exploratory sprints about teams intercepting being very intentional about soliciting feedback before you build things and sometimes it's not necessary that you have to do it intra sprint sometimes you have discovery teams discovery teams which are one sprint ahead doing the sort of exploration and then whatever they find they bring it back to the team so there's a staggered sprint if you will whether the discovery team that's one sprint ahead then there's a delivery team that's one sprint behind but the delivery team is always making sure that they're spending their time on validated ideas that's that's it yesterday so at the end of the sprint we talked about data driven solutions what does that really mean one of the things we talked about and I mentioned you want to stay away from vanity metrics just counting how many people came on to your website means nothing right so these guys back there I see the guys who wrote convention by the way I didn't realize was an perfect example of a lean startup product you know the convention is I had no idea that there's sort of it was a gracious pet project it sort of launched it and now look at it it's a whole blown platform right do you guys do lots of metrics collection yeah yeah so so these are called pirate metrics again Dave McClure he has a great on I think it's on slide share he has and he has lots of talks on meaningful metrics pirate metrics they call pirate metrics because pirates are all about acquisition activation retention revenue and referral so in our case for Sensei what we were interested in were a couple of things you know how you know people are and we built a metrics dashboard something that we hadn't done prior to the lean startup conversations and so forth the first thing we came back and we built ourselves a little dashboard that told us more than just what Google analytics was telling us which is number of people who signed in so what we wanted to see is how many people how many new organizations signed up how many of them actually completed retrospective look at this 200 organizations and some big organizations but look at the number of use cases that were actually completed just one that's telling to us that you know this is not going to go far right so again Dave McClure definitely read his and his articles and such on metrics and make them your own but be sure that you are tracking something a couple of other tools mix panel kiss metrics are again good tools for this sort of invasive metrics and finally validating the product increment so you built out a sliver of your features if you will or your stories the mvp and how many of you do ab testing multivariate testing you heard of ab testing yeah do you want to share what what ab testing is with us anyone it's not actually beta testing ab testing multivariate testing is you're essentially giving as it says you provide a certain interface a version of an intervention and you channel some users to it and another another set of users to do another version of the same thing it's not quite beta testing but you're really trying to see if I give two versions of a similar capability which one resonates with people these days this is this kind of stuff is very very common convention engine guys do you guys do that do you do ab testing not yet but they will soon enough and what's interesting is when we on our silly little sensei tool and this you know is very interesting on a psychological perspective we just changed where certain aspects of sign up were and suddenly we start to see there were different higher registrations when things are top left versus bottom right colors made a huge difference and tools such as optimizely again very easy tool there's other tools as well you can very easily and dynamically change aspects of your pages and and and redirect different customer segments to different versions of you know and for example we see we saw a huge uptick when we changed some of the wording we said this is a tool for conducting retrospectives for distributed teams we just put the word distributed we had a huge search in the number of registrations it's very fascinating to see how this sort of plays with uh uh you know our human psyche if you will um so these days even the where we are marry our things like that obviously they do ab testing for everything and we've got that's there's a huge enterprise even in the federal space where i'm coaching right now this is for immigration form and so forth everything is ab tested you're giving different versions uh so this is not such a new thing anymore and a few other tools that we tried um which is qualaru and wufu so i use qualaru when we put that into our sensei tool every time we introduce a product we said uh let me see i wanted to reach out to the to the registrant and say did you like this feature that was added and so forth um it it didn't pan out for us as i'd hope to but there it lets you conduct surveys right if you're launching something you'd let you conduct surveys and solicit feedback um very easy integration with javascript so uh this one helped us quite a bit which is and what we zopem and olark are a couple of tools that let you um we we plugged it in for customer service we wanted to you know you know lean start up says embrace your early adopters don't let them go right be nice to them so we said okay we we're going to provide real-time help and um and um we integrated this and what was interesting is was if while we intended it for it to be just a help tool it helped us connect with our early adopters and the most passionate avid users in a more intimate fashion that we could have never had i know at some point we put help text with a phone number our phones never rang but the minute we put something like this it's it's it's a sign of where things are moving right people are more comfortable with these sorts of interactions versus saying call hello 1800 garage what can i do for you that that doesn't really work so this this was really useful for us so um anyway so again data driven uh metrics embracing your customer and there's a plethora of tools out there that let you do that rather seamlessly and easily so we're almost reaching the end uh how much time do we have five minutes ago yeah so let me wrap this up very quickly i just wanted to again share some templates you know you can use kanban discovery and delivery boards i'm sure you have kanban delivery boards you can sort of rejigger it and make sure that your discovery aspects are also part and parcel along with your delivery if you so choose again and it's pretty self-explanatory and when i share the slides you can sort of take it make it your own and one thing i wanted to slide it in there because it may not be necessarily something that came from the lean startup moment of whatnot is the idea of story refinement and grooming you guys probably do that right how many of you do the three amigo sessions and uh all of that anybody do that for your hand up so you wanted to explain very quickly what the three amigo session is about yeah you listed your hand yeah very quickly yeah right it's really about engaging your developer's testers and ba's in refining your stories with acceptance criteria and sort of furthering that intertestable examples and the reason i bring that in is is about collaboration and communication you're trying to bridge the gap across the disparate roles within your product teams at these days there are tools such as cucumber with a given then format and so forth that is supposed to be a collaboration tool but recently matt when i um the guy who created cucumber for those of you are familiar with the tool cucumber uh it's uh he said you know he's having a hard time you know trying to express acceptance criteria uh and acceptance scenarios by using terms such as given then and some of the more technical terms so example mapping is a very simple idea all you do is for your stories are you as you're refining them you basically say try to come up with the rules that sort of surround your story i'm creating a login let's say i want to create a login login form so the rule for that would be what's the min and max length you want your password to expire after 30 days these are all explicit rules that sort of surround your story and further examples are examples that illustrate that particular rule so the reason i mention this is there's nothing rocket sciencey about this but it's a very simple way to make sure you have less technical people engaged in the conversations as you're as you're refining your stories with acceptance criteria these rules become your acceptance criteria the examples become your acceptance scenarios for those stories again it's out there very very simple we are using this a lot of when we talk to our field agents and so what we have no idea about the the technical aspect but they're important for us from a business perspective all right this is just showing an example of how we do this so let's bring that to the end i just wanted to just say of lean startup ideas are not just meant for the startup companies a lot of these ideas that i'd mentioned i give you examples of them are permeating the enterprise as well right um and where i am for i would i would encourage you guys to check this out i know it's not particularly directly relevant to you but um it's called the us digital services playbook it's a set of 13 principles that guide all digital asset delivery in the in the United States in the federal space and if you look at it there's at least four or five of them that you can directly see as inspired by the lean startup data-driven decisions understand what people need before you start building stuff address the whole experience not just about the delivery aspect the restructuring the budgeting structures and so forth to to increase the kind of experimentation mindset that means a lean startup sort of support so it's it's what just taking a look at it look at it it's a quick read um and this is a lot i want to close this out with the one last thing i mean oftentimes especially in the private sector there's brand concerns i can't be doing this experimenting putting things out fake ads fake you know concierge prototypes and so forth because i have a brand to protect but oftentimes people that have spin off their own little things google labs an example and recently last month i was at capital one um the huge bank uh in the in the u.s they have capital one labs where essentially they're spinning off branches because they don't want to harm the the parent brand to do a lot of these experimentation but this is something that's sort of permeating the larger larger enterprise for sure so um so that's it any questions i know i'm almost out of time i'll be around if you have more questions about specific tools um hope you've gotten something useful out of this session and at the end i have some lists of books and a whole bunch of other tools i'll share all of this in slides if it's if it's of interest to you any questions one i think we have a minute or so but who's going to go and do some discovery no no discovery yeah what are you going to do intercept to users yeah try that you don't be afraid to try it it pays off big the books in fact uh jess jess is here i'm reading his latest book this is fascinating it's really really good talks a lot about uh discovery and metrics and so forth it's called lean enterprise and i think he may be carrying carrying it around so how many of you attended jessamble's uh session really really good book highly recommend anything else we call it a wrap well thank you for being here and spending your afternoon yeah thanks