 All right, let's start from the beginning here If you ever hear me say anything like this, you know that I'm lying or at minimum. I'm fooling myself I've spent I've only spent the last 20 years in software development. I Had some distractions for the first 10 years of my my career But when I started early on I thought I was pretty good It's now about 20 years later doing it that I realized that I don't know much at all and that I've spent a lot of the past couple decades Fooling myself, and I'm just now learning how to stop that But what I want to talk about actually is how the most exciting thing for me Now these days I work as a consultant. I help a lot of different companies adopt Different ways of thinking and working especially when it comes to different especially when it comes to products and I'm starting to see whole companies That have stopped fooling themselves now when I started in software development in you know in the in the 90s back when 90s was it but back when mosaic was a browser and and there was this new Netscape thing I well when we did software projects we would imagine a big idea we would lay it out We would plan it we would design it and we would build it and we would launch it and it would either be wildly successful or Die big and then we'd find someone to blame and everything would be okay again But I'm starting to see companies that realize that that's a bad trend and they're doing things to break out of that and one of the big cultural adaptations is that they're becoming More comfortable with with failure. They've they've learned how to fail. They've learned how to accept failure Well that quote plays in my head a lot that if you're not failing or not learning and if you haven't watched that Derek Sivers video you should if you've got kids you should make your kids watch it Because it's good for kids But enough about these companies. I want to I want to talk about me for a bit And I've got to reach all the way back to when I first started and I need to talk about my biggest well my biggest instances of fooling myself I Started as a software developer and I think I'm a pretty smart guy and I was Pretty good and well, I started a company. I was given a first job Our company built software for brick and mortar retailers and we take it on this weird special job to build a Site to sell aircraft parts online. I still remember part numbers for aircraft parts The light above your seat is a 315 in case anyone asks now This is bill Bill is still at my company bill is a great guy and I can remember looking bill straight in the eye and saying look bill I've been working with these other developers. I can outcode them. They can't break my code. I can break theirs Bill I don't you you're not going to need to test my code and Bill smiled and nodded and and then bill tested my code and broke it within minutes and I learned that how these tests are people do do things and and You know one of the things I've seen over the the course of time is Over the course of the last couple decades is testers move in and be really part of teams I've seen the discipline of testing are really advanced now that we understand that Testing isn't just checking things that testers become really honest to goodness partners with developers I love this particular picture. Just it was a recent a recent workshop we were doing and the lady Well, the lady at the top of the screen She's a tester the the guy at the front is a developer and they've got this cool symbiotic Relationship where he's sort of talking out loud telling her what she's working on and she's kind of saying what she's testing And she'll say this is this is looking bad And and he hears it out of one ear while he's typing something else And these guys are coding and testing at the the speed of light The they're not waiting for any bug reports to be filed They're working in a very different way and well, I'd fooled myself about what testing is and what testers do and the lesson learned for me is that testing is a pretty critical discipline of It like there's a lot of other disciplines and it takes years to get good at them and if we're gonna do anything significant we need each other and well That's that's one area. I started by fooling myself in with software development and and well the the other area I started with was part of my distractions in my 20s as I had gone to art school and Back then it was called communications art which kind of meant a hybrid of illustration and advertising design and things like that and I stayed there long enough to realize that this is a lot of hard work and I don't want to do this and Software is pretty easy by comparison and So I went into software development and I eventually wandered back in and well If you're a software developer and you know anything at all about user interface that makes you the UI person and compared to other well Compared to other software developers. I was white hot I could make software that just really looked good and and working with my teams Well, they'd I could go from just a rough idea of what something should be to something that looked like a good usable user interface and Everything was great. I was working with teams who were on that long run of building things and things were coming out good And just out of curiosity who here in the room has ever set with a customer or user While they've used your product before wow, this is so different. It didn't used to be this way at all Think back to the the first time you did that how did that go? it doesn't go very well a lot of the time and For me when I shipped I started to realize that well user interface design is a lot more than than looking pretty This is a simple model that I draw on napkins all the time. So that is a napkin and that is a Time I've drawn it and this is me dissecting just these really simple concerns If I look at a piece of software at the bottom I'm going to put this utility layer if I find people that have a problem a user that needs to be able to accomplish something and I can give them a tool that allows them to do that. I'm going to call that the utility layer Now if I've given them something that really helps them And I make it easy to learn and easy to use and an efficient than I can layer usability on top of that and Then if I've given them something they need and they can learn how to use it and I can make it Sexy and consistent with my brand and well, there's more to aesthetics than just looks there's that expression you can't judge a book by its cover and the reason that's an expression is because people judge books by their covers and Software that looks bad on the outside people kind of believe it's bad on the inside So if it looks good on the outside that inspires confidence and we can do better with We can do better with software. We're confident in now if we can just get that raw utility thing, right? We get some reasonably good products. Everybody's familiar with Craigslist and it's it's not sexy and If you dig in too deep, it's not horribly usable, but man, it's it's it's super functional It has stuff we need to do Now if you skip over that utility and this is where well my first speed bumps where I'd first fooled myself Is by looking at stuff that I could do things in and that looked good but I had maybe I had some elements of usability and aesthetics and Can you think of any I had a hard time thinking of? Popular products that had usability and aesthetics because usually they die on the market But the only one I could think of what was that? Does anyone here own a segue? Checking, okay This is a product that's been looking for a use for a long time. I'm not sure I know one's ever come up to me and said, you know, I want a device that allows me to get around at a modest speed not too fast and and doesn't carry anything and looks really dopey, can you can you help me out there and It's a product that's still looking for uses these These guys don't look threatening at all. I died if they make me want to shoplift so And then there's just the aesthetics thing if you make software look good and we all know what that's called That's the lip sticking the pig thing and I was pretty good at that when I started So it took a few times in front of people to realize that Crap, I'm building the wrong stuff. I'm not building things that solve people's problems So the first trap we start to fall into is look What's wrong here is that we need better requirements and if we had better requirements all this stuff would go, right? Now I'm got I've got to give this little thing if anybody's heard me talk before I give this every time because I just can't Think without this little simple little mental model. I Need to start by telling people it's a lot of you've heard me say this before that you are not here to build software and And you never have been you're here to change the world now That may sound daunting, but in fact every time we ship anything We make a little at least a little ripple and sometimes a really bad one and the Here the model starts like this We look at the world as it is now as it is today and we find people that have problems or people that are confused or Frustrated and when we look at these people well we get ideas Now we might call those ideas products or features or enhancements When we hand those off to people to build we might start to call those specifications and at some point in time We might call them requirements now I'm going to I've got to go off on a slight rant here for a minute I I started in a in a product company, which is a lucky thing to do I had lots of customers my customers did not work for my company I knew that if I went to any one of them and they told me what they wanted of And I gave it to them I would be dissatisfying some other customer and I was responsible for the sexes success of my product And they weren't so I made reasonable choices about what should go into that product Now my company had 30-something people when I started but we grew up to a few hundred people and We started hiring a lot more traditional software people and I remember of this weird tipping point where we had a lot of people and A lady came up to me and said hey look Jeff the the team that you're responsible for we need you to add these things We need to add these things that into this part of the product that you're looking after I said great. Tell me a little bit about who's using the product What kind of problems they have and how this will solve their problems and she said well their requirements And I said okay. I get it. Tell me a little bit about What are the customers are gonna be using it What are they gonna be doing with it and she looked at me like I was stupid and said their requirements and That's when I learned that that word means shut up And at some point in time and software development we sort of forget that we're actually solving problems And we fall into the trap of we're just building stuff now Let me come reel back in from that tangent because well we we It's important to remember that what we call requirements are our bright ideas that solve people's problems Now we move forward through this cycle if you're using an agile process if we're using anything We eventually deliver something and that delivers into the world later And what we hope is true is those people that were unhappy are now Happier and not everybody's happy and there are people you just can't please now Well everything between that idea and that delivery I want to be I want to give some clear language about this That's all output if you're practicing in an agile process. That's the junk we measure with velocity That's the stuff we ship. That's the stuff we track with project plans But it's not what we want what we want is what comes after that Outcome is well what happens when things come out. That's why it's called that and Out those people that are happy aren't happy because they saw the release notes or they saw the box But software doesn't come in boxes anymore But they're happy because whatever they were doing before they can now do it different or better the way we measure outcome is in terms of well people's behavior how they behave differently as a consequence and and maybe how they feel after they behave differently and Then one of the word I'm going to splash in here is impact now The truth is the model doesn't start with us Solving problems for people it actually starts with our company being concerned about revenue and asking us to look into particular people to solve problems for and Ultimately our company wants some sort of return on investment And it takes lots of people doing things differently and feeling happy about it before we get that longer term impact so outcome refers to the well the things that we can see and measure and Adrian was just describing When we deploy something we start Running tests and what we're testing isn't how much money we made well it could be if we've got an e-commerce site But we're usually testing How people behave differently those are the metrics that we gather and those are the immediate outcomes but if we're trying to Do things like improve brand awareness or overtake a competitor the market those things take a little bit longer I don't test those with my metrics the least not hours after I ship Your job in this in software development is not to build more stuff It's to build less stuff your job is to minimize Output and at the same time maximize that stuff Now I keep this model in my head because I pay attention to people who are using this stuff and pay attention to what happens afterward and Hey, how many people in the room are using agile development before? Before now any time If you're using these user story things there's this common template that gets used and the template is the as I want to So that and well this template is about talking about who what and why What who's the person that matters here what? What I did oh I have for them and how do we turn that frown upside down? How do we turn now into later? What I learned early on building software Was that it does not matter what the requirements are if I deliver exactly what people ask for and the outcome is bad We lose The corollary to that is you if you deliver If you don't deliver what they asked for and the outcome is good you win That's the trick here now Given that you could say look then all right if it's not better requirements. We need we certainly need better research We need to better understand what's going on. We need more data Now I've got to reach back and tell a story. I talked to him actually just last week So I'm going to include the story of this particular guy This picture is not from last week. You can tell by the size of the monitor. It's not This picture was taken of well in 2001 when I when After a brief hiatus to work in silicon. Well, excuse me to work here in San Francisco to Get my dot-com millions after that didn't pan out. I went back to the company. I had been at and and This particular guy Andrew I'd hired there. He was fresh out of college He is now a VP of product development of this particular company and he's he's doing well Some of you might know me for that this story mapping technique and I should pull the picture down a little bit That row of cards on the top is actually the first story map. I ever built in 2001 But let me tell a story that I well the conversation I had with Andrew last week was about one of our customers This is them now we build software for large brick-and-mortar retailers and other other types of retailers and This particular customer was really unhappy with us With my prior company now unhappy with Andrew in this situation Because the return process on our software was Clujie and bumpy and and Andrew could look clearly at the data and see that returns were a fraction of the transactions that went through why were they making such a big deal about this they were less than 1% of all transactions and How bad could it be? but This particular company has some unusual return policies This is a company that's extremely liberal with returns They care a lot about how they handle people when they return things and when you walk into their retail store That's the the counter where people do returns now. This is before store hours, and I I Andrew didn't have a picture of what it looked like in during store hours But he was there he took this picture and he was there during store hours and there were lots and lots of people there and What he saw was lots of people jammed up lots of people behind counters trying to help them as quickly and politely as They could and the software that he had written Jamming things up and making life miserable for everybody Now the data showed that it was only a fraction of transactions and the transaction time wasn't all that bad But when Andrew was there he saw that this looked really bad. This does not feel good The returns all end up in bins and they bag them and they go to the back room and and there's a lot of those Now I pulled that picture just quickly because I wanted to point out that guy right there That guy is Eric and that's and I was lucky to start out Eric is the CEO of our company and whenever anything went wrong, we got on airplanes and we flew to where it was and we saw people and we saw it going wrong and The lesson I learned early on is you don't get empathy from data it's empathy that we're looking at and And there's this word that gets a little overused in in a lean in a lean context And I've worked in Japan before they'll often refer to Gemba as this floor where the developers work But for products it ain't there now. I want you to imagine for a minute that you are Jane Goodall Is that working Hey, my talk is supposed to end at exactly what time I'm watching my time to make sure get through all this Good, okay, because I haven't adjusted my time zone. Yeah I'm already really late Yeah, good. Thank you now Jane spends a lot of time out in the field. It takes a long time. It's arduous. She hides in bushes She she watches gorillas and and somebody might look at her and say geez Jane. This is horribly inefficient We've got this idea for you will make your research process much more efficient We'll call this chimpanzee on site And we will bring the chimps directly to you and then if you have questions about the chimp You can just lean over and kind of observe or watch or ask and well, that's not the kind of Thing we're doing we we don't get empathy that way either They might get empathy with us, but that's just not the way it works So don't don't do that do this and well when I look through my hard drive I've got piles of photos of where I've gone out to do this Hey, this picture was taken not all that far from here just off Fremont Street at a big company that does I can't say the company name, but if I said they screwed up Libor you might know who they are But in this particular company we were told gosh you can't you can't talk You can't talk to the portfolio managers. They're busy people Millions of dollars and and yeah, and we can arrange a meeting for you Sometime and I said great no problem. Where do they work? Well, they're on the third floor But you can't go down there and I said okay No problem. Hey We're curious about getting some pictures of their work area Can we take pictures and no no no that's that's not okay We'd have to get permission from legal to do that and I said okay that's that's okay, and then I went over to the elevator pressed the third floor went down and and and this guy says nobody ever comes and talks to us at all and They are they're thrilled to talk and And we learn a lot about context and well these guys are traders also and they work together in teams of four and and because they work closely together in teams to They're not traders excuse me their portfolio managers They need frequent conversation with each other So somebody in their wisdom had said these guys need a chat feature in this software and it's got to be high priority because they really need to talk and They hadn't actually gone down and see that they have arranged their decks so they can stare at each other in the face But so being there showing up helps a lot and The only reason I want you to see this picture is when the one of the mornings I showed up their analytics had crashed and this is a this is a big deal It causes big trouble and they're all looking at that the quanta there Who's trying to fix the models and get it going the important thing for you to pay attention to is how freaking dark It is there because for these guys to do this work. They've got to be up and at the They've got to be there five in the morning and I had to get up it I don't know what time 3 30 or something to get from Alameda over there That was if you want to see where things happen. It takes a little bit of effort Now actually I want to go to this particular guy And I'm gonna mention Edmunds comm a couple times here because this guy works for Edmunds. This is a teak A teak was responsible for analytics and Edmunds was a company that well years ago We'd been asked to help them convert to a design thinking process And one of the first things we did is say look you have to start talking to the people you build your software for And then this was a frightening thing and So a couple of those people are developers and one of them isn't can you guess which one isn't I'm just curious But a teak is talking to someone and they During this day he interviewed several people and he's in charge of analytics I remember at the end of the day he was sort of just struck that Look, I can tell you exactly what people do I've always been confident that I can tell you precisely what they do But it's not until today that I that I realized that I could never tell you why You get what people did from data, but you're missing this one critical component the lesson learned for me again is that well data isn't empathy and So okay great Let's move on from there then if we can't do research if we really need to get out there and get empathy What we need is a better product owner and because they're the ones that screw things up If we don't get the right person in that job It's it's their fault and they need to be able to tell us what to do Now I want to give you this simple model that comes from a guy. I've known for a few years now and a Marty Kagan Marty started as a product manager one of the first product managers in Netscape and he joined eBay I believe he was the third product manager hired at eBay and he when he left eBay He was in charge of 60 product managers. So he kind of grew up in this And he will say that if you are a product person you are responsible for identifying a product that is valuable to the people who are going to Use it and buy it and to our organization It has to be usable by those people They need to be able to accomplish what they need to with it or there's no value and Finally it needs to be feasible to build given the time and tools that we've got that the ideal product lives there and Because this is a lot of things to worry about Usually we might have a product a person a product manager that looks after the value But we need someone who really understands users and works closely with them now and products in software product companies That's usually a user experience person in IT centric companies that might be a business analyst And then finally we've got this person that's usually left out the one that receives the requirements the engineer It's these three people that work closely together Now I forgot to plug in the audio. There's just a little bit of audio. I had here Because I had to hear from Marty Hey, is the audio on? Yes It's in Organizations that I follow behind Marty life is lot is great for me because he's already said look if you if Product manager is going to work. You've got to have tight collaboration between this small balance team you need people that work together that understand how things are built why things are built and who's going to use them and Well, I generally find that engineers are left out and I grab this little snippet from Marty On feasibility the little secret in software teams is that truth is Usually the best innovations actually come from the lead developer And the reason for that is that the lead developer typically knows what's possible better than anyone. I Consistently see really fabulous things. I just left a Just left a workshop with a company Monday and Tuesday, I was in Scottsdale, Arizona The minute we start involving developers with talking directly with users with actually getting a chance to come up with those Requirements actually solving problems. They light up. They come up with a lot better things than well than anybody has before and they And they light everyone else up with them I'm a hater of that product ownership term because I want everybody to own and I want product owners to be better leaders So product owners lead a cross-functional team and they help everyone take ownership then There's this last thing here that basically if fine if we get the right people and they really Work well together and they understand their customer that then we'll get it right right Now there's Marty again, and I was last summer. I had to work with him on something I was sitting in the car next to him and he nonchalantly said this phrase like if you're really good at this stuff You'll be right about a third of the time and I said we say that again. He said well typically Between 50 to 80, you know 50 to 80 percent. I had him repeated back And it got a little more concise than I like but 50 to 80 percent of the things that we put out there fail They don't meet their original expectations. I said you're kidding me because I still was fooling myself about that I thought I could we could get this right and he said no no no if you're good at it you fail most of the time That's the way it is and a lot of our ideas fail So we pull them down and then no well the ones that might succeed a little bit We've got to iterate them a lot before they succeed this is hard stuff and I start to recall those Charts that people show especially agile trainers show these standish group these chaos report things where Depending on the year you look at it anywhere between 65 and 75 percent of features are rarely or never used That sounds bad when you hear it Ruling out some obvious things like you know airbags are rarely or never used in cars So they must be worthless but The truth is a lot of ideas seem like good ideas at the time It is difficult to you know, we might start with an Opportunity and from that opportunity we might come up with a couple different solutions Now, I don't even remember what this old quote. This is old now and everybody knows the the zoons gone away, but the there are some people that really believe that zoom thing would work and That can get a little scary for those people Actually, yeah, I do have to get the next picture. Yeah, that's the one I want Now this is Eugene Eugene's in charge of product management at Edmunds comm and I said look this is my I think Marty's crazy Eugene I'm hearing from Marty and you guys have been Designs thinking stuff a long time. You're getting a lot better at metrics. You're paying attention to how well things work You know now how often these things work Marty's saying that if you're good You'll be right about 30 percent of the time or something like that and Eugene says to me No, we're probably right about two in ten two times out of ten. You can tell he's not a developer He would have simplified that fraction to one in five But one of the things that's happened with Eugene over years You know, they hadn't been talking to customers. They hadn't been experimenting, but over years. He says, you know, we've we've built we've had to really Build measurement muscles up We started by using simple testing tools like test and target But we found to really get the precise metrics that we want. We've had to start rolling our own Stuff to to measure and now we're pretty good at this and that's why we know we suck at this and The lesson for me is that this is hard. We're usually wrong and we have to start planning on failing a lot more often than we used to Now the one other team and I'll talk about a couple teams These are the teams that have the longest term relationships with and I've seen them evolve over years This is a company called snagajob.com. No one hears anyone here ever visited this site before If you were looking for a job at Burger King, this is where you go so This that was a team at their regular stand-up and they've got this nice board and because they're focusing on things the last edge of their board isn't Done the last edge of their board is ready for release and then it sits once it's released It's tagged it sits there and it gathers data and it has to gather data depending on how long it's been there for for hours or days or weeks and they've got metrics they're trying to influence and Well, they explicitly released they explicitly measure and nothing leaves that board and until they've learned something Now I wanted to put this guy on there. This is Thomas and I talked with Thomas in preparation for this I'm trying to gather more recent lessons I find that there are companies that are in this inside this journey and things aren't going so well They don't want to talk about it. So that's been tough but the people that are coming out the other end have really learned some things and This particular project where these guys have been measuring I watched them thrash for close to a year on this thing They were trying to replace an old version of their website with a new version of their website and the revenue per visit The revenue per visitor would just never come up They couldn't replace the the old with the new because they were going to lose money and not just a little a lot of money and At the end Thomas who's in charge of products that yeah, yeah, we we scrapped that But the important thing was that after chewing on this thing for a year. We got a really great architecture out of it and I'm learning more and more that great architecture is sort of important to doing this stuff and Well, one of the big Assumptions that I hear that is that really great architecture is all about scalability and performance and What I'm starting to understand now is that's not it. In fact, that's the stuff. We've been focusing on for a long time Now stag a job has had this weird architectural journey and I think How many of you in the room have been involved with your company's architecture before? How many of you got a crap legacy architecture that you're trying to get rid of right now? It's almost the exact same number of hands Now a stag a job had they had they originally started their site and the architecture didn't have a name And they needed to fix it or make it better so they named their new architecture eagle And so they built eagle, but it had a long oota loop It took months to build this it's not months excuse me years to build this thing and it was kind of hard to use and and bumpy and and it wasn't getting adopted well and and People got tired of it and they said look we need to launch a new architecture and we're going to call it instead of eagle We're gonna call it Phoenix because it will rise from the ashes of eagle and then Phoenix trunged all along for a year or so and It was starting to run the same course and then they wanted to layer a new architecture on top of that and They decided to call the new architecture Tucson because Tucson is some kind of sideways from Phoenix if you look at it on a map and The idea was that they kept trying to get a good strong scalable architecture but one of the things that happened when they were trying to chew on this problem is they focused on getting things released fast on Making things measurable on making things testable and well their new architecture is called somersault and it's sticking Oddly it's easy to use. It's easy to build things in and along the way They've built scalability underneath it. They built performance underneath it, but it is optimized around speed to get things done now I was recently Bill Scott came from Netflix. Was he big at Netflix? Or is he just a troublemaker? He's big yeah and well, he's at PayPal now, so he's definitely a troublemaker there and I saw him recently sort of complaining that look at traditional engineers start with Reuse and they end up with something. It's just way too complex He had a nice mantra this use before reuse mantra Look, let's make sure that people can use it at all before we focus on that reuse thing and When I go back and talk to Eugene just recently he said look we built our CMS with AB testing in mind We well, I think what I had already said they used to mix in a lot of different things But over time they've realized that we need faster more surgical results and the companies I see they're doing this Well aren't using off-the-shelf tools. They're rolling their own And one of the interesting things Eugene said is you kind of have to go through you kind of have to go through this Having nothing adopting a big tool realizing it doesn't work and throwing it away and building your own if you try and leap frog Frog those things. It doesn't work You kind of have to pay your dues with it. We have such an uncomfortable thing to learn But the lesson learned here for me is that engineer We need to engineer first for experimentation and focus on scalability performance later Now great fine week. We've got an architecture Then fine all we need to do here is to build measure learn now I'm not gonna explain agile to this crowd. I probably I don't need to explain lean startup to this crowd not much and The build measure learn loop is a simple idea. It's it's it's the new UDA it only has three steps, so it's better and It basically means let's get an experiment out there and There's I've seen this weird bias where we can just get quick simple experiments and well a lot of the organizations I work with will build simple paper prototypes or other things like that and and Talking with Bill's cohort and talking with Cody recently He said look you can only sir learn so much with the highest fidelity prototypes here. You eventually need to get real Now I'll see if I can explain what I mean by that here And I love this quote from Bill Buckston who's a noisy guy in the user experience world that that he says a Difference between high fidelity and low fidelity is just stupid. There's only right fidelity and wrong fidelity So a lot of what I focus people on is trying to find right fidelity for the right time And the two axes that I think of is well all the little the looks real and the works real axes Now if I create something, it's a simple paper paper prototype. Why does it look all that real? It looks kind of like software But I can move the parts around and I can use it if it was software and so it kind of works real and What I used to see a lot of and still see too much of sometimes Is lots of really sexy photoshop documents? And they really look great, but they don't work real at all And then one of the things I'm starting to see a lot more of is a lot of faking it a lot of faking it with manual Processes are using spreadsheets to calculate things and I'll explain an Edmunds example here in a second and They really work real, but they don't look at all like software and Ultimately we want this when we finally want to make sure that people will actually use this stuff We need something that looks real and works real But it isn't engineered for release It's it we might a B test that in our environment and we ratchet this up We can start to test value with very simple things and and prove that people value this We can start to let people step through things on paper or simple Aksure prototypes to see if they can use it, but at some point in time We want to start measuring outcomes and remember outcomes are what happens when things come out We have to let people use it in the wild and see if they actually behave the way we expect them to and Well in terms of architecture one of the things that you they've at Edmunds They now have an architecture where front-end developers can Not deploy headless apps. They heads only Basically only front-ends with with mock databases and they behave like real software, but they can do this very quickly on their own so I Want to describe an Edmunds experiment and one of the things I'm seeing more and more in organizations is that to develop To develop a real experiment. We've got to spend time and money and sort of use the whole organization This is Edmunds site today and they launched a feature called price promise and for price promise The idea was simple. I can go in to site. I want a car. I can look for a car And I can find dealers that are offering guaranteed low prices. They've learned that people really hate haggling with dealers and Well to test this particular feature Edmunds had to get the cooperation of a few dealers in the in the Portland area They had to build enough software to go live on the site. They had to put it in their a b switching frameworks only a certain amount of people got it and The whole test took many months to run They shuttled data back and forth to dealers on in spreadsheets. They manually loaded it into databases on their end They had lots of phone calls with dealers but at the end of the day they had to see if people really did use it and Really did go to dealers and really did buy cars Because if it didn't work end-to-end It wasn't really a test Marketing had to support it by giving a branding sales had to support it. This is whole organization experiment experimentation and What's interesting as that Edmunds would have never done this a couple years ago. It was not okay. They were They either spent a lot of time arguing about it and either do it all in spend millions or not do it at all That's the big change that I'm seeing now I Know I'm over time and I want to make a last point here. In fact, this is the last point Well, and I have to make it with three slides. Is it look we keep discovering things we discover Working closer together helps and actually kinds of things help We discover understanding more about users and doing something that's more like a design or design thing thinking process That helps we discover experimentation helps and we are pretty sure that we are going to crack this thing But What we're finding or what I hear from Eugene is that look we've been down this road enough times that we realize that We keep falling into this trap that We keep thinking the process or the methods will automatically produce success But we now know that that's usually not the case We are on our own and what was interesting was Thomas. He's in a completely different way He's a little newer and from his perspective. He says look the process is here to help us learn. It gives us some structure for learning Now, I don't know what's true But the lesson I'm gonna my last takeaways here are I don't know pay attention and Don't assume you're right Now I'm over time Well, hold on. I never have time for questions and Jez is too young to know who the who is Who remembers the who? Good It's time for a break, right? Do I want to take questions? We should let people go. I never have time for questions But I think you should watch Pete Townsend This guy is incredible He is so wiry All right, that's all I got. Thank you very much