 I want to thank everybody for coming to my talk today. It is the ninth one that I've given out of nine Ruby conferences, which is kind of cool, I think. So number nine, nine for nine. And I'm really hoping to make 10. I have big plans for talk number 10 next year, so don't miss it. But thanks for coming out. It's fantastic to see so many Rubyists having fun and talking about cool technology and talking about other things that aren't technology. It's really fantastic to gather with everybody. Now, as I was planning this talk, if you were paying really close attention, you may have noticed that the title changed about, I don't know, four weeks ago or so. It changed from, it was MDD and now it's EDD. And that's, I'll just let you know that if you propose months ahead, you have full prerogative to change your talk title before you actually give it. And that kind of fell out of preparing for the talk. The other thing that I'll warn you about is that I designed the title specifically to get you into the room. So it may, you should dump any expectations you have about what the talk may or may not be about. It's gonna be awesome, but it may not be what you think it is. So just take it for what it is. And my goal with most of my talks is to get you to change your behavior. So if you go out and you think a little bit differently or very differently and you act very differently than I will feel that I've succeeded, so don't let me down. Before we really get rolling here, I do want immediate feedback. This is a laptop's open, iPhone's out kind of a talk. So feel free to Twitter, use the hashtag EDD. If you need to say something really mean, you can email me directly. You can of course always blog about it. All fantastic, I'm very interested in feedback. So don't hesitate to whip out your iPhone and give it to me. So I'm gonna start with a brief history of test-driven development. So test-driven development, right? How many of you have heard of test-driven development? Okay, so pretty much the whole room, which is awesome because then I don't have to explain it, but one sentence, test-driven development is writing tests before you code as a design tool and not simply as a verification tool. So where did TDD come from? Where did we get this concept, right? It hasn't always, it's not a sort of something that was originally around when we started writing software testing was, but not test-driven development. So where did it come from? Well, in 1986, Kent and Ward are pairing at Techtronics and we don't know exactly what happened, but what we do know is that from them telling us about it that a lot of the ideas that we now have in terms of what agile is and what extreme programming ended up being, this was a very formative time when they were working together and they're both like super introspective people and thankfully they thought a lot about what they were doing in such a way as they were able to communicate to us and one of the things that they did very early on was a precursor to test-driven development. And then in 1994, Kent Beck actually published a paper on testing framework, which later was renamed to S-Unit and testing framework was a small talk unit testing framework and this was, nowadays it's like whoop-de-doo, somebody published a paper on a testing framework. Wow, we've never seen that before, but back then it actually was something pretty unique. It was kind of a new idea having a generic framework to support testing. And then in 1996, we have the C3 project, the Chrysler Comprehensive Compensation Project where a lot of the XP ideas flow from, they were all kind of put together at that point in one place and test-driven development was one of those practices that was used in the C3 project and that later made it into the XP canon, if you will. Now, we could argue about the sort of the effectiveness of the C3 project or XP based on the results of the C3 project, but I don't think we can argue the fact that it was incredibly influential on the development community from that point on. And then in 1997 Kent, back in Eric Gamma are flying over the Atlantic and they decide to code on something and out comes JUnit because Kent wasn't so much a Java guy but Eric was, but Kent knew SUnit and so they paired and came out with JUnit and this is kind of a turning point because you know, small talk, I mean who used small talk, right? So whoop-de-doo, there's a testing framework in small talk, awesome language but not a lot of usage, but Java was the hot thing at this point in time and so having a unit testing framework in the same mold in Java really changed people's perceptions of it and just exposed it to so many more people. And then in 1999 XP Explained is published, the XP canon is sort of put forward in published form and people learn about test-driven development and I would say that XP was plenty controversial but test-driven development was perhaps one of the less controversial pieces of it. Didn't have anywhere near the controversy of something like pair programming. And then in 2001, I wrote a little project called Lapidary that later became Test Unit was included in Ruby Core and gave a talk at the first Ruby conference in 2001 called Testing in Reverse. And I hope that in some small way that helped to infect the Ruby culture with testing as sort of a core principle, a core idea. And we've really seen that testing has become something that's kind of expected in Ruby. We'll talk some more about that. So TDD as standard operating procedure. First of all, I have a hypothesis to test. How many of you have tried test-driven development on at least one project? Okay, fantastic. So my hypothesis was that a majority of you would say yes. And a lot of conferences, even technical development conferences, you wouldn't necessarily get that many hands raised. And that says something about the culture that we have of testing and of test-driven development in Ruby, which is fantastic. And if you think about before versus after test-driven development, before you have, you know, there's not sort of, it's not accepted. And so you have a lot of coding by conjecture, coding by constantly poking at the code and running the code manually. Automated testing is something that you do because you have to at the end of the project. It's not an integral part of what you do on the project. Whereas now it's fantastic because I can walk up to almost any Ruby project and run rake and it'll run the tests for me. And that's just fantastic. And it's kind of a game changer in terms of the code quality that we have as community. And of course, a big factor in this is that you run a Rails generator and it'll spit out test code right alongside implementation scaffolding to at least encourage you and push you towards hey, you should really be writing tests. So even if you don't, you'll at least feel guilty about it. And also on a Rails project, if you run rake, what happens? The tests run, right? The very, the shortest rate command you can run on a Rails project is to run all the tests, which is fantastic. And so there's this culture and Ruby is really, I feel like and I'm admittedly a little biased but I think Ruby has led the way in a lot of ways in having a test infected culture around a language where it's an expected thing. It's something that we know we're supposed to do. And even when we're not doing it, we feel guilty about it. And part of that has been the amazing proliferation of testing frameworks that we see within the Ruby world. I mean, I think I would run out of fingers if I started counting them. And that is actually, I think fantastic that we have so dug into testing and are so interested in it that we can't help but just continue to tinker with the tools and try to find better ways and experiment with it. It's fantastic. But there are some problems in test-driven development land. A couple of them are that first of all, it's often misunderstood as a verification tool. Oh yeah, we do TDD so that our code is correct, which is really, or our code doesn't break, which is really different than we do TDD so that we design good code. And so test-driven development is often misunderstood as a verification tool. It's also often hidden from the business. The business doesn't really see the same benefits from TDD that we see as developers. And behavior-driven development, I think really grew out of these two sort of problems and tried to really get us to do a semantic reboot, sort of snow crash style, if you're familiar with that. And really, let's get people to think about this differently by talking about it differently, which is interesting. And I think has certainly helped some. Cucumber has sort of taken that even further and said, hey, let's make this even more accessible to the business by making it so that maybe they could even write these things. And I haven't gotten anybody in the business to write Cucumber specs yet, but I like to write them so I'm okay with that. But it is a possibility and a cool one at that. How many of you consider TDD to be a best practice? Yeah, so again, a majority of the room says TDD is a best practice and the rest of you feel guilty about not raising your hands, so it's good, it's good. And yet, so we've got BDD and it's helped us out maybe some to sort of overcome some of these shortcomings and yet, and yet, we just have in software development in general and this includes the Ruby community, we still have massive amounts of project fail. So many projects still fail. Why is that? And I think a lot of it has to do with the fact that test-driven development is fantastic at helping us find the right solution to a problem. Here's the problem, now let's find the best solution for that problem. But what if you're solving the wrong problem? Because you can build a fantastic solution to the wrong problem and the whole solution is waste. Because nobody's gonna use it, people aren't gonna pay for it, the business is gonna go under. So what if the problem is actually the unknown, not the solution, at least not yet, but what if we actually don't know what problem to solve? How do we currently find the problem? Well, the state of the art I would say is basically guesses and conjecture. It's okay, I say that the problem is this, this is what we should do, this is what we should solve. And you say something else and we sort of have this battle of egos and we see who's, willing to stand their ground harder, et cetera. You know, the sort of the pinnacle might be, hey, I had this problem and so I know that it needs to be solved and that's great. But still, even with that, we still end up with massive amounts of projects failing. And you know, another sort of state of the art is to whiteboard the business, right? So either in reality or metaphorically, we get in front of a whiteboard and we start drawing, okay, here's how it's gonna work. These people are gonna come to my social network and they're all gonna talk to each other and then advertisers are gonna come and it's gonna be this huge win. And you know, whiteboards are great, but they don't actually run, they don't actually make you money. We guess at what people's needs are. Does any of this remind you of pre-TDD on the development side? What did we used to do? What did I used to do? I used to go up to the whiteboard and say, wow, this would be a fantastic architecture. I love how my boxes and my arrows and my circles all talk to each other. This is fantastic. And then you go to build it and find out that it's got nine holes, nine ways from Tuesday, right? And you know, ego, it tends to be ego-driven. Chance rules, the projects that succeed tend to be the ones that happen to stumble on the right problem initially or stumble into it after a try or two. And you know, the other thing is that new ideas end up getting tried very haphazardly. So it's like, well, this isn't working, so let's try this and then this. And you just start throwing new ideas at it and trying all these different things and you don't really know if they're effective or not. It's just all very messy. So the really sad part is that what ends up happening is that the development organization ends up becoming this giant waste producing machine that just churns out all this waste because we don't actually know the right solution to work on because we don't know what the problem is that we're solving or the right problem to solve. We know what problem we're solving but whether anybody will pay us for it is a whole other matter. How many of you have been on a project that did test-driven development but flamed out? So I was hypothesizing that a majority of you would say that but it didn't look like quite a majority, so interesting. But what? Yeah, the rest of you are just embarrassed to raise your hand and admit it. So let's talk about EDD. So what is EDD and how does it solve this problem? Well, first of all, it's not a replacement for test-driven development or behavior-driven development. That's not really the goal. It is, EDD is experiment-driven development and the goal is to have a process that does for the business what TDD does for the development organization. How can we have something that allows us to vary in a much more systematic way, answer this question of what is the problem that we should be solving? And in particular, how can the development group contribute to answering that? Because the business people, and we love them, don't we? I'm one of them now, so in a lot of cases. The business people have a bunch of responsibilities in terms of this. They need to go out and talk to people. They need to be getting outside the building instead of standing in front of a whiteboard and saying, here's how things are gonna work. But what can the development organization, what can you, as an individual developer, do to help the business figure out the right problem to solve? And that's what EDD really aims to answer. You need to, the basic idea behind EDD is that you formulate hypotheses. So instead of just throwing opinions out and going off and implementing them, instead formulate your ideas as hypotheses. Here's what I think. I think that this will be more effective at getting people to sign up. And you think that this will be more effective. Well, fantastic. Now we have a hypothesis. So let's go and test it. Well, then I can't control my slideshow. Oh, sorry. So, sorry. Actually, just a second. Let me try this. This might help. I can actually turn off the phone. I don't know if you knew this, but you can turn on airplane mode and then you can turn the Wi-Fi back on so you can use it on airplanes too. Oops, let's see. Watch, I won't be able to reconnect to my Wi-Fi network. There we go. Yes. No, I'm good. I wouldn't be able to read my notes then. So the basic idea, like I said, is to formulate hypotheses and then run experiments. So to get scientific about proceeding instead of just running on conjectures and hypotheses. And another part of this, you can't, of course, just define an experiment. You actually have to start defining what are the desirable outcomes of development? What is our goal? What are we after? And this is, again, becomes a question back to the business where they have to actually think, okay, are we after trial conversions? Are we after people moving from trial to paid plans? Are we after people signing up at this particular price point? What is our goal? And so as they do this, then we're able to actually have this conversation with them about things instead of just getting features thrown over the fence to implement. And I think the primary way that this manifests or perhaps the first thing to try is the AB or split test. How many of you have ever run an AB or split test? Okay, so a decent number, but certainly not the majority of the room. My hypothesis was that a majority of you would not have run a split test before. Before we can really dig into EDD though, we need to talk a little bit about MVP. You gotta love all the three letter acronyms. This is great. You can tell I'm a business guy now. So MVP is the minimum viable product. Basically the idea behind minimum viable product is that before we can start running experiments, before we can start really learning about the market in an experiential way, we have to actually have a product that we've delivered, that we can run experiments within. And users that are using that product that we can run experiments against. And so we have to start getting to MVP and getting there faster. The focus is on what's the, basically if you release your minimum viable product and you're not embarrassed about it, then you did too much. You should be embarrassed when you release your minimum viable product. Because the goal is to get validated learning. It's not even to have a saleable product. You wanna be able to get some initial users using it so that you can start getting direct feedback from them and start running experiments. And then moving from the MVP into experiment driven development, we can start running experiments. So what's an example of an experiment we might run? So we might say you and I might be having a conversation over the table or over the internet or whatever and you're going, I know that we currently collect like these five fields of data and you think they're really important. But if we would only collect just these two fields, then we would get more signups. And I'm like, no, I disagree. I don't think it's really that big of a barrier. Well before it would just be basically whose ego was the biggest or who was writing the check or whatever. It would be based on conjecture and opinions. And oh, we might go out and cite like Jacob Nielsen saying that less fields is best or whatever. But how much more effective to be able to go? Fantastic, you might be right. Let's run an experiment. We will send half of the people, show them five fields and half people, we will show them two fields and we'll actually see which one converts better. And then when we have our next conversation, instead of it being about you saying, oh, look, signups are down a little bit. Obviously it's because we're still collecting those five fields and me saying, no, it has nothing to do that. It's this other thing. Instead, we have something measurable that we can actually have a conversation about facts instead of having a conversation around opinions and conjecture. And there are other aspects, I think, of EDD that we'll develop over time. One of them is measuring usage. So you might say this feature is really important. Everybody loves this feature. And I might say nobody uses that feature. But again, it's just opinions. Like you talk to one person who loved that feature, but I'm not really convinced. Well, without some way to measure that usage, again, we're reduced to having an argument and it being whoever's ego's the biggest or whoever's less more interested in keeping the peace. With some way to actually measure the usage of that feature, we can actually say, oh, look, two people use this and one of them is you. So we're cutting it out versus having to being stuck with just the conjecture. So one thing I think that's really important from this is that it still requires a whole lot of creativity. I think sometimes there's the impression that we're gonna start running experiments and it's just gonna cut out all the creativity. But actually, I think experimentation takes more creativity. You have to actually think through what are the alternatives here? What are things that might be more effective? And then the cool part is that you actually get to actually measure the effectiveness of your creativity, because you could be just super creative, but if it's not actually effective towards your goals, then who cares? But here you can be creative and actually see which forms of creativity, which ideas are actually effective and which aren't. And then finally, one really interesting property of EdD is that just like test driven development, EdD is primarily about design, but it also has this verification value of telling you whether your code works. Well, EdD has the same property, except basically what it's telling you is whether the business works, because what you can do is actually start to track, you're tracking goals, right? So you're tracking like, here's how many trial signups we have. Well, if you do an appointment and you look an hour later and you see that trial signups have dropped by like 80%, you're gonna go, huh, how do we break the trial signup process? We should fix that. Whereas currently, would you even know that trial signups had dropped off? Probably not. You wouldn't realize it for a few days or till somebody called up and said, I can't sign up for your trial, what's wrong with you people? And this way you can actually see when that happens. So EdD sounds fantastic, but the thing that really helped EdD to take off was actually having frameworks that made it so simple that it was like, why are you not doing this? It's just dumb not to do this because it's so easy to do. You don't have huge technical hurdles. You don't have these things that are holding you back. The business isn't saying, wow, every time you guys write a test, it like takes a whole day. I mean, that would be dumb. But currently with, if you wanna run an experiment, it might take you a day to set it up. And we need better frameworks. So let's talk about what's available and where we might wanna go. There are a lot of moving pieces for experiments. As I've dug into it and started really looking at this, there are a lot more pieces than I even realized. It sounds really fantastic when somebody describes it to you and you're like, yeah, we should totally do that. And then you go to start trying to do it and it's like, oh, this is hard. But it doesn't have to be that way. A lot of it's infrastructural things that can be shared between projects. And so by having these frameworks, it can really help us to do this. How many of you can name, at least hold your hands up and keep them up? How many of you can name at least one AB or split testing framework in any language? How many of you can name two or more? Okay, that's kind of what I figured. The majority of people can only name, actually I was wrong. The majority of people can't even name one framework in any language. And that's kind of telling. If this is important, if it's actually something that we should be doing, why is it so hard to think of things that help us do it? But what are the necessary properties of an EDD framework or the desirable properties of an EDD framework? First of all, it needs to be so, and this is probably the most important. It needs to be so easy to use that it would be crazy not to do it. Like you would say to somebody, why are you not doing this? This is so easy to do. Why are we not doing this? It provides so much value. Whereas currently, that's definitely not true. If you look at trying to do EDD, it's like, oh, this is really hard and expensive and I can understand why the business doesn't want to spend on it. Towards that, you need one line or maybe two lines max experiment setup. It should be like, okay, we're coding along. You think we should do this. I think we should do this. We just code an experiment going along and it's like we never even notice the fact that we just put an experiment in place. And it should be usable from day one. You should be able to start running experiments on the business people who are using the app and staging. And they're like, yes, this feature is so important and then you're like, well, you're not using it. So why is it so important? So it should be usable from day one. It needs to be accessible at all levels of the application. It's best if it's not something that you only do in the view. And much better if you can use it in the model or the controller or the view or in an app that's not even a web app, right? So we want something that we can access at all levels so that we don't have to have this thought of, oh no, I have to promote this into the view so I can actually run an experiment on it. It should be general enough to be used in different contexts. One of the awesome things about TDD frameworks is that when I go to somebody else's project, I know the general way that it's gonna work and I know generally how to run the tests and there are sort of some basic assumptions there. Probably you're using a testing framework that I've encountered before or if not it's probably patterned on one that I've encountered before. And so we don't want sort of a framework per application, much better if we can have something that may be customized but can be used across applications. And then finally it has to be fast enough that we don't hesitate to run experiments particularly in production. Do you notice any parallels here to TDD frameworks? There's definitely parallels here to TDD. And in general when I looked around even a month ago, this did not exist. So let's talk first about what does exist. So Google Website Optimizer is kind of the one that probably more people here have heard of than any other. GWO is nice, it's put out by Google. It works, but it has some issues. So it's not Ruby, it's JavaScript. So you can't really, it's certainly not accessible at all levels of your code. It has a really involved installation. There's several pages of instructions and they're rather involved and confusing. It mucks up your code. Yes, it's very ugly. What it does to your code, I'm not at all impressed with that. It's hard to set up experiments and it's hard to strip out when you're done because one thing you need to do is be able to strip the experiment out afterwards. I was gonna show you a code sample, but I had a problem. It wouldn't really fit. So no code sample. But basically what you do is you create an experiment on their site, you add control scripts, a control script and you add tracking scripts to track your goals. And all of this is basically JavaScript that you jam into your pages. And it just really doesn't hold up against what I think the four desirable properties are of an EDD framework. Now I think that it's actually really, it definitely serves a purpose. It's really fantastic if you have marketing folks who wanna try out multiple versions of copy and you don't wanna have to go in and redeploy the app every time they want to. But that is not really EDD. To me that is like marketing optimization. So what we need is the ability as developers to do this as we're coding and this does not serve that purpose. There's another one called seven minute abs that's not maintained. You might wanna have a glance at it but I'm not really gonna talk about it because it's not really maintained. A bingo is actually pretty cool. It's lightweight, it's actively maintained, it's being used currently on an ongoing basis to optimize a website at least one. So it's working code. I mean I never like to diss working code, right? This is code that's in production that's helping somebody, which is awesome. And he's released it for the rest of us to check out. What? Ruby. So here's what it looks like. You can come in and set up an A-B test. You have a name and then you have a couple of alternatives and then it dumps out an alternative for you. One of the things that a split testing framework is gonna do is always deliver the same alternative to the same user. So it delivers that and a different button gets rendered depending. And then you can track and say here's my desirable goal. So I'm gonna track this experiment against this desirable goal and tell you how the two different alternatives do in getting to that goal. Now there's some downsides to A-Bingo. First of all, there are no tests. So we just talked about how fantastic TDD is and there's not any here. But, and you can kinda tell when you look at the resulting code. I mean there's just some interesting design decisions in there. But I don't really like, like I said, I don't like to diss working code. So it's definitely worth looking at. Perhaps even worth using. Or at least trying out. So definitely check it out. Here's how it looks in terms of outputting. You basically say grab the experiment and describe it in words and it gives you something like this. Image button one had 15 conversions. The other had 12 conversions. It's not statistically significant which will flip over once you actually have enough to be statistically significant. So how does it do against the four properties? Well, you know, I think it definitely makes things simpler. Simpler than most other alternatives, which is fantastic. It, it, I don't remember. I think it is at least somewhat accessible from the different layers of the application. It is, it does look decently fast in the way that it's implemented. I'm not sure how sort of extensible it is. So I think, I think it does okay. So you've got one bingo, can you have four different bingos and see how your experiments do against the four different kinds of? Not sure about that. I don't think so. Yeah, be very much worth looking into and maybe it's easily extensible to do that. Don't look at this slide. Metri was my, is my project. It's half-baked. It's in the Tralee and GitHub repo. But it, it's like super half-baked. I was hoping to come here and be the hero and be like, I have the awesome EDD framework and you should all use it. But instead I'm putting out a plea for help because it ended up being more work than I expected. And you guys are all smarter than me anyhow, so. And that kind of brings me to this. We need you. The, the, the Ruby community needs you to create awesome EDD frameworks for us to use. This needs to be something fantastic that it just needs to be a part of our culture. We need to have an experimentation culture where hypotheses are the core of business and development communication. Where there's this habit of asking the business when they throw a feature at you, do you have an experiment for that? Where we reject overdeveloped MVPs. We need to start saying no when the, when people come to us and they're like, we need this and this and this and this and this and this and this. All these things we have to have before we launch. We just need to start saying no. And working with them to figure out what's the minimal thing that we could release that would actually help us to learn something as opposed to just guessing. And, and we need to build these tools to enable the experimentation. So that it's cheap enough that the question is, instead of, oh, that's, that's awfully expensive. Do we really want to do that? Instead the question is, why are we not doing this? It's so easy. So we, the goal becomes to have, through make rigorous experimentation, rigorous experiments as common, as test driven development, as standard operating procedure, that every project we start, every project we do, we run experiments to see if we're actually solving the right problem. Instead of just blindly, you know, going down this path. Oh, one more thing. So I gave this talk at Pivotal Labs a few weeks ago. Wow, there's still Legos on the floor from the blimp. So I gave this talk at Pivotal Labs a few weeks ago. And the day after I met with Asaph, you guys may know him as the author of Lab Notes, the Lab Notes blog, amazing hacker. And I told him about this talk that I was gonna be giving and we talked about it and he got kind of inspired and he went and made this thing. Vanity, so vanity.labnotes.work, you can go there now and check it out. It's kind of the new kid on the block. It's officially launching today. He's got this really nice blog post that he's gonna put up about it. And I'm just really thankful for the serendipity. He kind of had the need, concurrent with me telling him about it and he really kind of caught the vision for Edd and went and developed this. So let's see how it works. So step one, oh and it has tests too. Step one, you install the gem and you tell it, here's how I know if it's the same user or not so I can deliver the same experiment. Step two, you go in and you define an A-B test. So you say, it's a price options A-B test. Here's the description. And then here are my alternatives for this test. Pretty simple. Step three, you just go into your view or wherever you want to and you say, okay here I'm running an A-B test so give me one of those alternatives and always give me the same one for whoever the current user is. And it plugs that in. And then step four, you need to track your outcome. So we're gonna track against the name pricing options when somebody creates an account. And then step five, you output results and you get some pretty cool output. Oh, option A is this and it got 9.4%, this got 6.1, this got 2.3 and the best choice is option A but it's not yet statistically significant. So I think it does pretty well against the four properties although I don't actually know about the speed. Y'all should go benchmark it and let us know how it does. But I'm sure ASAC can speed it up and it's something that you can go out and contribute to and extend and hopefully I don't, vanity is fantastic. I think you should all go and try and use it but I also hope that you'll get irritated with it and you'll go and make your own because I want different options and I really think that we need some different options that will start playing off of each other and competing off of each other so that we'll find out what are really the best ways to do this. So this is not, vanity is definitely not the end. So that's it, would like any questions and feedback that you have immediately, yes? I'm associated with 1, Ruby 1.9 is not an option right now just not compatible. So do you know how hard it would be to make vanity work in 1.8? Is it 1.9 only? Okay, that would be a question for ASAP. ASAP are you here? Where are you? Come up here and help me answer some questions. So did you hear that question? So he wants to know if vanity can be made to work with 1.8. Any other questions, feedback? So question for all of you, do you want to go back and run some experiments in your current project? Let's see, okay, awesome, awesome. Any questions about how you're gonna do that? Or, I mean, hopefully I've helped to give you some ideas but there are challenges, yep. You gave the example that you would like to test a login form, a sign up form. In one case you have only two fields, the other half you have five. Right. And you run an ID test with that, you have to deal with the end of both results. Yep. You decide to find those and you have users that only have two fields, how do you do that? Well, again, we're talking about experimenting hopefully against a minimum viable product. So we're not, you know, in a case where you're Facebook and you have millions of users that might actually be an issue. But if you're running experiments again in an early stage project, like who cares? Like you can get those people, if those turn out to be really critical fields you can get those people to fill them out later. Yep. David? So in terms of XP and the idea of like customer tests up front or in EDD using something like food cover, how do you see this overlapping with that part of the process or do you see the work? See, I think that customer tests or EDD are fantastic and important when you know what the problem is that you're solving. They really help you to come up with the right solution. But if you don't know what the problem is, they don't really help. Like they can help to clarify some around like a specific problem and say, okay, here's how we should actually solve it or here's the some nuances, but they really don't help you select from the plethora of problems that you could be solving and from the variation. So I think there's actually limited overlap. But Matt, it seems like this is not technically experimental driven development because you have to actually develop both answers before you run the experiment. And then you decide which one to pick. So it's like both of these guys were talking about. So maybe, maybe semantically a little bit the wrong term. I think like more like experiment driven design. Well, you have to design both solutions too. Yeah. They have to both work. Yeah, they both have to work to some extent. So one thing that I've seen folks who are talking about doing this, touting is you actually don't necessarily have to implement them both. All you have to do is trick people into thinking you've implemented them so you can actually get them to make a decision. Because all you want to know is which way will they go. So don't implement the whole thing. Implement the absolute minimum you can to actually validate things. Making an experiment, all you have to do is assert what you think the actual answer is going to be plus hypothesis. And then you use the, so advantage, maybe have some assertions. Especially if you have these kind of literary prep or code. And you can say, these are my assertions. These are my hypothesis. Right. And we've heard once again there are some of the problems out of the system that automatically through the drive-off is just wrong. Right. So I have a question about vanity. Is it useful at all if you have multiple web servers or is it at the level where basically you have to ask each web server what the performance was. Into which? Redis. Redis. Yep. It's still AB. It's still AB because you just have nothing here, right? You have it display nothing versus this where you have it display something. So you can still do that. For instance, in the vanity example here, right? So you have, sorry, went back to one too far. So you have those alternatives. Like you don't necessarily have to output those alternatives directly in the view. Here in the view you could actually do an if and check which alternative it is and do something much more complicated, like not have a block of code at all or actually do some computation or something. So there's a lot of options and nothing is certainly a valid AB test. You don't know what's going to choose and not like the variety. Well, the goal, again, you're tracking against some goal. So what you want to know is if you display nothing, did they then accomplish the goal or not and in what amount? So back here. I said you need, but user experience people do this kind of testing all the time. So did really, what about EDD is different than regular user testing and do you feel like some of the precepts, you know, AB testing is just one tool and what's normal user experience is likely to consider user testing. You feel like those other tools should also be something that should be focused on? We should look at them, but the difference with EDD is that it's specifically focused on how the development organization can contribute to the process. So UX people are fantastic and the tests that they run are fantastic, but there are things that the development organization alone can contribute to this and that's really what EDD is focused on. How can development contribute? So we should look at what UX people do and say, what can we pull from there and apply to the development process? And the same with what the business people are doing when they're going out and talking to customers, what the marketing department's doing when they're going out and talking to people. So we're looking particularly for what we as developers can do to contribute to the business finding the right problem to solve. The biggest issue though is should we do the feature or not? And so, and the investment is in building. So in order to run the test you have to have built it, so it's always. Not necessarily, well that's the thing though. Think in terms of minimal. What's the minimal experiment that you could run to figure out whether to build the full feature? You don't have to build the full feature to be able to run an experiment to figure out whether it's worth building. For instance, I've heard of people offering, putting in their preferences, paying an extra option around a new feature that they wanna build and just seeing how many people actually click that option to try to turn it on and if they click it, they go, oh, this is an option that we're considering. Thanks for letting us know that you're interested or whatever. You don't actually, I mean, why not? Why not? So, but, so what? I mean, it's not. If nobody is, so if nobody's interested in it then nobody will click on it and nobody will ever care. If people are interested in it you've just validated that so now you can actually invest in making it the best thing that can possibly be. So I don't feel that it's actually disingenuous to the users to present options like that. Yeah, there's a lot of different forms. I think, so I have seen some that I think border on the unethical. There's some challenges in there but I definitely think there's a wide range of things that are very ethical to do that don't involve having to build the whole feature to be able to test things, to experiment. Yep. Yeah, it seems like a key piece to this is being able to measure the outcomes here not just using it to drive, like using it to drive design but because you're using measured outcomes not just a business analyst saying well I like that better. So do you know of any tools that sort of facilitate that process like maybe not specifically using Vanity or like something that is like specific just something that you can, maybe it's just the reddest backend or whatever that you can deploy to two different servers or deploy to two different branches or whatever and say like I want you to record these outcomes so I can use these in a variety of branches. So one interesting thing that Vanity doesn't do yet but I hope that it will down the road is instead of testing outcomes against a particular experiment instead of finding outcomes that are across the whole application saying a trial conversion is always desirable. So now as we define experiments you give me output that says, okay here's how we're doing against this goal and this goal and this goal because one interesting thing that you want to know is oh this turns out that promising people free stuff that we're not gonna give them increases trial signups like this but all of a sudden conversions to paid are going like this because people find out that we're crooks. So you want to be able to track multiple experiments against multiple goals. So that's and that really becomes something where it can actually be part of your so in the lean startup world they call it an immune system for your application where basically you can be watching these metrics and even have automated processes that watch these metrics and say oh we just deployed and signups went down like we're not getting any signups anymore we just deployed bad code roll it back automatically. So that's definitely a direction that I want to see things go. Yeah. Yes. Not a question just a resource. Eric Reeves and Steve Orton. Yes. So EDB is in many ways is very much inspired by Steve and Eric. Steve if you're interested in the business side of figuring out the problem you need to read the book The Four Steps to the Epiphany. Fantastic book. It is not copy edited and it's published by Cafe Press and it's like course notes but the content is absolutely golden and it's totally worth reading. You need to read that book. And Eric is writing about melding basically customer development which is what is in The Four Steps to the Epiphany with agile development. And the EDD grew out of me hearing Eric say you need to be running these experiments and they're fantastic and they're great and then me going how in the world do I do that Eric? I actually tried to do that and it's really hard. So thanks for telling me about all these great frameworks that you had at this place that used to work but it's really not helping me very much. So could I please have a framework that I can actually use? So EDD is basically how do we as a development respond to what the business is starting to do in terms of really paying attention to what people want and how can we contribute to that process? So one more. I just wanted to ask about vanity and just in terms of the randomization. If you have multiple experiments running does the same user always get A? No matter. Yeah, okay. So simplification. Yeah, and there's way different ones because it might not be the same program. Everybody is important to you. Interesting. Those are options. Cool, cool flavor. All right, well thank you all for coming. Really appreciate it, oops, sorry. If you have, again, feedback, very welcome. And I'm around all conference. If you want to talk about it. Thank you.