 Well done. So Layne asked me what I wanted her to say to introduce me and I said don't worry about it because I have a little explanation that is actually relevant. I am an author, I am a recovering consultant, I spent about ten years traveling the globe trying to help organizations become more agile than I refer to it as I went home to the mothership. That's when I went back to Pivotal Labs which is where I had originally learned how to do agile. I now work on Cloud Foundry at Pivotal which is a spinout that includes Pivotal Labs but is a much bigger thing and I'm happy to talk to anybody who wants to talk about that. Afterward I am the director of quality engineering there and because Pivotal Labs does not do QA in anything that looks remotely like a traditional practice and because our software development practices for Cloud Foundry come from the DNA of Pivotal Labs that means that I don't have an entire QA department somewhere off with people running tons of regression tests because that would be sort of it just isn't the way that we do things. In fact we don't have a QA department at all so I'm the director of quality engineering but there are no quality engineers that I direct that's just not how we work. Instead the way I refer to my job is that I see to the care and feeding of feedback cycles and then people look at me funny and say what do you mean by that? This talk is my explanation of what I mean by that but first let's talk about what we need feedback on. So if we're going to build something somebody somewhere recognized that there is an actual need presumably or an actual desire of some form. There's some desire out in the universe for a thing to happen and they set an intention about what that thing is going to be and then somebody goes off and acts on that intention and builds something. So we have an implementation and then we hope that the implementation somehow actually had some relationship to the actual need. Right? And the thing is we need feedback on each of those relationships. We need to know the extent to which our intentions did in fact relate to the actual needs out in the universe. We need to have feedback on the extent to which our implementation matched our intentions and then ultimately we do really want to have feedback on the extent to which our actions related to the original need. These are feedback loops and I'm sure that since you heard of from Adrienne Cockroft this morning about the OODA loop and Plan Do Check Act that was the one that I learned about before the OODA loop and then of course for that matter lean startup. We've got build measure learn. Jeff Patton taught us that's better because there's only three steps. Right? These are feedback loops and why do we want feedback loops? Because feedback loops help us steer so that if we got our intention wrong, we misunderstood the original need, we can course correct. Well, there's a thing about feedback latency though that makes it really super hard to steer. And I first learned about feedback latency when I was reading Peter Senge's fifth discipline. It's a book on systems thinking and he has a marvelous example that stuck for me because you see his example involved being in a hotel and at the time I was traveling a lot. So he had my attention right away because he was talking about a world that I lived in all the time and his example in particular involved the shower at a hotel, a very old hotel, a very very old hotel where the time between when you adjusted the temperature on the shower and anything actually happened related to your motion to adjust the temperature on the shower was incredibly long. And so his experience was much like mine in hotels starts off cold. Adjust the heat. Nothing's happening. Adjust the heat. Nothing's happening. Adjust the heat some more. Crank it all the way. Now we're burning. Oh my God. Oh my God. Oh my God. Down. Cold. Only nothing happens. Stepping out of the shower. Okay. Okay. Okay. Now we're freezing and repeat. You've had that experience, right? Too many of us have had that experience with software, right? Why would that be? Well, this is the development cycle that I grew up in in the industry. Nobody does this anymore. I'm really hoping none of you have actually ever seen this anymore. Is there anybody in the room who has never been on a project that involved more than a year worth of planning and specking and oh, that's wonderful. There are new people in this industry who have never experienced this. So I'm going to share my pain with you guys. Everybody else in the room is going to cringe because they've been there. All right. So with the best of intentions, we've recognized a real need. Somebody goes up, analyze it. They called this requirements gathering. So we go off and we gather requirements and then we stick a bunch of really, really smart people in a room together so that they can design a solution to the requirements that we have gathered. Months have passed. Large documents have been produced. They have been signed off by many, many, many executives. We have sign off. We have agreement. We go into the development phase. I was at this point in my career always in a traditional looking QA department. And so the development phase was theoretically going to go to this date. And what actually happened was that it went well past this date and the amount of time was for test was compressed. But test was an entire phase that was happening at the end. And that whole thing was somewhere between 12 months and 36 months depending on what I was working on. This sounds familiar? This was painful, right? This resulted in the shower. Because what's going on here? The very first opportunity that you have to see the your intentions match to your implementation or your intentions match to your implementation was 12 to 15 months into this project. You can't steer. There is no time left. You have a massive amount of investment in the work that has been done so far. Worse, the very first thing that you have that the, yeah, the microphone's cutting out, fortunately for you, I'm really loud. The first opportunity that you have to see that your intentions were, in fact, although well intended, misguided, is the point at which you release. Or, depending on the organization you worked with, they might have had some period called beta, which was always fun because we'd get feedback that we couldn't actually act on. Beta was like the way that people phrased it was beta is for your next release, but it doesn't actually help you with this one. That's the hot cold shower problem. You can't steer. You can't adjust. So why, what, let's look a little bit more at what's going on here and why it's so dysfunctional and why nobody does this anymore. So we look at the analysis phase. At this point we're speculating that we actually understand the problem that people are describing. We go into the design phase and we are now not only speculating that we understood the problem, we are now speculating that we have a solution that will actually address the problem that we think that we understand but aren't sure because we were speculating about the problem to begin with. Now we go into implementation and it gets worse because we are speculating that the implementation actually matches the intentions which actually match the actual need that we theoretically actually understood, only we have absolutely no idea whether or not we understood the real need. And we've done all of this investment. And so we look at this and the speculation curve is going up and up and up and the first point where that starts to collapse into something approximating reality is when we get into the stabilization or test phase. All of that area under the speculation curve is risk. So we can't steer and we've incurred a massive amount of risk. It's no wonder software projects failed. Speculation is empirical evidence is going to trump speculation every single time. Now if you know me from Twitter or GitHub or just about anything, you know that my handle is test obsessed. And some people think that that's because I came out of QA and therefore I'm obsessed with testing and that this is about that whole software test, software QA, self identifying as the bug hunter, they've got me all wrong. I am test obsessed because I am obsessed with getting empirical evidence and dismissing the speculation as fast as possible. So what we learn from this massively long feedback cycle is that there's actually a time value to information. Anybody who's taken an econ or business class has probably countered the idea of the time value of money. A dollar today is worth more than a dollar tomorrow assuming that you don't have negative inflation. A piece of information today is worth more than that same piece of information tomorrow. There is a time value to information. So we want to get it as fast as possible. So I know we'll do this thing called Agile. I read about it in CIO magazine. Lots of people, lots of organizations think that that means that we're going to chunk the work into small iterations and so that means that obviously we're going to be doing away with all of this risk that is associated with really long running cycles without a lot of feedback. And so we've got little risk curves, but they're little and this is going to go great. This is awesome. This is going to mitigate all of those risks. Only it turns out it doesn't actually work if you're not getting enough feedback during those. Remember we need feedback on the extent to which our intentions match the actual need, the extent to which our implementation match the intentions, and then actually ship it and find out how people react to it. We need all of these kinds of feedback. So during Agile iterations it is incredibly common for organizations, especially those practicing scrums, sorry all of you CSMs in the room, but especially those to completely forget about massive amounts of feedback that are needed and instead plan for this idea of a stabilization sprint. Those things make me cringe. Anybody ever been on a project where you had a stabilization sprint? How well did it work? Yeah, not so much. Here's why. Let's take a look. So it turns out that we've got this speculation. Now with each iteration we are speculating that it's going to be good enough once we get to the end. So what's happening to our speculation curve? Guess what? It looks exactly the same as if you're on a waterfall project. So it turns out that just Agile alone isn't exactly going to be the answer. That is in fact fragile. So let's look at the different kinds of feedback loops and then we're going to look at how to improve the feedback loops in your organization. So we've got automated unit tests and we've got automated system tests and we've got CI build with all of those automated tests and we've got exploratory testing which happens to be something that I really enjoy, hence the whole book on it thing. But anyway, then we've got manual scripted tests. Hint, don't do that, but we'll talk more about that in a minute. But then we've got statistical analysis of production metrics and you look at some of the stuff that Netflix is presenting and oh my goodness, it's been so much fun to hear Adrian Cockroft and Roy was just giving a talk on how Netflix does operations and some of the stuff that Etsy is doing with taking a look at stuff going into production, watching for anomalies in the behavior in the system and using that as an indicator that we've got a real problem. That is an incredible feedback cycle, a super powerful one. And then ultimately of course customer opinion and if you do the whole lean startup thing then you can form hypotheses about the extent to which people will click through or pay for or whatever it is the metric that you're measuring for. Have that hypothesis, take an action, put it in a practice and see how it panned out according to your original hypothesis. Oh look it's a feedback loop, it's an OODA loop or it's a plan-do-check-act loop, however you want to frame it. It's a feedback loop. Okay, very important that you might care about. Now the trick is that if you look at all of these and I'm telling you fast feedback is important because otherwise you have really long feedback latency and if you have really long feedback latency then you're going to have the hot cold shower experience with your software projects which turns out to be a lot more painful because it's gone for a lot longer than 15 minutes in a given morning. How is it that you're going to do all or most or or cover the most important bits in here while still simultaneously being able to have all of these things be fast because this is a lot of stuff to do, right? So this is the place that a lot of organizations get themselves stuck in especially those organizations that want to have a very traditional QA organization that's responsible for the feedback loops. See in a traditional QA organization they're the ones who are writing the tests, who are automating the tests, who are running the tests, who are reporting on the results of the tests and so you've got built-in feedback latency so those traditional models don't work at all. So what do you do? Well here's a very basic recipe and then we're going to break it down a little bit more. First of all, automate all of the checks. So let's separate the notion of testing. There are two sides of testing. One kind of testing is checking to see the extent to which the thing that we're producing conforms to the expectations that we have of it and one of the reasons why we had regression tests to begin with was because we needed a mechanism by which to tell if some change caused the system parts of the system to violate the expectations that we had for it. One of my favorite examples of that was in an organization that had a system where it involved shipping software and various mechanisms to ship and so they had to have some code that dealt with DHL versus Federal Express versus UPS versus the US Postal System. One of the changes that went to production was to fix a bug with one of those. Let's say DHL. I don't actually know which one but it kind of doesn't matter. So they had this fix to fix a bug with the DHL integration. They put it into production. That bug fix worked flawlessly for DHL. That was now the only shipping mechanism that worked because fixing the bug for DHL caused all of the other shipping mechanisms to break. That was an organization that did not have automated checks to verify that with every single change not only did we do what we intended to do but we also didn't break anything else that it was already working. So we need to have automated checks. However that's not enough. We also need to explore to discover the risks that we weren't thinking about at the time that we were designing and building the software. So there are those who will say that you can't automate every test. I agree and disagree because I believe that you actually can automate every check and if you're finding that the argument you're having to have internally within your team is we can't automate every check because it's too expensive then you've got a software design problem that you need to fix. The problem is not that you cannot automate those tests it's that your software is inherently untestable. Now that said remember I said testing is checking plus exploring and although you can have automated exploration and Netflix's Chaos Monkey and I just learned about Chaos Gorilla and Chaos Kong next door. Those are great examples of things that go off and do stuff that's automated but it relates to exploring and the exploring is an inherently human activity in the sense that humans have to figure out how to explore. You can have software aided exploration but the exploration piece is inherently human and much of it is in fact manual. So you want to automate all the checks so that you can very quickly find out if your software does what it was supposed to do without violating any expectations that it was previously meeting and you want to explore to discover if you've got any risks or vulnerabilities that nobody had thought about before and then you also want to be releasing frequently. In fact when we talk about Karen feeding in the feedback loops the entire point is to shorten that latency from the time that you recognize a need set an intention and then realize that intention in code and then get feedback about the extent to which that realization actually met an actual need to shorten that as much as possible so that your organizations can steer much more effectively. And that's what organizations like Netflix and Etsy and others are doing so phenomenally well. Facebook and Google and so they release early and often. Even if you are in an environment where you're shipping software and I hear this argument frequently or I did when I was a consultant remember I'm a recovering consultant. Hi my name is Elizabeth. Yeah so when I was a consultant I would hear frequently from my clients that who were very large enterprises that they didn't see or were software vendors who were serving very large enterprises the software vendors didn't see any reason to have a shorter release cycle because they knew that their clients didn't want to take or their customers did not want to take those releases anymore frequently but it meant that they were inherently building these cycles that were so long they would be unable to steer and shockingly the people who are reaching out for consulting help were in fact the ones whose lunch was getting eaten by much faster competitors. Go figure. I love all my clients they were great. Yeah anyway you aren't any of them. Don't mean to imply that I am criticizing my clients but so even if you are in an organization where your core customers do not want to upgrade more than a year once a year or once every two years you can still be at least rehearsing delivery early and often or in fact releasing and it's just that any given one customer is going to stay on that version that they first adopted for n number of months or n number of cycles. All right so let's talk a little bit more because so far this is all motherhood and apple pie shorten your feedback cycles we're done right now how that's nice it's cute that you think that it's that easy Elizabeth right how do you actually do this let's look at a feedback cycle so at some point somebody sets an intention they wrote a spec they wrote a story depending on what your software development process looks like somebody put their hands on the keyboard and wrote some code then it gets checked into a source control system and then somehow it goes off for testing and then after testing it goes off for some form of productionization whether that's a deploy or whether that's a burn it on to whatever you're going to ship it on and then we get to observe results and generate idea feedback into this cycle this is the this describes anything right this describes a one month iteration in an agile shop this describes a three-year release cycle in a very large enterprisey kind of company so far you're with me right let's dissect it a little bit more the very best thing that you could do starting today to shorten your feedback cycles is to slice your stories at pivotal we like to work in stories that are something that a pair could account we always pair program because we do because that's just the way that we work and we want to have stories that a pair could accomplish in a day or two if it's any longer than that we need to split it because it means that we are likely to be fish tailing our way through a very very long cycle and we're not going to have any way of gauging the extent to where which we're actually getting momentum getting traction so the very first thing that you can do to shorten your feedback cycles is to make sure that you're working in really small pieces at a time as you're working as developers are coding testing locally shouldn't be anything new to anybody anymore I actually did work on projects where testing locally meant they compiled it I have heard result from other people who tell me that their engineers didn't even do that I am stunned and appalled but I have not heard such stories in years and years and years so mercifully I think our industry has moved beyond that just about every organization that I had worked with for the last several years was in the habit of testing locally but oftentimes the testing locally was just the engineer testing their peace the integration wasn't happening at pivotal we practice continuous integration the way that Martin Fowler described it which is to say not only do we run all the tests locally or at least the the fast set that the team has agreed on but we also then do a fresh get get all the code down run all of the tests and if anything breaks we're doing a local integration and we do the local merge and then we also fix any red tests before we check back in so at the time that we have checked in our code we have no reason to believe that the continuous integration build is going to fail now in my current project on cloud foundry cloud foundry is a massive distributed Harry system at the time that I joined the project last November in order to have just the most minimal cloud foundry that you could have that was still an actual deployment of it I think at the time I joined it required 45 VMs. Yeah it's pretty ridiculous right you try to deploy to 45 VMs you're going to be there for a while and so testing your changes and their effect on the system locally just wasn't really an option. We got that down to 18 VMs within my first few months there then we invested even more to get it to the point where we actually could deploy our entire system to a single VM running a bunch of containers so we our containerization system is called warden so we can now deploy our own system to our own containerization system if you haven't heard of warden you might have heard of Docker if you haven't heard of either one of those don't worry about it it's virtualization within virtualization it's inception all the way down. Oh yeah and we also test drive everything which means that we've got immediate feedback about the intent of our programming I mean that's test driven development is actually a design practice that gives you immediate feedback about the usage of your stuff. We also eliminated a bunch of wait states so automating all of your checks is super important even once you've done that you may discover that your tests run super super long in CI and in fact we had this problem you can eliminate the redundant tests I don't know who coined it but my favorite phrase for automated testing is playing Tetris with your tests you know how in Tetris you want all of your pieces to be at the lowest level possible and you'll maneuver until they can get there test Tetris is pushing your tests down to the lowest level possible so make it a unit test if you possibly can mock out the rest of the world and just just make sure that you your expectation is met in the code that you're writing don't try to make it a full-stack test and then ultimately run them in parallel so on cloud foundry I mentioned before about the whole deploying thing 45 VMS took a long time we also had another problem and that was that the process that the team had adopted involved code reviews that were managed with this thing called Garrett anybody dealt with Garrett I'm so sorry I some people love it and so I'm gonna offend you right now if you're one of those people who love it but Garrett can just die because when we talk about feedback cycles what we were seeing one of the engineers reported to me that she spent an entire week just waiting for one of the senior level people today to review her change so she had had to make the change on a on a separate branch on a private branch and then she waited for the code review and master is changing out from under her and so hard days were spent go see if there are changes sink the changes make sure her stuff is merged in rebase push it back up resubmit bump the the request for a code review keep rerunning the system test which by the way they had to run manually they were automated tests but you had to kick them off manually there was no ci that could run this thing set up at the time that's a week of engineering of a talented engineer that's a total waste of time and when we look at feedback latency it's an absolute killer so we took out Garrett we pair all the time anyway so we're getting automated automatically does a side effect of how we work we're getting code review we're running local tests we're investing in being able to test even more within our local environments that are sitting on our desks then we check into master and then ci takes over and tells us the extent to which is healthy so the end result is that we went from days to weeks to get feedback on a given change to being able to get feedback within hours and that sped up the pace of our development immensely wow even worse than i expected it to look because my final point is that it's not enough to have the feedback cycles you also need to radiate the information out and apparently i need to make another picture space because that's more blurry than i thought but what it's showing you is a radiator that's showing the state of all the builds in our environment we've got something like 70 builds across all of the various teams and across all of the various parts of the pipeline and if we weren't making those visible to everybody we wouldn't be able to coordinate at all so here's some lovely references and i also didn't know we were going to get the phoenix project in the bag but there you can check one off of your checklist um and i recommend these other books and i don't think i left myself time for any questions at all oh good anybody got a quick question so i'd like to request that if people have a question please come up to the microphone and we have time for one lengthy one or two quick ones elizabeth are you using um the cloud in your test configuration to get faster cycles as well um so it's kind of funny because we're kind of also building the cloud but yeah we use travis ci extensively we also um pivotal labs has an open source uses chef to provision you a jankin server in the cloud on aws it's called um cyborg c i b o r g horrible pun um and so we're using that we're not using any of the other services that you might be thinking of but then we also don't have any web much web based stuff all right so uh one thing i experience a lot when when working in these kinds of highly automated environments is that you still end up spending a lot of time maintaining that what do you see as the normal ratio for getting work done and then also maintaining all that infrastructure to keep it working so there's there's an interesting dichotomy that you're drawing that i'm not sure i believe in because what i heard you say is what what percentage of time do you spend getting stuff done versus messing around with your infrastructure and i think it's all getting stuff done dealing dealing with the fact that the infrastructure has problems so um we do have this challenge right i mean we are dealing with the fact that the infrastructure can have challenges and especially if you're working on aws they don't actually guarantee that anything is going to continue to work and oh look your instance is going to get retired lucky you um so we do end up spending a fair amount of time however i wouldn't be able to give you a percentage of of what percentage of time partly because our engineering teams are responsible for that as well and so it's sort of baked into how they work thank you very much elizabeth thank you