 Well, enterprise is kind of a negative word. Enterprise kind of means something that's so large that it's completely dysfunctional. And DevOps means having everyone work together to enable, I don't know, rapid learning. And so it's a cultural thing. So we've got kind of, I never realized this before, two words that describe the culture of organizations. And I guess, what's it called when you have two words that contradict each other in a sentence? Anoxymoron. OK, so we're going to start with an oxymoron. And we're going to try and find a way to resolve this problem over the course of the next hour. So I've prepared a diagram of the enterprise, which looks like this. So we have the business. And then we have project teams building projects. And then we have infrastructure and operations, who actually has to run things. And even with the operations, there's different silos. There's the service desk, infrastructure team. And then behind's a large filing cabinet and a tiger. There's the DBAs who hate you. Any DBAs in the room? Any DBAs? OK, that's fine. We can be rude about the DBAs. Yeah, OK. And why do the DBAs hate you? The DBAs hate you because you make their life miserable. Because you add columns to tables and forget to index them and cause a 100 times degradation in performance. You create queries and add joins and forget to do query optimization and cause 100 times degradation in performance. And it gets found out when you do the database migration during the release. And it takes 12 hours instead of five minutes. And then the DBAs are up all night trying to fix it. So this kind of represents Martin Fowler assures me this is valid UML for a big ball of mud. And of course, we have a machine that goes bing to let you know that everything's working. That's the most important part of operations, the machine that goes bing. And so what you find is we have projects. Someone in the business who's very highly paid decides we're going to create a new product. And at that point, a project is started to build a new product. And after many months and potentially even years, this thing eventually gets released into production and wired up to the machine that goes bing. Hopefully it works. And then at that point, the project team disappears and gets assigned to something completely different instead and never actually gets to see that thing working in real life or gets the feedback on whether or how it works or whether it gets a return on investment, god forbid. So it turns out this is very unsatisfactory, mainly because the feedback loop is really slow. And when the feedback loop is really slow, often by the time you've released the thing, it takes you a year to release it. And then you find out that nobody actually wanted it. And it's kind of, in enterprises, what happens is when something is too embarrassing, we pretend it hasn't happened. And so what happens is we pretend it hasn't happened until it loses so much money that someone gets fired in senior leadership, and that kind of gets swept under the carpet too. So the idea of agile is that we should get faster at doing this, so that at least it's not quite so embarrassing when things go wrong. And I think the key idea of agile is expressed in something that Eric Rees came up with, which is the Lean Startup. So anyone heard of Eric Rees? Lean Startup? OK, a few of you. So the central idea of the Lean Startup is this, that we should be able to take ideas, build something, and then find out if the idea was good in the first place. So any time you have a business idea, you want to start with a new product. You think it's a really fabulous idea, and everyone's going to love it and pay you loads of money. And most of the time that's wrong. The ratio of product ideas to actually successful products is something of the order of 20 to 1. So for every 20 ideas you have, one of those will actually be successful in terms of delivering a return on investment. Now, when you start up a project for a year, you're kind of basically assuming that you're going to be that one idea that's going to be successful. Statistically, that's unlikely to be true. And yet that's the path that we optimize for in enterprises when building products. And God knows I've been on several of those. And so the idea is instead of building it, which is the most inefficient way you can possibly get feedback on whether it's a good idea, maybe we should try and run experiments and try and find out if the idea was a good one to develop in the first place before we actually go and build it. And so this is what the Build, Measure, Learn cycle is. You build something, you get data from your customers. Was it a good idea? And then you iterate rapidly. And once you've started on product development, you can keep doing this. Every time you have a new feature, don't build the new feature, come up with some experiment you can run to find out whether the feature will be valuable and run that experiment and keep building small increments in order to make sure you're actually working on something that's valuable to your customers. Now the problem is the economics of that are hard in the enterprise because we've got these silos. So to take an idea and give it to engineering, engineering needs specifications. That takes a long time. And then you've got to get it tested and then get tossed over the water operation. So the transaction cost of taking an idea and pushing it out to production is very high. And so when the transaction cost is high, the economic solution to that is to create large batches. So that's why we create large batches because the transaction cost of putting a change to it is high. And so this is kind of a natural effect of the structure of our organizations. And inevitably, I'm going to talk about continuous delivery, what we want to do is optimize for time around this cycle. So go through this cycle as fast as possible. And that time is called lead time, the time we can get around this cycle. And when you optimize your process for lead time, lots of really interesting things happen. But the point of continuous delivery, which is my personal thing, is that you want to optimize for lead time. Now, how good do we think we can get optimizing for lead time? Well, there's a company called Etsy. Anyone heard of Etsy? OK, a few of you. So Etsy is a shopping site. Well, rather it's a storefront. And people can make their own craft products and sell them through Etsy. So that's great. Anyone have relatives in the USA? OK, so if you want to buy your relatives a nice present, go to edsy.com. You'll find something nice for them. Etsy deploy 30 times per day to production by 200 plus people. They had over 10,000 deployments in Etsy. And the reason they're doing that is to learn from their customers. They have an AB testing framework. They always have a ton of experiments in production. Would it be better if we structured our page like this or like this? Maybe we can give people special offers. And they find out if these feature ideas are good by running AB testing production and actually seeing which of those ideas leads to higher revenue for the shops that they're selling from. And they make about half a billion dollars of revenue every year. So this is quite big scale. But maybe it's not quite enterprise scale. So let's consider something really enterprise scale, like Amazon.com. Who's heard of Amazon.com? OK, so people sometimes ask me, well, this continuous delivery thing sounds good. Can we do continuous delivery at scale? And I say, have you heard of Amazon? And they normally say yes. So these are the deployment stats from Amazon for last year. You can see that they had a mean time between deployments to production of 11.6 seconds. On the maximum number of deployments in a single hour, 1,079 deployments to production in an hour in Amazon.com. And on average, 10,000 servers was being updated by each deployment, a maximum of 30,000. So that should hopefully be surprising to you and give you an idea of the scale of what's possible with continuous deployment at a large enterprise company. Yes, yes. So for example, they might change the color of a link or change the position of a button. And if you look on Amazon.com, you can see the bit where you actually buy, the presser button to buy, they're always changing that. So they'll do things like they'll add a drop down for the quantity or they'll add a gift wrap option. They're always experimenting with different ways to structure the site. So the changes can be very, very small. And as long as they produce statistically significant results in terms of changes in revenue, it's valuable to do it. And the great thing about this is it means you can run a ton of experiments. Most people never run experiments on the ideas they're building. So this is turning the dial up to 11 in terms of what's possible in terms of, I've got an idea. Let's run an experiment. Let's run an experiment. Yeah. Well, is running the experiments valuable? So if you've seen an AB testing framework, what happens is the experiments typically run for many days. And until you can see a statistically like a 3 sigma or 5 sigma difference between the results, it's not statistically significant. Once you have that, then you find out if the experiment was valid. So I mean, people are doing this in a proper scientific way where they actually have the number of sigma difference between the A and the B in order to find out. And the revenue might not be big, but when you're running thousands of experiments, the incremental benefit is large. Absolutely. So was there another question at the back? So some of these are going to be bugged. I mean, I don't work for Amazon. Amazon is extremely secretive about what they tell you. It's very hard, and God knows I've tried, to find out things about Amazon. They don't talk about it in public. So I don't know the answer to your question, but I'm guessing this is anything from a CSS fix to a bug fix to an increment of a story. Again, I don't know the answer to that, but I'm guessing any change that's made to any existing product plus the addition of new products. Well, again, I don't know, but I know that Etsy.com are very public about what they do. They always have a large number of experiments in production at any time. So they're always running the order of between 50 and 100 experiments in production at any time. And that's Etsy. I'm assuming that Amazon is running hundreds, if not thousands of experiments at any one time. That would be my guess based on relative scale and extrapolation. So I'm not going to take any more questions, so I can get through the rest of my material, but if we have time at the end, I'll take more questions. So how do we get better at doing this? There's this thing called business agility that the Economist Intelligence Unit talks about a lot. And there's becoming popular. The main obstacles to improve business responsiveness are slow decision making, conflicting departmental goals and priorities, risk averse cultures, and silo-based information. Does that seem familiar to anyone in the audience? Right. So this is why you can't be Amazon, basically. And these are difficult problems to overcome. Most people who want to achieve continuous delivery, it's not the tools and the practices and the processes don't exist. They don't use them. And the reason they don't use them is because you have to change the culture of your organization. And that turns out to be the hardest thing to do. What people tend to do instead is buy tools and then assume that using the tools will somehow fix the problem, which it never does. You can be as horrible at continuous delivery with Git as you can be with Clearcase, it turns out. So there are several enterprise-class problems. Organizational ones, projects and program management, governance and culture. And I'll address some of these. So agile is supposed to be the answer to some of these problems. And so agile is normally inflicted, first of all, on the engineering department. And what do you think the response of the other departments is to this? You think everyone's really thrilled? That's not normally the reaction. And why is this? Well, at the business end, it's because you actually need a lot more collaboration between everyone involved in software delivery. And the business likes to say, I've got a fabulous idea. I'm going to write it down. And then you're going to build it. And I don't want to talk to you until it's done. And that is stupid. And it's the leading cause of the inability of organizations to adopt agility is that the business isn't interested in validating their hypothesis. The business isn't interested in talking about the details of their products, and certainly not finding out if customers are interested in them. And the problem is, if we were building trust bridges, that would be OK. Trust bridges have well understood characteristics. You can model them linearly. And once you produce a specification, you can normally build them, and they are normally delivered on time and on budget, and with known quality, and they don't fall down. Whereas products are complex. They're complex systems. You cannot predict the interaction between your products and the customer. So for example, the Obama campaign was doing AB testing on campaign emails. And they wanted to find out which subject title would lead to the email being read most. So maybe you can think for 10 seconds about what subject line in an email might actually cause you to read the email, which is the most likely. So I can't remember the exact subject that won, but I think it was either hi or hello was the subject of the email that got the most readers. That's probably not what you thought. And if you think about that, think about building a whole product. Think about how many assumptions you make about what your users may or may not find valuable. I can guarantee you that you'll be wrong about every one of those. AB testing yields surprising results. And you will be surprised. And the reaction of lots of people to that is to prefer not to be surprised and just to assume that they're going to be right, which turns out to be poor. The problem on the back end is actually getting stuff out. So he was at Neil Ford's keynote this morning. So he showed this diagram, which she stole from me. But then I stole most of my stuff from him. So that's fine. And most of the problem, actually, with building software comes at the back end of the process. We're good at estimating and doing this agile stuff. But then nothing actually gets delivered to customers as a result of this. It has to go through this process before it gets to customers. And it's only right at the end here that it starts actually delivering value to users. And by the way, when we actually find out if what we did was valuable. So it turns out the agile manifesto does talk about this. The first principle of the agile manifesto is our highest priority is to satisfy the customer through early and continuous delivery of valuable software. And apart from the subliminal advert for my book, I think this is the reason why we all get out of bed in the morning, whether we're dev, or ops, or testing. Whatever we do, even the DBAs get out of bed because hopefully they want to do this. So how can we do this? Well, what we want to do is get fast, automated feedback on the production readiness of our systems any time there's any kind of change. We're good at testing source code changes, but do we test Apache configuration settings or thread pool settings with the same rigor or router settings with the same rigor we test source code changes? Who has automated tests for their router changes? Anyone? I can bring down your system much more quickly by changing a router setting than by changing source code settings. I can do it like that. So why aren't we testing these changes with the same rigor that we're testing our source code changes? It's crazy. Any time you make any change to the system, again, complex systems. Complex systems have emergent behavior. We can't predict how they'll work. Why aren't we testing all these different aspects of our systems, not just source code changes? And we need to do that rigorously. Any time there's a change to the system, we need to get some fast level of feedback. I mean, first you'll run unit tests to test the source code changes. But then any change which passes the unit tests, we should trigger acceptance tests, which run in a production-like environment. So we're testing things like does it actually work on a cluster, not just a single machine? Will it work with realistically sized user load? We should be running more rigorous tests against builds which pass the automated unit tests. And then any build which passes that should trigger downstream. We've got a build that's passed the automated tests. Now we need to do manual validations, like exploratory testing, usability testing. Maybe we're doing manual security testing, or performance testing is expensive. And then any build which passes all those validations, there should be an email sent to the ops people. Here's a build that's ready for deployment, and they should be able to press a button to deploy that build or any other builds to their users. And this pattern is called the deployment pipeline. And the idea is that every change results in a build, and every build is a release candidate. And then a release candidate that wants to marry the beautiful woman. You're the beautiful woman for the purposes of this demonstration, thank you. I want to marry the beautiful woman, but it's a swayamvara. First, I must pass a series of tests to validate that I can actually marry the beautiful woman. I have to get the big pile of gold, and have the shooting match, and prove that I am the most virtuous. And then finally I can perhaps marry the beautiful woman. So what we've got is a swayamvara for builds. And if a build passes all these different validations, and we have even a slight nervousness about pressing the button to release the production, what that says is that our validations are not good enough. We need more tests. And so this is what continuous delivery implementation looks like. It means creating a path to production, having a series of validations, and then taking every change and trying to make it, trying to take it to production. And what this does is it proves that that change is not fit for production. And if we can't prove that the change is not deployable, then by definition we should be able to deploy it. You can never prove the absence of bugs. What you can do is detect the presence of bugs and kill any change that contains bugs. So in terms of moving from nothing to continuous delivery, it's hard. So I want to give you one example. Who works in embedded systems here? Embedded? OK, a few of you. So a lot of people say, well, this sounds great for Amazon.com, but I work in embedded systems, and we can't possibly do this. So I want to give you an example from the HP LaserJet Firmware team. Now in 2008, the HP LaserJet Firmware team had a problem. And the problem was that they couldn't deliver new printers fast enough. And they looked at what they were doing, and they found out that 10% of their time was spent doing code integration, 20% on detailed planning, 25% porting code between branches. They had a separate source control branch for every model of printer. And any time they added a feature or fixed a bug, they had to merge it between the branches to apply that new feature or bug fix to every other printer. 25% of their time on product support, 15% of their time on manual testing. And when you subtract that from 100%, what you're left with is 5%, which is the time left to spend on innovation or more likely lying down with several aspirin. So they had to change this. And what do they do to change this? They did a few things. Firstly, they had to re-architect the product so that they didn't have to have branches for every model. So what they did is they created a new architecture where it was all on trunk. And when the firmware started up, it would see what printer it was on and switch features on or off, depending on the printer model. So it was an implementation of feature toggles, basically, so that they only had one firmware version, and it switched itself, the individual parts on and off, depending on the printer model. And they started delivering that in one month increments. So they would build thin vertical slices of the application, so complete features that actually worked, but very small features. And they would showcase every month new features. And instead of having detailed planning for all the different teams, because it's a large team, more than 100 people distributed across the US and India, what they would do is they would set high level goals for the next month, and then the teams would have to work out how to accomplish those high level goals. And it was just one or two goals. And then on the technical side, what they did is this. Everyone trekked into trunk, and every time you trekked into trunk, it would continuously integrate. It would produce a build, and then it would run automated tests. And they had thousands of automated tests that ran in a virtual machine that simulated the printer, and on logic boards. So they had physical logic boards, and they would take the build and deploy it to the logic board and boot the logic board with that version of the firmware, and then run automated tests end to end against the logic boards. And if any of the tests failed, the changes between the last good test run and this test run were automatically reverted out of version control. So if you make a change to the system and the test break, there's no discussion. It's not, oh, don't worry. It's just this problem. I'll fix it. No. Your change is straight out of version control. So the system automatically rejects anything that takes the system out of a known good state. That solves a lot of problems, it turns out. So they would prototype stuff. So they would have prototype boards that would approximate what was going on, and they have a lot of stuff in PLAs or ASICs, in fact. So they use ASICs for a lot of stuff, and they have a prototype ASICs that they program using that. But often, the logic boards are kind of derivatives of existing ones rather than completely new designs from scratch. But if you look at what the Mac people did, the Mac Intosh people, when they were developing the Mac, they actually always had some circuit boards. And they would start up with something that they could test the software for. There's a great quote from Andy Hertzfeld, where he says, sorry, I'm just going to get this for you. So even on the Mac Intosh, instead of arguing about new software ideas, we actually tried them out by writing quick prototypes, keeping the ideas that work best and discarding others. We always had something running that represented our best thinking at the time. So they actually used PLAs on the early Macs, and they treated the design of the logic as if it was software by using PLAs and having most of the logic in PLAs. And if there was a problem with the logic, they would just make the change, reflash the PLA, plug it in, boot it up, and try again. And they were very iterative about the hardware design by making sure that most of the logic was actually in PLAs rather than in discrete units where it's hard to change. So you need to think about hardware design in the architecture of your hardware, how we can make that more flexible as well and be more iterative about that. And you can't fix all the problems. I mean, the lead time of actually delivering an ASIC is still very large. So you can't solve that problem, but you can do things about to create prototypes and test with those. Another great example is BASIC. So when Microsoft was developing the first BASIC for the 8088, there were no 8088s. And what they did is they built a virtual machine based on Intel's specification that simulated the 8088. And they built BASIC against that. And when the 8088 actually came out and they got a test board, it actually worked first time, which I think is superb, but also they lived in a less complex age than we do now. So with sufficient imagination, you don't need to wait for the actual logic boards. And this is why the HPP believes virtualization. By economical, do you mean having working software from early on, or do you mean making it cheap to get to the stage where you have a massive integration problem when nothing actually works, and then you have suspend agents fixing it? So it turns out that economics has to be measured over the entire cycle, not in terms of the costs of the current financial year's expenditure. So I'm not going to take any questions again, because I've got more to get through, but hopefully we'll have some time at the end. So I have to find the actual presentation that I'm giving. OK. So it took these guys three years to actually get to the stage where they'd implemented all this stuff. So that should be a warning. I mean, it's not something you can do in six months and bang, you're done. But it takes three years. And they got to the stage where actually they were spending a lot less time on this non-value add stuff. And they had a large increase in the amount of time they spent innovating. And the cost thing is important here, because it costs them money to actually build automated test suites and do this integration and re-architect the project. That was expensive, and it required investment. What's the return on investment? Overall, development costs reduced by 40%. Programs under development increased by 140%. Development costs per program down 78%. Resources driving innovation increased by five times. And they wrote it up into this book, A Practical Approach to Large-Scale Agile Development, which was published by Addison Wesley two months ago. So I highly recommend reading that book. But this is important. A lot of people think lean means reducing cost. Lean is not about reducing cost. Lean is about reducing waste in the delivery of value. And it will cost you money in order to do these things. But what you get is the return on investment. And so if you're driven by a CFO who only cares about cost, it's really hard to do this stuff. But what happens is, if you control costs very rigorously, you create very inefficient processes. Functional silos are efficient and cheap to run. But they're very inefficient in terms of overall cycle time. So you've done this fabulous agile thing. And now you're delivering these fabulous increments all the time into operations, small incremental deliveries. And what happens? I mean, operations is not necessarily very thrilled about this, about these new increments they keep getting. Because they've got to run these existing systems. And now they're getting all this code deployed. And they say, hold on with this. We can't keep our system stable against this rate of change. And the engineers say, well, what do you mean? Our test-driven code follows solid engineering principles. And operations people say, well, that's very nice. But it doesn't actually work. And the rational response of the operations team, the rational response, is to create a barrier. And that barrier is called the change management process. And the job of the change management process is to prevent change. So what do we do about this? This is where DevOps comes in. And DevOps is a movement which derives some kind of the 60s counter-culture movement that Neil Ford was talking about. And it basically says, why can't we just love one another? Why can't we be friends? We all do the same thing. Let's work together. And that really is mostly it. There's kind of cultural elements. There's some automation stuff. There's some stuff around measurement and sharing. But it's really the culture that's the most important thing. So what typically happens when you take a cultural movement and it gets exposed to an enterprise ecosystem, what happens is suddenly everyone wants to buy DevOps tools, and they want to hire DevOps. That's the actual net impact, because changing culture is too difficult. So suddenly, there's a big market in DevOps. And if you put DevOps on your profile in LinkedIn, you're going to get an awful lot of people like recruiters bothering you to try and hire you. And here's a blog post from a recruiting company entitled How to Hire a DevOps. I'm just going to read you one part of this. The dirty secret. Here's the confusing part. There's no such thing as a DevOps. Who knew? You'll have a damn hard time finding someone whose business card just says DevOps. And there's a grammar mistake, because it's a recruiter. What you're actually looking for is a developer with some experience and knowledge as a systems administrator, or possibly a systems administrator with some experience and knowledge as a programmer. When you post a job looking for DevOps, you might want your title and description to reflect this, as it will greatly increase your talent pool, blah, blah, blah. So there's no such thing as a DevOps, because what we're not trying to do is solve our silo problem by adding a new silo called DevOps. It turns out that won't work, because you just added an extra transaction cost and made the problem worse. So if you hire a bunch of DevOps and create a DevOps team, whose job is to manage the release management, you just miss the whole point. The point was to enable DevOps to collaborate more effectively, to make releases lower risk, so that we could have throughput and stability at the same time. When Flickr was acquired by Yahoo, Flickr was one of the early adopters of continuous deployment. Flickr said, well, we have a very normal change management process, and you can't do this 10 deploys a day thing. And the guy who was head of Opset, Flickr, a guy called John Osborne, looked at Yahoo's uptime stats. And what he found is that Flickr had a higher uptime than any of Yahoo's major services. They had four nines, I believe. Their downtime for the year was less than 10 minutes, which is better than any of Yahoo's services managed. So one of the things about continuous delivery is, if you do it right, you can increase throughput and stability, and certainly that's true of Amazon and these companies that are doing it at scale. And you don't do that by having a DevOps team and inserting it in the valley stream between development and operations. You do it by having developers and operations people talk to each other and have lunches together and have joint parties. I mean, if you're a developer and you do a release, if you have a nice project manager, what happens? Release party, that's right. And the developers all go and have a release party. And then about two o'clock in the morning, someone gets a phone call from someone in operations who says, I'm sorry, the system's down. One of you needs to come back in. And the developers say, look, I can't get a bit drunk, so I can do. And the operations people are really pissed off. And why are they pissed off? Is it because you're drunk? You didn't invite them to the party. Why didn't you invite them to the party? Simple things like that can really help fix the relationship between DevOps. The other thing that can fix the relationship is by developers actually carrying pages. I'll explain that in more detail later. So if you do the whole DevOps thing, hopefully the barrier between engineering and operations becomes somewhat porous, and these people talk to each other and collaborate, and operations don't find out about the release one week before it's supposed to go live. And then you can do some of this continuous delivery stuff, which means developers are writing production-ready codes. Everyone is collaborating throughout the life cycle of the product, crucially, faster feedback loops. So developers can find out quickly if the database change they want to make is going to cause 100 times degradation in performance, not on release day. And then what you get is lower risk, more reliable releases. And the crucial thing in terms, one of the crucial things in terms of project management, is that we redefined done. Done does not mean dev complete. Done does not mean I can showcase the story from my workstation and get my story points. Dev complete or done means the system will actually work in production. What I'm showing you on my laptop will actually work under realistic loads with realistic data sets and production-like data patterns, usage patterns. However, we still have a problem. Ops is still running all that legacy crap that you've been developing for the last 30 years. Who works in an organization which has production systems running on unsupported hardware? Several of you. Who has production systems running that nobody knows where the source code is to it? Right, several of you. Mission-critical systems. And the more mission-critical it is, the more likely it is that this is the case, because nobody wants to change those systems. And when you don't change it, you lose the skills on how you change it. And then when the vendor says, oh, we're upgrading, you don't have a year's worth of patches to apply. You have 10 years' worth of patches to apply, and that's a really high risk, and no one wants to do it. And so it's cheaper just to pretend that it's going to be fine until it isn't. There's no way to rationally decide which of those services to retire, because all you have is a big cost center, which is called operations. And we burn money sacrificially in front of it, and hopefully our systems carry on working. And there's no way to work out what the cost of any of these individual systems is over its lifecycle, or the revenue it's delivering. And thus there's no rational economic basis for making decisions on which of these services should be retired. Thus, deciding what to do is command and control. There's a great story from the Soviet Union days, it's probably apocryphal, that there was a famine in Moscow. And the head of bread production of Moscow went to visit London. And he went to London and set up some meetings. And he said, I want to speak to the head of bread production in London. You can see the problem. This is how our PMOs work. They don't have any economic data on which of the services delivering the most return on investment. So they set up a Soviet Central PMO meeting and decide the priorities for the next five-year plan based on who has the biggest paycheck and it's a willy-waving competition. Is it surprising that we have all this crap in production and we have all this enterprise dysfunction when we're using a Soviet Union model to actually decide what to do? And who's feeling the pain? It's not the PMO. So, yeah, customers, developers, you've just got to look for the people working overtime, basically. Who's working overtime in this room? Who's working more than 40 hours a week? Well, I mean, there's a whole sustainable pace thing, right? So, Werner Vogels, the CTO of Amazon, has a solution to this. I like putting words into people's mouths. Fortunately, they've said it. So he says, you build it, you run it. And this is how Amazon works. Amazon has, well, what happens is you have a bunch of services running in production. And what you want to do is treat them as if they were products. So we get rid of all these projects. Projects are a stupid idea. And we rearrange the organization cross-functionally around customer-facing teams. And the customer-facing thing is really important. And we use a service-oriented architecture so that each of these teams is working on individual service. So the way that Amazon works is if you take that front, that home page, the product catalog is built by a team. The shopping cart is built by a team. The recommendations engine is built by a team. And they're all customer-facing, but they all do a small thing and manage it for its entire life cycle. And there's a rule in Amazon called the two-pizza team, which is that no team can be bigger than you can feed with two pizzas. Now, this is Seattle Mine, so the pizzas are quite large. But still, it's 10 to 12 people. And those people manage the system throughout its life cycle. So operations doesn't go away. You still have operations management. And what you have is a team that builds the platform as a service that the systems run on. And this is a product. So again, if you look at Amazon, everyone knows about Amazon Web Services, right? AWS, the cloud-based self-service thing. So something that not a lot of people know about Amazon is that that was not developed for amazon.com. What happened was there was a bunch of South Africans who wanted to move back to South Africa from Seattle, being a guy called Chris Pinkham. And what's the name of the CEO of Amazon again? Bezos, Jeff Bezos. He says, OK, well, don't leave. We want to do this cloud thing. You guys go and build that in South Africa. And they built it as a customer-facing thing. And the head of ops at Amazon, a guy called Jesse Robbins, they asked him for hardware to build this thing on. And he broadly said, get lost. You're not having my hardware. Screw you guys. And they had to build it on their own. And the people in Amazon.com were not interested in helping them in any way. And in fact, it was developed as a customer-facing thing. And they got external customers long before Amazon.com moved to AWS. Amazon.com only moved to AWS about two years ago, finally. So it was a customer-facing thing. And this is a mistake that a lot of people make with service-oriented architectures is the services are delivered to other teams. And if you internally, and if you think like that, you can't make it work, the services have to be customer-facing. And you need to design them in a customer-facing way. If you Google for Steve Yege's platform rant, it's one of the most important bits of writing on service-oriented architecture and product development that you can possibly read, contrasting Google and Amazon. And this is one of the things they talk about. So then we can create a product development model where we have an inception. And following the lean startup, the job of the inception is not to create a set of requirements for a year's worth of work. The job of the inception is to come out with an idea, how will we change the world? What are we going to do to change the world? A way to measure that, what's the business metric that we care about? And then a minimum viable product, an experiment to gather data in order to validate that our idea is good. So that should be a few days' work for a small number of people. And then you build the experiment, which is the minimum viable product. And you gather data. And at this point, you have three options. You can either get the data that actually nobody is interested in this. And your whole idea is bogus, at which point you can just kill the product and just go away and do something else. Or you could say, well, the response hasn't been great, but we still believe in the idea, so we're going to pivot. And pivoting means taking a completely different business strategy to achieve the same ultimate goal. And so you've got to kind of start again from scratch. Or if you're really lucky, one in 100 maybe, you find out that your business strategy is actually a good one, and then you can carry on delivering increments. So you get feedback, and then once you've found a business strategy which is viable, then you can keep delivering small increments. And what we've done here is actually something very important, which is by keeping this small, the work of a few people over two to three months, we've managed the cost. The cost of employing a team of five to six people for two to three months in an enterprise is very, very small. And what this does is it changes the economics of product development. So has anyone read a book by Nassim Taleb called Anti-Fragility? There's a new book that's out. So he's got lots of other books that are very good. But anti-fragility, if you were at the last panel, we were talking about this. So things where like glasses, if you drop a glass and it breaks, it's fragile, right? Fragile objects respond badly to volatility. And people think the opposite of fragility is resilience. Resilient objects, if you apply stress or volatility to them, they don't care. But actually, the opposite of fragility is, well, it's given away by the title, anti-fragility. Anti-fragile objects benefit from stress and volatility. So an example of that is humans doing exercise. When you do exercise, you apply stress to your body. And you don't hopefully get hurt. But neither do you stay the same. You actually get stronger. If I apply stress to my body by working out, I get stronger. I benefit from that stress and volatility. And the same thing is true through many different domains. And one of the stories he tells is the story of an ancient Greek philosopher on an island in Greece 2,000 years ago. And this philosopher was always being bothered by his colleagues who used to play pranks on him the whole time when he decided he was going to show them all. So what he did is he went around all the olive farms on his island. And he said, I'll pay you a few drachmas if you will give me the option of using the olive presses at harvest time. And at harvest time, it turned out that it was a really good olive harvest. And so he was able to use all the olive presses and make a ton of money out of that. And he became very rich. And he never had to worry about money again. And nobody took his case and played pranks on him again after that. And so what he'd done is for a small outlay, he got this option to use the olive presses. If it had been a bad harvest, his only loss would have been the money he spent on the option for the olive presses. But as it turned out, it was a fabulous harvest and he made a ton of money. So if you think about financial portfolio management, when you're investing your own personal money, what's the most important strategy for investing your money in a portfolio? Sorry? Diversity. So that was the response I was looking for, diversification. You invest in a diverse range of portfolios. So if you look at what enterprises do, they start up a small number of projects which are going to go on for a year or two with a very high cost base. Would you invest your money in a small number of let alone companies, products within companies, that are very high cost and very risky? Would you invest your money in that? Of course not. It's stupid. But this is what enterprises do all the time. It would be better to take the money that's spent on new product development and put it in unit trusts. That would provide a safer rate of return than investing in software products for enterprises. Instead, what we want to do is have a large number of software projects, except that many of them will fail, but some of them will have a large and unbounded return on investment. How do we manage the cost? How do we get that optionality so that we know that we're not going to spend too much money on something that may turn out to be wrong? Well, I already gave you the answer. By managing the cost of the initial investment and saying that if you only invest money for a few months for a small team, we've managed the downside. We've created optionality and managed our portfolio of products. So therefore what we can do is we can measure the cost and value delivered by our products over their life cycle. Because remember, we're doing you build it, you run it, one piece of teams, building services or products, and trying many of them out. So we can measure the cost and value delivered, which allows us to make rational economic decisions on what we're going to spend our money on. Those teams can self-manage using the Lean Startup methodology. And you still need high level groups. It turns out that if you just apply free market principles, markets tends occasionally to fail catastrophically. And this is true of any kind of self-regulating mechanism. So you still need a regulatory thing. The architecture group regulates to make sure that cross-functional requirements of the organization are met by the individual design decisions of the teams. And you still need a PMO whose job is to regulate and make sure that the way that the teams are being measured creates the right behavior that aligns with the strategic goals of the organization. So we've got 10 minutes left. Oh, I guess there's a couple more things I want to talk about. A big problem is around skills acquisition. This changes the way the organizations work, and it changes the skills that are required. So if you look at these teams, I love that animation, these teams need different skills. So a lot of people say that this model gets rid of operations people. That's not true. The operations people are still there. They're just working on the individual teams instead. So you still need similar skills, but people will change their roles. You need what's called the T-shape model. So the guy's head of IDO talks about the T-shape model. People who are broads in a wide range of things and deep in a few particular skills. And so actually enabling people to acquire the skills for this particular model of working is hard. And there's a great book that came out called Toyota Cata. And for those of you who didn't see the panel, the story is that Mike Rother has been researching Toyota for a number of years. And what he found is that people were adopting the tools like Kanban and Andon Cores and all this kind of thing, but they weren't seeing the business results they were expecting. And so he went to ask the leaders in Toyota, how is it that you create these teams that are so great at building fabulous products? And they said, well, I don't know. And so he's like, well, do you get training or anything? So this was weird and kind of fascinating for him. So what he did is he went and he studied how leaders within Toyota learn to be leaders. And one of the critical things is that the only way you can become a manager in Toyota is by first working on the shop floor. And then once you've actually done real work, then you can become a manager. And the way that they work in Toyota is they don't do big product specifications and then do a big bang delivery. As you might guess, they know they need to achieve this result. They don't know how they're going to do it because we're innovating. So they just run a lot of experiments to try things out and do a lot of prototyping in order to find out what's actually going to satisfy their customers and obviously get real feedback from their customers. And so productivity in Toyota means running as many experiments as you possibly can. We all have all these crappy measures of productivity in IT. Does anyone measure lines of code? So lines of code are reliability, not an asset. In fact, a colleague of mine once went to a company where the auditors wanted to put a hook into version control so they could measure the number of lines of code added and put them on the balance sheet. So if you started deleting lines of code from the code base, you were destroying company assets. And that turns out to be a problem because the most virtuous thing that anyone can do as a developer is to delete code. That's the most noble goal of any developer because then what you do is you remove complexity and make the systems easier to change and to understand. How about who's measured on the number of hours they work? Who has to fill in a time sheet? So when things are going badly, we're encouraged to work more than 40 hours. Well, a century of industrial studies proves that when you work more than 40 hours, you become stupid. And what happens when stupid people work on software? They create bugs. And then when you have lots of bugs, you have to spend lots of time fixing the bugs, which means you need to work more overtime, which makes you more stupid so that you create more bugs that you then need to fix, which slows you down even more. So you can see that having developers work more than 40 hours or anyone is a great way to slow down the rate at which you're able to create well-factored software. So what should we measure? What we should measure is the number of experiments we can run and the amount of data we can gather. And so the job of managers in Toyota is to enable the teams to learn as fast as possible from each other by acquiring skills and from their customers. So you need to optimize your development process and your product development process for experimentation and build organizations that learn fast from each other and from their customers and try and measure the value delivered. In terms of changing your organization, there's two basic approaches. There's the top-down approach where the manager says, fetch me 100 of your finest certified Scrum Masters. And we will change the organization. Or there's the bottom-up approach, which is, no, we will deliver crappy software late-through self-organization. And it turns out that neither of these approaches will work on its own. You have to have executive support. And you also have to have people on the ground being excited about what they're doing. If you're trying to change an organization, like anything else, don't do it in a big bang way. Do it team by team. And start with a team that, A, is excited about doing it because you have a team that doesn't want to do agile and you try and force them to become agile. That doesn't work so well, it turns out. And also a team that has some level of maturity. So if anyone heard of Bazavod's Scrum Test. So he had a test that he went around to go and find out the people who are really doing Scrum. And the reason he was doing that was to find out teams that were reasonably good at what they were doing and start with them. And by starting with the teams who are quite good and excited about doing it, you'll learn stuff that then you can apply to the other teams in your organization team by team. So the final things I want to say is, if you want to go through this journey of transforming into a kind of DevOps organization, you need to first understand why you want to change. What's the business goal? You've got a business metric. What's the impact on that? If you don't understand why you want to change and have a measurable benefit, it's going to be really hard to convince people to do all this stuff. Try and find a way to get that measurable change fast, even if it's going to take you three or more years to reach your final goal. So HP, they were delivering monthly increments. Demonstrate some value in a few months or more quickly than that, and keep doing that. A good way to start is continuous integration. Ops is not going to take you seriously until you can prove that you can actually create new versions of the software that aren't going to break things into production. So start with proving that your changes are safe. By starting with continuous integration and then building a deployment pipeline. Don't go out and buy a bunch of tools as the first thing you do. Focus on organizational issues, culture, and systems architecture. And systems architecture is often one of the biggest barriers to being able to achieve this stuff. And then the way you fix these problems is finding ways to incrementally improve them. My final rule is actually not from me. It's from a guy called Jesse Robbins. Jesse Robbins was the guy who was head of Ops at Amazon, and he's also a firefighter. And Jesse's rule is this. Don't fight stupid. Make more awesome. You're never going to be able to convince people that what they're doing is wrong by talking to them. The only way you're going to convince people that there's a better way is by showing them and doing it. So a lot of organizations create skunk works and hide them away. And that's a problem, because no one can see it. The best way to try and experiment with your process is get a team who's interested and capable. Put them out in the middle of the organization, and make it really clear. We're just running some experiments. We're trying some things out. A lot of the experiments will disprove the hypotheses, but that's OK, because we're learning. And they'll try things, and a bunch of them will fail. But then a bunch of them will succeed. And then people will see, oh, we did something cool here, and then they'll want to do it. And when someone says, I like this. Can I do it? You say, yes. Of course you can do it. We would love you to do it. Here's the resources to help you try and do that. And that's how you spread awesome throughout your organization. Don't fight stupid. Make more awesome. And you do that through the process of experimentation. So a few minutes left, what questions do you have? No, you've got it on your local workstation. It's just not in trunk, at the back. So a big problem is re-architecting existing legacy systems. How do you do that in an incremental way? So we had an organization who actually wanted to do this. And they had two divisions. They had a main division, which is building the web front end that was very agile. And they had another division which had a mainframe booking system. It was a travel website. And the mainframe booking system, obviously you can't. They were not agile. And so every time they wanted to add new features, they were constrained by this. And the thing that they did that actually had the biggest impact is they would talk to each other. And it turned out that no one was talking to the other team to find out what their next priorities were over the next six months. So they had the mainframe booking system. They had these things they were going to do. And the web people wanted to do these other things. And by actually talking to each other and saying, well, here's my list of stuff. Here's yours. What are the dependencies? How can we actually change stuff around in a way that makes us aligned? That had a really enormous impact on their ability to do that. So that's step one. And then step two is a pattern that Martin Fowler has written about called the strangler application. And the idea is this. You know you have these trees in India that creepers grow up around. And eventually, more and more creepers grow up around the tree. And eventually, the tree dies. And only the creepers are left, right? So you take a legacy application. And instead of starting up the big bang legacy replacement project, where you start a two-year project to replicate all the requirements of the existing one, but in a completely new technology. And then those never work. They either go on much longer than they're supposed to until they get canceled. Or they deliver something. And it doesn't have feature parity with the additional products. And it's more horrible, but in a new technology. So instead, find incremental ways to do that. And the idea is when you have a new requirement, build a new service that meets that requirement in your new technology using the continuous delivery processes. And then that may need to talk to the old system for some of its stuff. But incrementally, over time, you can create more and more of these things. And eventually, you strangle the legacy system. And this is how Amazon implemented their SOA. Amazon used to be a big ball of mud called Abidos, and if you went to amazon.com in 2000 or so, you would see slash Abidos. And they created all these two pizza teams. And they strangled the application. And in fact, Jeff Bezos sent a memo round to everyone in Amazon saying, we're going to move to a service-oriented architecture. And he hired an army ranger, whose job it was, to go around all the teams. And if your system talked to another team system by looking at the database instead of the service API, you've got a warning. And then if you didn't fix it, you got fired. So Jeff Bezos basically sent this memo saying your number one priority is to move to a service-oriented architecture. And he enforced that with an army ranger, which is kind of a nuts thing for a CEO of a company to do. But it turns out that Jeff Bezos is very smart, because he was creating a way to strangle the existing system with all these new applications. So that's kind of the high-level picture. There's a lot of complexity and pain. It's hard to do that. And it seems easier to do a big-bang replacement. But in my experience, the big-bang replacements always end up taking much longer and often get canceled. And so my question is, would you rather do something that takes a longer amount of time that is going to be harder, but actually isn't going to get canceled, as opposed to something which is shorter and easier, but won't actually work? So that's it. Thanks very much for your time. I'll be around all day.