 little bit, but basically the point of today's talk is, like it says, I don't believe you should call it continuous delivery. If you don't have a system you can go to in your organization and click a button right now and deploy to production. It's not CD. It might be really good automated building test and what have you, but you're not quite in continuous delivery yet. But there's a trick to that. When I say that I mean safely. Having done all the right things and you know etc and not ended up in jail or broke or those kinds of things. So that's really the point of this talk. The talk comes from a personal thing of mine where I'm probably a little bit too binary on some things. So like I'm not anymore, but I used to be a manager and my pet peeve was, hey, is this thing done? Well, yeah, but I just need to or yes, except you know etc. knows a perfectly valid and okay answer, but done is done. And so that's when I when I say it's not CD if you can't do it right now. So who am I? I have a very large dog. He really does sit there. My name is Ken McGrage. I've been with ThoughtWorks for about 10 years. I'm currently a technology evangelist for some reason they pay me to do this. I go around and talk about continuous delivery and DevOps and so forth. I'm one of the global organizers for a conference series called DevOps Days, which before I got involved with it is actually where the term DevOps came from. There's a DevOps Days here Thursday and Friday, which is one of the reasons I'm in town. I don't know if there's tickets available or not, but anyway. I never update my blog, but there it is. You can feel free to check it out. Happy to go on Twitter and still forth. One of the things I like to do before any talk is give some definitions of words and I do this not to say this is what the word means. This is what you have to adopt. Go forth. I do this so that for this talk or when you're talking to me, you know what I mean when I say these. So it's also I think important inside your organizations that you agree on what things mean. If you say I'm doing DevOps or I'm doing continuous delivery or if I'm running a unit test that you all agree in your team on what that means. So the first one is DevOps itself. I wrote my own definition so that you know what I mean. So when I say DevOps, I mean that it's a culture of people working together. So I'll get into this a little bit, but no such thing as a DevOps tool. You can't actually do DevOps. It's a way of working and so forth. There's more about that. A little test that I actually end up doing in my head is a word substitution game. Whenever I hear somebody do it, I substitute culture. So I'm not a big fan of the title DevOps engineered because that's a culture engineering. That's not a thing really. So when I say DevOps, that's what I mean. I mean the culture behind it, not the technology behind it. Continuous delivery on the other hand. This one I'm not going to be so presumptuous to try to define because they actually did it. Jezz Humble who co-wrote a book called Continuous Delivery with David Farley. They both worked for ThoughtWorks at the time, although now they're famous, so neither one does. But that's his definition of continuous delivery. Now, the important thing to note here, and I have a new toy so I'm going to play with it, is he said all types safely and quickly. So it's not just your application. It's everything about the changes, the configuration, the database, everything about it. And safely. So no Wild West and so forth. Okay. So first off, and this is probably going to be very much review for a lot of folks here, but why do we do continuous delivery at all? So when we first started talking about continuous delivery in ThoughtWorks, gosh, 10 years ago, we saw it as a completion of the agile promise for lack of a better way to put it. If you're familiar with the agile manifesto, if there's actually a second page to it. And it's called the principles behind the agile manifesto. And the first principle is the highest priority to satisfy the customer through early and continuous delivery of valuable software. And so from agile 17, 18 years ago, we said we wanted to deliver software to our customers faster and so forth, but we actually didn't. We most of the time delivered it to customer advocates or product owners or what have you, but not to customers. Why do that? Anybody take agile, like fundamentals courses 10 or 15 years ago, you might have seen this. You know, it turns out partially done might be useful. So we want to be deploying all the time so we can get feedback from customers and know, are we going down the right path? You know, they don't paint the Mona Lisa on the way top that painted the way at the bottom. It's actually there's a couple websites out there if you want to have some fun, go look at x rays of famous paintings and how much they changed over time because they canvas was very expensive. So they didn't scrape it off. So you can go and they did x rays. You can see how much the artist would change their mind. Security threats, these aren't going anywhere. We need to be able to react quickly. You know, when something comes out next week or next month and not not if when we need to be able to react quickly. When Heartbleed came out, it's been a couple years ago now. There was a huge number of web servers affected by this open SSL bug. And many of them were exploitable for weeks. Some of them might still be I wouldn't be surprised if they are. Anyone know the story of Night Capital? So a couple. Okay, good. I'll finish it at the end because it's his favorite story. Continuous delivery is how do I put it? It's not where you start. It's, you know, more closer to an end state, although you never really done. And so one of the things I really want to encourage people to do is consider continuous integration, a prerequisite. Okay, you really do have to have all the tests. So things like, you know, XP processes like test driven development and good unit tests and good coverage and all that kind of stuff. You can't deploy to production safely and give it to your customers if you're not running all the tests. So if you're not doing CI yet, and we'll get into what I mean by that a little bit, don't worry about deploying production yet, right? You got a different problem to fix first. Or I don't even want to say problem, different work to do first. We do a publication called the tech radar, I think there's some over on the table over there, where we talk about a bunch of different things and what we recommend people, you know, stop doing or do or what have you. One of the things that came out in the last year or so is a concept called CI theater, where people would say, Yep, I'm doing CI. We're like, Okay, do you have this? No, you have that? No. It's like, Okay, you're pretending to do CI. You're not really doing CI. And so you're really not safe. A particularly disturbing study that we did inside my division of ThoughtWorks is we sent out a survey, and we said, Are you doing continuous delivery? Are you teaching integration? Excuse me? Yes, are you doing this survey monkey stuff follow up and so forth. Only 10% of the people differentiated having a product from doing the thing. They're like, Yep, we're doing CI we installed Jenkins doesn't quite work that way. You know, I mean, there's a lot of Ferraris went around town doesn't make them race car drivers. Okay, you actually have to do the practices. So what do we say are litmus test if you will for continuous integration. This one was written by Martin Fowler and jazz humble to people who are way smarter than me. So I'll believe them. First one is this is probably the most controversial. Everybody commits to a shared mainline every single day. So call it trunk based development or master or what have you. It is not long live branches. It's not feature branches. Not saying you're evil. But I am going to say that if you're doing feature branches, you are not doing continuous integration full stop. Okay. The next one down is every commit triggers automated building tests. So it's not nightly builds anymore. I'm old. I remember when our build took eight hours. So it's every commit you want fast, fast, feedback. Yeah. Sure. Daily. Oh, absolutely, especially in today's world, the distributed version control systems. Like when I do work on get the first thing I do is get checkout dash be something. Yeah, no, it's it's it's mostly daily. That's that's I think what Jason day and Martin said was daily, but it's usually more often than that even. But I've been at customers where there's I was at one that had 26 feature branches, the youngest one was three weeks old. Yeah, let's not do that. And then if building test fails, just says 10 minutes, that might be a little aggressive. But it gets fixed. You don't leave it red. It's okay going red. I'll get into that towards the end. Okay, but you don't leave it red. I'd rather have no test than a flaky test. Okay, you really need to be able to trust your pipeline. That's first. This is the continuous integrations first get this done. Okay, so we just said no longer feature branches. But I also said, if you not can't deploy right now, you're not doing continuous delivery. Okay, well, how do we justify that because features take longer than a day. So we need ways to deploy in complete work. There's lots of different ways. I'm just going to go over a couple that that are been very successful for a lot of people, including us. First one is concept of feature toggles who's using feature toggles or feature something like that. Okay, should be more. But basically, the idea is, you're going to work on a new feature. The first thing you do is create a configuration that turns that feature off. Then you start implementing the feature. And so that feature could be half done. And if I deployed the software, it would be off it wouldn't that code would not be executed. Obviously, it's more complicated than that. But the idea is just this, I'm going to do a pet survey. And if it's false, then it doesn't show up. If it's true, then it does. That allows you to deploy and even release and I'll get into the difference between those, even if stuff's not actually complete. Now the blog on the bottom down there and we'll send out these slides later. Martin actually wrote in 2010. So this is not a new concept. But it was followed up last year, two years ago by another colleague, Pete Hodgson, to go even further into some of the ways Thought Works does feature toggles. Toggle routers where we can use geography, that kind of information, database stuff, percentage of deployments, etc. These are really, really powerful. There's even now SaaS companies that can do feature toggles as a service. So they put a little bit of code on your page and you configure their thing to set it where not things should show up. I haven't used it in personally, so I'm not going to mention any names. But toggles are a very big deal in today's world where we want to deploy a lot. What's really cool about these two is that they're useful for other things. So just some examples again from Pete's article. When we're talking about deploying, we're talking about release toggles, when we're talking about incomplete features. So those hopefully don't live very long. They're very dynamic. By the way, this is tech debt. If you're adding toggles in there, once the feature is done and accepted, you need to go back and clean it up. Okay, so the feature takes longer to do it this way. It's safer, but it is a tech debt. But notice some other things in there, experiments. So if I do it this way, do my reservations go up or down? So like AB testing, permission toggles, we're having a really bad performance outage or something. Change it so only admins can log in. Things like that. Operations, same kind of thing and so forth. Is Netflix big here? Streaming, okay. So Netflix is a big streaming movie thing. They actually have a bunch of toggles in there that are operations toggles for big features. So I'm streaming a movie. They have a major problem in the outage. The recommendation engine might shut down. They just shut down entire parts of the thing but not the streaming of the movie. And not the ability to sign up because that's where their money comes from, you know, and so forth. But they'll actually shut down parts that aren't important as important. Big shopping holidays. Some e-commerce people do that. They close down all the recommendations so the page loads faster. So you can do all those kinds of things. So very, very powerful thing. If you could only change one thing in the average place to get to continuous delivery, it ain't your tools. It ain't your technology. It's not your tech stack. It's your org structure. Okay. So I want to talk a little bit about what I mean by that to enable this kind of thing. So first off, this is a chart that shows up a lot if you look up continuous delivery. If you look up continuous delivery on Wikipedia, this comes up. And it's credited to Jezz Humble and it was actually created for a sales deck that we did together. It thought it was about nine years ago. And basically the idea here is showing you the feedback cycles of delivery team makes a commit, something fails, they fix it right away, it goes a little bit further. The idea is things to get to the right. Spoiler alert because I'm going to get to this a little bit later, but the number one regret that David and Jezz have from the continuous delivery book is that pipelines were always shown in a linear fashion. In fact, there's going to be a lot of things we need to do in parallel and so forth. So it's not really quite this simple but it's basically the idea is the feedback loop. So if we want that feedback loop, then let's look at the average organization. Now this is going to get very hand wavy. So there's going to be team names and those kinds of things that may not exactly match but I think that you'll get the point if not, tell me. But a lot of organizations, you have your development teams and they're the ones that write the code and they have a code complete and so forth. And then this is the old Agile chart almost. Throw it over the wall to, I just call it a testing team here, but call it anything you want, compliance, security, what have you. This is the, yeah, I'm doing continuous delivery. My pipeline gives it to security and then they do something. I don't know what. And then from them, somebody has to run it. The operations team. These poor folks have no idea what happened. Does anybody identify as an operations person? Is your job to run the stuff somebody else wrote? Sorry. That's my background too. Pager going off at 3 a.m. to break something that I never even knew got deployed. Okay, that's not exactly fair. There's a thing out here called Conway's Law. It was written 51 years ago. And I know that because it was the month I was born. Anyway. But it basically says that your architecture is going to end up mirroring your org structure. So if you have in your org, this group does payments and this group does, you know, shipping and what have you, you're going to end up with an architecture that roughly mirrors that. And when he wrote it, he was talking about nuclear plants and dams, but the same actually works out through the software. The same actually works out for continuous delivery pipelines too. If you have groups that are doing the different things, your pipelines are going to end up mirroring that. You're not really going to get your development teams the ability to deploy quickly to production. So we go back to that model. So I'm going to talk about some things that don't work for fixing this. And if you identify with one of these, please feel free to as their fruit they can throw at me or I don't know. But there's patterns that get into why this. Things that don't work. Renaming your operations team, a DevOps team actually doesn't do anything. Okay, you know, you know, learn a little bit of Chef or Puppet or Ansible or what have you. But it's the communication barriers are there. The people doing the automation still don't have any idea why that features there and so forth. Not quite as bad, but creating another silo that does the automation for everybody else also does not typically work. So you know, we put a DevOps team down here and then they're going to create the pipeline that everyone else is going to use. But they don't really know the tech stack all that well. They don't really know what the support matrix is going to look like and they don't, you know, etc. etc. There are ways with platforms and so forth. But this also is generally speaking an anti pattern. Jez Humble will go on a screaming rant if you talk about that. He's like building another silo is not a solution for silos. So what we'd rather see is this concept. You might have heard of products over projects. So product teams. So teams that have cross functional capabilities. I don't mean everybody on there has to be a Solaris admin or you know whatever. Although I tell you if there was a Solaris admin on Jez's team in 2006 the continuous delivery book would not exist because it came out of a horrific failure where they didn't know that NFS was going to do a good thing and it did. So we want these kinds of teams. Everybody heard this? You build it, you run it. This is Werner Vogels, the CTO of Amazon in 2006. He was doing an interview and he said our rule is you build it, you run it. If you're going to build the service that processes the credit cards, you're going to run the service that processes the credit cards and you're going to be the one that gets paged and so forth. People often think that they did this to pay less operations people or because they're using AWS which by the way amazon.com didn't do. People think they started selling the thing that Amazon runs on. Nope, didn't. What's funny though is if you take it the whole context, I don't expect you to read the whole thing, what's funny though is the why. If you look down here what it really was was to bring developers into the day-to-day operation of their software to bring them closer to the customers so that if they changed a feature they would know if they sold more or less shoes and if they sold less they would take the feature back out and they could see that and they could see the performance things and they could see that okay it now takes a hundred milliseconds longer to process an order which people will leave over in the aggregate. Okay, so it wasn't one team because we want to make their organization smaller, it was one team because we want fast feedback and we want to know if what we're doing is successful. So how do I do that? We often hear well that's great if you're doing you know if you're doing the Kubernetes but that's not going to work for me and it's possible. One of my favorite stories about this is Hewlett Packard though. So Hewlett Packard for the firmware on their printers had a problem where well over half of their time was spent just basically setting up test stuff not on new innovation because they want to make a change but there's like 19 different drivers on 19 different printers or you know whatever it was and they couldn't do these so they end up actually having to re-architect the way the drivers work and now and of course bandwidth and disk storage and everything else is a lot cheaper there's one piece of driver software and it just works on a bunch of different ones and so they did containers delivery not by doing more tests or what have you they actually did have to re-architect their software and in their case they're now doing 80% innovation time so that was the financial payback don't do it just to do it don't change your architecture just to change your architecture but there might be something so we had a client in North America travel agency that had a really bad problem so they had a monster monolith and the first time I talked to them they said yeah we deploy like once once a week well that's not all that bad I mean this was 2009 that's not that bad I mean he said yeah but it was the build from six months ago that's not as good then they literally had a six-month hardening cycle okay and so that's not good but so their problem was that the services that they offer the things that they sell the demand for those doesn't scale evenly and so an example is well here in Singapore lots of hotels okay and you can hire a car but that doesn't seem to be that popular but if I want to up my capacity to book hotels I have to up my capacity to rent cars because it's all one process in a typical monolith okay and so their problem was whenever they wanted to scale they had to scale everything not just the thing they wanted to and so this was a significant cost again don't change things because they're shiny this is a significant cost so they had to put this out on multiple servers when we say microservices we don't mean the buzzword we actually mean something very specific by it so a microservice is an independently deployable piece of software so if you say these two things have to be deployed together not a microservice if you say these two things share a database not a microservice okay they're independently deployable the advantages to that for them is what they got to do is take out that piece that was costing them the most money the piece they couldn't scale only that piece they didn't decompose the whole thing created a service for that went to the original monolith pointed it at the service scale the service up scale the monolith down saved tens of thousands of dollars in hosting costs alone just because they're less infrastructure to run the monolith because they can now scale it separately so that allows them to go now and distribute these across servers the way they need to not the way that they're being told to do it so lots of stuff out there are decomposing monoliths actually I didn't put a reference in here but on martinfeller.com a colleague from Central America just wrote a really good thing about decomposing monoliths this might be necessary to achieve continuous delivery in your environment it might not but it might be so what does that do for us we go back to our product teams and now they can own a part of the business so this team is responsible for rental cars this one's responsible for hotels this one's responsible for airlines you know etc so they can own it there's a model out there I didn't do any slides on this people might have heard SRE Google's been talking about system reliability engineers and so forth think about more like traditional ops is the way I think about it well a lot of times there's a lot of larger organizations that these teams can earn the right to have somebody else run their service okay but they have to earn the right so if I'm doing the rental cars and I deploy my service and it's a certain level of stable and it does a certain level of whatever I want to measure for a certain amount of time then the SRE team will take it over and now they'll be responsible for there's a failure in the data center and they have to move things over or whatever they don't just get to hand it over but most likely they own it from idea John Willis says idea to Cha-ching so I want to talk about platforms our services for a second real quick because Segway clumsy segway but segway nonetheless if you think about service offerings and what you can do to provide infrastructure for people I like to use this analogy of car as a service although I keep forgetting to change that on premises because on premise isn't a thing but anyway if I buy a car I'm responsible for everything about that car I have to finance it I have to change the tires I have to put the fuel in it everything about the car so think about that that is on premises lease a car some of it gets taken over hire a car you know etc I'm a big fan when it comes to continuous delivery stuff is platform as a service where and I'm not necessarily saying pay Amazon I'll get to that a little bit but where we can make a platform available to these teams because again we're not really going to have Solaris admins and DBAs and that kind of stuff on every team so why am I a fan of that an example from the from the US you probably could tell by my accent there's an organization called cloud.gov if you're a U.S. government organization and as I understand that the regulations for Singapore are relatively similar if you're a government organization and you want to run something on public cloud there's a whole bunch of things you got to make sure you're doing tests you have to do compliance etc so cloud.gov was stood up it's an official service of the U.S. government so government agencies below a certain risk factor can call up cloud.gov and they'll host their applications and what they do is there's 325 required security controls total cloud.gov takes care of 269 of them so some of these are around DNS and operating system patching and back up in recovery and those kinds of things and so cloud.gov says we'll do that we'll take care of hard drives catching on fire because that happens 41 of them are shared they're you know it's the application responsible for part of it the platform's responsible for part of it and 15 are handled by the customer so if you're the one creating that service if you're the one hiring out cars for the government you just went from having to do 325 things to doing what 15 and sharing 41 and so we can do these things for our teams to give them the ability to do all these deployments because spoiler alert which I'll get to in a minute we really not going to have all the skills on every team I fly a lot a couple hundred thousand miles a year been up in the clouds and hadn't seen a computer out there once okay so there are real computers they have power cords and hard drives and ram and all of those things and you can hire this out but you don't have to so when I say platform as a service again I'm not saying okay you got to go hire amazon got to go ahead by the way ThoughtWorks doesn't sell any platform stuff so it's not a pitch when I say these things I'm not saying you got to go do public cloud necessarily you're going to do these things internally got a smile for the camera anyhow so if we go back to our product teams it would be great if we had truly cross-functional teams I work for ThoughtWorks products and our teams are so Ganesh back there is a developer on GoCD everything about GoCD from marketing to support is on that team and that's awesome that's not not all organizations can do that so one of the things you can do is something like this so you might have people inside your organization that it's their responsibility to run cloud fusion open stack Kubernetes Google cloud Amazon whatever so there's people inside the organization who do operation stuff as a living that it still can run the platform you are not going to have people with compliance expertise on every team but it's kind of an important thing you're not going to have security people on every team I wish we did and security should be front of your mind but there might be these other teams and so now how does this play so what I want to do now is take a look these are our teams this is our continuous delivery pipeline what's their role so in a CD pipeline a lot of times what happens and this is the dreaded linear pipeline is the development team has their pipeline as unit tests and functional tests is often in production and this rightfully scares the heck out of some people okay there's nothing stopping me from creating an application putting a ThoughtWorks logo on it deploying it to AWS and people thinking it's a ThoughtWorks app that may or may not be in compliance with our privacy policies that may or may not be secure you know etc etc and so I mean I work for a team a company that makes a continuous delivery tool this scares me so some examples of why things that are bad deploying insecure software we can all pretty much agree that's bad right non-performance software there's been study after study of your you know X percentage slower than your competitor you're going to leave I am horrible if your site's slow and I have an option I'm gone okay so we want to make sure we're testing these kinds of things all the time non-compliant software do y'all deal with GDPR here much it's a Europe thing but I see like all the cookie notices if we like collect the wrong information about somebody and don't have it in a way where we can easily delete it we could be subject to millions of dollars in fines and so we need to make sure our stuff's in compliance we need to make sure we're not using old versus open SSL or what have you ineffective software nobody tests this we deploy the feature that our sales go up or down okay we don't test this we should I hinted at this earlier but a continuous delivery pipeline has one purpose and that's to kill a release candidate every time you make a change source code configuration whatever you have created a release candidate it's the continuous delivery server's job to kill it it's a continuous delivery job server's job to prove it's not good enough to give to your customers okay you can't prove something's good you can't you can't prove something secure because there's things out there that just haven't been exploited yet or what have you you can prove it's bad and this is why you have to have all the tests and you actually have to run all the tests so if we go back to our organization what I would recommend you do in your pipelines is you have all the other things in there too and there's tricks around this okay so you know we have our security team and our compliance team here this is the concept we call Fannie and fan out but I'm going to give the development team fast access to production I'm not going to slow down their pipeline by sticking these things in the middle because I don't want to wait for a staging deployment until a watch security tests are done okay because I want to deploy the staging as often as possible because that's testing my deployment I don't want to make a product owner wait to do user acceptance testing because some other kind of test isn't done so this is why we do a lot of things in parallel now what you can do though is notice at the end the arrows up to deploy production those are not just theoretical those should be enforced so what you can do is you know the functional test and deploy staging might run 47 times for every one time compliance test run and that's okay it ain't going to production if that bill didn't pass all of them so you can do these things modern tools can all do this I mean I recommend go CD because I helped create it but you know use yours I mean there you can do these and you should and these are just examples those are compliance and security but I mentioned performance and those kinds of things and when I say it's not CD if you can't deploy right now this is what I mean I mean safely having run all of the tests so it's an eye chart I don't expect you to be able to read it but that's a picture of a real pipeline okay so it's not linear you make a code change and it builds Linux and Windows in parallel Linux gives us Mac and Solaris so I don't need to do all those if that all passes then we do the plugins if that passes then we create an installers if that passes etc okay but I can do these things in parallel I don't have to wait for the Linux stuff to finish to do the Windows stuff but also if anybody cares I carry this because I have a bad back I'm not a Windows lover but this one takes twice as long as this one surprise okay so the Linux pipeline might run twice as fast times for every once the Windows pipeline runs okay but that's okay if these parallel jobs were all in one pipeline the whole thing wouldn't finish and tell Windows stuff so the Linux build ages would be sitting there idle half the time I want fast feedback so if that means that they run out of sync that's okay because we're going to enforce that they all have to pass before we go here that's the idea behind parallels that every stage can give you the feedback in a very fast manner so some deployment patterns first off another thing out of the radar separate in your mind the difference from between deploy and release okay deploy is put the software on the server release is make it available to the customer so if my deploy process takes an hour and a half because I have 4,000 servers globally I can deploy with the feature toggles turned off for example and when I'm ready to release click a button and it turns on okay so you can deploy constantly but you don't release you know again it goes through these on-premises software if we released as often as we build you would be doing apt-get update three times a day and you'd get really really angry okay so we release it once a month but we deploy it quite a bit more often than that just some other patterns so people familiar with canary releasing this helps me know how much time to spend on these things okay so there's a thing that's actually true the canary in a coal mine in the U.S. when they're mining coal years ago they would take a bird into the coal mine with them and if the bird died they would leave because birds have very small lungs and there was gases in the air that were going to kill them eventually and so what a canary release is is basically the same idea I'm going to toggle on that feature for a subset of my employees in this case it's facebook so when they do a new feature the toggle goes on where that feature becomes live only for facebook employees and so they can now test in production they can see is it still performant at production scale is it doing what we wanted it to do do people hate it you know etc if that works then it goes to a larger set and a larger set and a larger set lots of different tools out there to do this containers make this a ton easier but canary releasing is a great way to find out am I going to sell more shoes with my new feature or not so I put it out there if I sell less shoes I turn it off okay we don't want to we don't want our business to go down I love this one dark launching so again facebook years ago facebook released the feature that allowed y'all to have a unique username so you could go get you know the facebook.com slash whatever your name is okay for a long time they didn't have that they had like these random strings or what have you so they're going to release this new feature it was well publicized lots of people wanted it but there's a problem they're unique and they're unique globally and so I can't release this in a canary I can't release this only to the U.S. and let them grab up all their names and then people in Singapore can't get theirs okay that's not fair and so I have to release this feature to everybody at once and they already had tens of millions of customers how do you test that? they did a thing called dark launching what they did is they put javascript in the code where when you logged into facebook it made a call and tried to reserve a username for you randomly generated but it got to it did that and so they got to see okay what did that do to our system and they did it you know six p.m. or what have you in pacific time where they are but globally all of a sudden everybody you know canary certain number, certain number, certain number tested it, tested it and tested it over and over and over had to fix a whole bunch of things when they went live this chart I recreated it you can see the real one on the I recreated it because it was tiny image this chart is the memcache so and all over the world everybody got to reserve their username at the same time and so if you have a relatively unique name like mine it wasn't a big deal but if you're you know Tom Smith in the U.S. there's probably no exaggeration probably 5,000 of you and if you want T. Smith you had to race and so everybody did people were sitting by their keyboards so you can do this if there's a major shopping holiday coming up a major anything else what have you you can do these things and test these features in production because you can't set up a testing environment at this scale you just can't okay night capital night capital was past tense well they're kind of still around it was a trading firm so what they did is they traded stock at a program called the retail liquidity and they were manually deploying this and there's a whole bunch of things that went wrong here but they're manually deploying it and the person responsible for it deployed it to seven of their servers the problem is they had eight I want to pause here and say I'm one of those people who believes there is no such thing as human error doesn't exist the problem wasn't that the person didn't deploy to the eighth server the problem was they didn't have automation to prevent him from deploying it to the eighth server okay that said didn't deploy to the eighth server what this program did was offer to buy and sell stock at ridiculous prices to then simulate what would happen to the market so stocks ten dollars it would offer fourteen or six or you know whatever just to test what happened and it was not really doing anything but it was running and watching the real stock market except for the eighth server was really doing it because it wasn't turned off on that one so they had four million stock executions thirty nine seven thirty hundred ninety seven million shares in roughly the time it took me to do this talk they lost four hundred and forty million dollars one automated script would have stopped this and a whole bunch of other things I mean like I said it's not it's never just one thing but the point of the automation here too is to reduce your risk a lot of people think that oh if I only deploy every six months that's less risk no it's more risk because now your deployment has six months worth of code in it it's actually less risky to deploy more often if they had deployed this in little bits and found the little problem they probably wouldn't have got acquired the next day because their market cap was four hundred and twenty million somebody just came in and liquidated the company so summary the goal of continuous delivery is to make sure your software is always in a deployable state don't confuse continuous delivery with continuous deployment continuous delivery is I can deploy anytime I want continuous continuous deployment is I do deploy anytime all the the test pass I believe it's the same amount of testing either way so all the things you do in continuous deployment you should be doing continuous delivery but it's a business decision of do I actually release to customers it's possible order structure changes we went and saw a client today and they were saying should I deploy GoCD this way or that way and like don't care it doesn't matter to the product what do you want to do from an organizational perspective and they're like oh okay it's more likely this is where the issues are if we really want to get development teams in contact with customers obviously smaller pieces easier to deliver please don't skip your security tests and compliance tests and so forth I put my credit card online about nine times a day and I don't want to go broke these things are important I'm not sure how the laws are here but in the U.S. after some major failures people in technical management positions can go to jail over this if something goes out there and there's a security problem or a compliance problem and they lose a bunch of money they lose stock value and they'll come in and say you didn't take their safeguards you should have taken that a reasonable person would have taken they go to jail this is important sorry do you have the CIO? CIO, CTO, Heads of Development yep there was entire retirement funds wiped out by a couple WorldCom and a couple other massive failures where they should have known these things were happening but they didn't have any testing in place so we have a rule set of rules now called Sarbanes-Oxley that are serious yep Sarbanes-Oxley for going to jail no it may not yeah I'm not a lawyer yeah yeah uh yeah I don't know who goes to jail but people go to jail yeah no I mean it's usually people that you know should have had the oversight is my understanding what their titles are it's hard to say anyway that's my story thank you for coming I'm happy to answer any questions thank you so questions at all