 We saw five minutes to launch here, but you can tell I'm kind of like Ready to kind of get going so we can do a little preliminary stuff. I'm curious Who here's like doing PHP unit stuff on a like a more or less regular part of right? certain amount how about be hat certain amount of be hat going on and like Selenium or Night watch Any any other testing frameworks that I have mentioned that people are using Cyprus and what is Cyprus test? and So full stack across yeah, and Any like link checkers Load load load testing any of some of that stuff. Yeah going out This isn't this isn't specifically a talk about testing and we could you know that would be a great talk to have to but But you can't really separate Delivery practices from testing they're kind of bundled together and So mostly we'll be talking around the edges of these these testing frameworks And so how many people here are on? One of these like hosted Services that bundles delivery like pantheon and platform that sh and I think you mentioned azure is has a bundled pipeline They have they have pipelines and and somebody also mentioned git lab is I'm not familiar But git lab also has like bundled pipeline so increasingly you don't have to build this stuff Right the way we did a few years ago Because it's all right there you can buy it. We're still up here, right? So we here's another I guess this is the little preview section is that This slide deck is done in remark J s Instead of using keynote. It's the first time I've done a remark J. S slide deck and it's pretty cool I like it right so it's and so it's sort of like a Like a meta version of continuous delivery because it's like continuous delivery for the slide deck Hey, because they because the and I can I we have a minute so I can show you that the Right Yeah, I don't know I don't know if I could be my screen management. It doesn't seem to show up I Can't really so but in any case the whole slide deck is just It's just marked down, you know, it's marked down in CSS and And it's just all pushed out to a website. It hangs off. It hangs off of a website And so and I could do the presentation from live this is my my Blog site Michael godeck.com and you can actually follow it on your on a phone or a laptop Go to Michael good at Michael godeck.com talks and then there's this triple con talk here And then down at the bottom. There's a link to slides and it's the same presentation that we're doing here Right. I'm just doing it from local host, right? So it's the same. It's actually the same Kind of thing, but I suppose it's more reliable to do it from local And so it's remarked js with Uses at Iran deck is a CSS library and Discard days is another CSS library that does the top typography And so what I'll do is I'll make a I'll make an additional slide that describes the technology that goes into the slide deck And I'll add it to the deck and I'll and I'll push it to the to the deck afterwards So we have that all out there So looks like we're on time so We're gonna start right and So we're gonna start with this question right which maybe I already know the answer to but you know Have you ever pushed code to production only to wish that you had known something first right something that you only learned After you push a few of you have been in that situation How many post-mortems have you been into when it turned out that the cause of failure could have really easily have been avoided If you'd only known something first And so really how would you like to be able to go back in time if you go to fix these things? So that like they're fixed before anybody sees that Well, you wouldn't really be the first person, right? This is so He's going back all the way back in the 18th century. There was this guy Irish writer named Samuel Madden and he published the first time travel story that I've ever heard of right he was 17 28 he was thinking about the stuff he wrote this book called Memoise of the 20th century and he didn't really have the technology Angle to work with and so he had a guardian angel like go into the future and his guardian angel goes to 1998 would have placed him in the Clinton administration, right and he finds these diplomatic letters I guess there were Madeline Albright's and he brings them back to 1728 to correct some, you know mistakes that people were about to make right, you know the story so That was pretty cool going back and that was a century before like Charles Dickens did Christmas Carol and what is it yeah Christmas Carol and then Washington Irving did Rip Van Winkle and then you know now we're in like we got dr. Who back to the future Black Mirror everything beyond well The news here really is that the company I work for which I can't tell you the name of it for legal reasons Has got a product called time orchestration Right, it's gonna really beat all of these CD tools because you're gonna be able to orchestrate time to go back and fix the Things before they actually happen and we've got a few Approval issues with the FDA about this product, but we're working on it And I've seen it with my own eyes other people in this room have seen it as well, although they might not admit to it But while we're working out the kinks about how this time travel stuff works then that you know the next best thing to time travel is Getting to know something Sooner and that's what we're gonna talk about here for the next half an hour is This poor man's version of time travel this continuous delivery So I'm Michael Godek I'm from little town in Texas just east of San Antonio and I work for this architecture and engineering group in a global pharma organization Where we really have a very big appetite to know a lot of things sooner Because there's a lot of stuff that we really wish we had no look just a little bit sooner So we're working we're working really hard on on these kinds of Problems we're putting a lot of resources into it And so I wanted to address at least this one part that you might wonder You know if the problems of like a global giant really how relevant that they could be with you I just spoke with a lot of you and a lot of you were really just getting started on a CD path And we've got a lot of resources going into it. We've got some amazing talent that works on it full-time but The playing field is really a lot more level than you think right because in a large organization With a lot of resources We have a lot of things that are going against us that are just the nature of being in a large organization That being in a small organization. You don't have those same problems. You can make decisions for yourself You can solve your own problems. And so we all face different problems but the question of getting faster feedback of Knowing things sooner right this is the common ground that I want to Focus on here and in software practice. We call that continuous delivery right as a practice and so just to provide kind of a definition of it is that Because there's a lot of confusion. We thought there's a lot of discussion around DevOps. There's continuous delivery Pipeline is a set of tools and protocols that use to control the quality of changes to production And so even if all you have is a checklist that's written down on the back of a napkin Then you've got to start right? That's your protocol for for delivering right and so if you can If you have some protocols without tooling that could actually be more valuable than having a lot of tooling Without really clear protocols Right so start start with just understanding what it is that you're trying to get done and get those protocols done Now I've done us sessions on continuous delivery at Drupal con Austin in Amsterdam 2014 Drupal South Melbourne in 2015 and there's a QR code to that YouTube video there And things have changed a lot in those last five years You can buy a lot of the things that we had to build five years ago And so this talk isn't really about like the state of the art of build automation but really just to try to get your thinking about CD to Not be so much around automation but thinking of it in terms of The way you go about improving your business practice, which is really what it is. We're trying to get at with CD And then we talk about Continuous delivery is agile successor that came off of a talk by at Andrew bin stock of bite magazine a Number of years back and it implies that agile has become the standard of practice in that like in responsible hands It's effective and I believe that so I'm taking that as a given for this group And we won't talk about it much and leaving aside late the problem of fake agile frameworks If we're just looking at the agile manifesto, right you can print it out on a single piece of paper You don't need a book or a PhD Embracing change coding and short sprints an emphasis on automated testing You know all those things have really helped us all improve in quality and productivity And a central problem that agile has been very effective at is adapting to changing customer requirements And so when we have a realistic grasp of agile then that's getting to be more and more something that we do well but the problem of Maintaining quality and delivery is still very complex and elusive problem Even for people that have put a lot of resources into it And for all of the enthusiasm around DevOps the delivery practices are still very much in the early days Right agile is pretty mature, but the delivery practices I think that we have years of work to do to get there and so I'd like to take you back in time a bit right to look at this pre-internet period And look at lean practice before the era of the internet especially at the Toyota production system And so a few years ago. I went to Japan I went to Kyoto like everybody does and I visited shrines, but I also visited the Toyota Museum in Nagoya Which is like a shrine and lean practice itself and the curators They went beyond making mere showcase to inventions that you might Expect, but they tell a story of innovation as a process So they tell the story to take you back to the 19th century Japan where these manually operated wooden Shuttle cock looms were the cornerstone of the economy and there was this teenager Sakashi Toyota who was really fascinated by how these things work He thought about it a lot and put his mind to it And he developed a model that you could operate with one hand which was a big improvement to productivity That was 1890 he got his patent for that and in the following years he iterated relentlessly and he came a number of a number of great designs including the first Automatic shuttle can't change loom in 1903 so when the spool runs out you didn't need any hands It would automatically shift one on and so that was a long and prodigious series of advancements that his competitors sought to copy and Toyota famously said quote The thieves may be was what happened was that people were stealing his designs and his colleagues Asked him what they were supposed to do about these patent violations And he'd said the thieves may be able to follow the design plans and produce a loom But we are modifying and improving our looms every day They don't have the expertise gained from the failures. It took to produce the original. We do not need to be concerned We need only continue as always making our improvements Right, so Toyota's insight into the value of learning from failure and Persistence in making improvements, right? That's the foundation of lean practice today So that stretches all the way back to 120 years ago 130 years ago and So the story goes on this the in kind of an exercise of recursive innovation because these people this family just continued to innovate and in in the 1920s They sold out of their business. They sold all the patents to their to their lip to the loom Technology that they developed they created a new company named Toyota with it with a T And the family name was with a D and they launched into the automotive industry that which was already dominated by American and European manufacturers and they came into it with decades of metal casting experience they thought it was really gonna they were gonna excel in this particular area, but the defect rate of the early engine box was just horrendous and there wasn't any guarantee of success and of course We know that you know, they did actually Pull out of that and the process that they pioneered at Toyota was Eventually championed in the West the way we know about it through was through the work of W. Edward Deming Who was assigned to work with Toyota in the post-war to period in the reconstruction period? He was a statistician a management consultant and just an all-around really great guy and Deming was a catalyst for the science of productivity and he merged this culture this culture of Toyota that already exists with his his own American culture that's identified with Benjamin Franklin of innovation and an invention and with his own really unique contribution of applied statistical analysis that Deming brought to the thing and so these early Japanese auto imports in the US were cheap and low quality But the fusion of Deming and Toyota and other contributors produced what became to be known as the Toyota production system and so Deming's you know through his course of his life at the end of his life his notoriety was limited to like management circles but when Ronald Reagan awarded him the National Medal of Technology and this generated publicity It sparked a series of articles and interviews With Deming he was 87 years old then that was 1987. He was born in 1900 and His unpretentious optimism is vitality resonated with people working beyond spheres of manufacturing where he was known including software thought leaders at the time His books were the cornerstone of the 1970 manufacturing evolution of just-in-time delivery and is a 1982 book Quality productivity and competitive position provides really a perspective of lean theory about the time when it intersected with us software development and So this lean lean manufacturing conception of just-in-time delivery was really when it came to America It was often misinterpreted by the American bean counters as merely a technique to reduce inventory carrying costs but software thought leaders recognized the essential value of establishing a systematic way of quickly capturing and correcting defects and They restated this quality assurance principle as a new standard of practice called continuous integration And that was popularized in 1999 By Kent Beck's book extreme programming explained Right came directly out of this So we'll take let's take a step back and we'll look at how the The assembly line in the Toyota production system informed the software delivery practices because the first time to run this Toyota production system in the US was the Toyota GM joint venture at the new me plant in Fremont, California and That was in 1984 and this same plant is the Tesla Motors factory today And so it was the GM plant it was this new me joint that GM Toyota joint venture and now it's Tesla So it's an awesome building has a lot of stories to tell that building So this GM plant in the 70s had the highest defect rate of any automotive plant in the 19 in the 1970s Which is saying something right and they were producing the Chevy celebrity in the old cutless era Until finally they shut the plant down because the quality was just so bad right And there was a UAW worker from that plant who had said quote I can't remember any time in my working life where anybody asked for idea my idea is to solve the problem There's nobody to pull you out at General Motors, so you're just gonna let something go hundreds of misassembled cars Never stopped the line was the motto at GM right just keep pushing things into production And it may sound odd, but it's like a lot of some organizations I've been into pushing code into production basically has that kind of feel to it like push to production Right don't stop So when taking over the Fremont plant the Toyota insisted on retaining the same union workforce that GM blamed for the quality problems Right GM shuttered the plant because of these terrible workers, right and the Toyota's guy the GM guys warned the Toyota team They said okay, we understand you're from Japan you're naive, right? You just don't understand really how bad Americans can be and so Toyota took these work these United Auto workers from the failed new me plants in Nagoya And they showed them how the plan is run and Toyota production system and these UAW workers were shocked to find that under Toyota They would have the power to like Shut down the line over a quality issue right they had the authority as an individual anybody on the plant for it had the authority so in the Toyota production system you have these Pull cords over every workstation you pull once and it turns on an amber light manager comes over immediately to inspect Whatever the issue is and if they couldn't resolve it immediately Second pole shuts down the whole line that is the foundation of what CI is in our system Right where we push code in to some process that we run a bunch of tests. That's the equivalent of the end-on cord From from the Toyota production system So this thing about finding and fixing defects in real time It seems like common sense, but there was a time when it was like not really the pattern and really made radical improvements to car Manufacturing and it brought brought radical transformation to how we build and deliver software And so identifying defects sooner is going to save a lot of time and money. That's pretty obvious, right? And it should be enough to justify investing in DevOps, but when you consider that Every break fix that you're looking at is an opportunity to discover root cause and To take action to try to prevent the same kind of defect to occur in the future That's really what you're after. That's even a bigger value than the cost savings of just finding and fixing things before they get to production And so if you think about that quote from Toyota when he said They don't have the expertise gained from failures. It took to produce the original Right, it's like he was looking at the whole thing of like the learning process of Examining defects right and Toyota plant when they Went from having the end-on cord pulled a thousand times a day to seven hundred times a day Was that do you think that was a positive or negative metric? Right when it happened the Toyota management brought the team in and they said what's wrong, right? The reduction in the number of times that the end-on cord was pulled was not taken as a positive thing Right. Nobody was getting criticized for stopping the line So learning from our defects. That's what's really allows us to improve our process our commitment to improving the process Is what sets us apart from our competitors? so if you look at it from the perspective of Lean theory right you can spend, you know three years getting a six sigma certification or you can take Three points from my talk and consider the job done Okay, and that is that one Lean focuses on identifying what creates value from the perspective of the customer so that's the first thing You know look at it from the perspective of the customer. What is it that creates value for them to? understand the process That creates that value and delivers that value and so that's CD practice Right step three finally never give up on making that process work better So if you do those three things you can skip the six sigma certification and just call yourself a lean practitioner Right those are the those are all the principles you really need in order to understand how to Make the most out of a CD practice at your CD practice as you build it out and so then you have this issue of like Deming if you read Deming It's like he constantly comes back to this question of fostering creativity Right that if you really want to understand what's wrong with your organization? What's holding you back then you need people to feel free to think and in an unrestrained way To not never have any fear of calling out defects or being caught for Being the source of a defect right defects in the internet a production system. It's always an opportunity to move forward Right so if you have a culture where it's like people are afraid of what's going to happen if they're the cause of a break then That's going to be a problem How many people are familiar with the Netflix Chaos Monkey? Right, so yeah, it's pretty broad. So Chaos Monkey. They run this process in their in their build pipeline That just goes and shuts down services and closes ports at random, right? It tears out your certificates and stuff and your and so it's like a pushing software into the build This is like the regular build process, right? It's like a game and so if you don't have a sense of humor then you there's no way you're gonna survive in Netflix, right? That takes there's a there's a certain attitude about this stuff It's like it should be there should be something that's like fine about this And if if the build pipeline is like soul crushing then you want to take a step back and think about it so the build pipeline is where you formalize the ways that your team responds to errors and You know creates a context where you're not just fixing things but learning and improving the delivery practice in the process And so it takes us back the reason why we're all doing this right the first principle of the agile manifesto is You know our highest priority is to satisfy the customer through early and continuous delivery of valuable software Right, that's a principle. That's the practice of CD The kind of the canonical reference to that is this book by Jess Humble and David Fairley Continuous delivery and so there's a lot we could talk great another 45 minutes into that But there's they talked specifically about Configuration management, which is really hard under Drupal and so we have a tremendous amount of work to do in that Configuration is going to be one of the biggest hurdles to running a good CD pipeline Feature toggles there's discussion of that and other things so It's a great read all the way through highly recommended So to try to kind of wrap things up. I want to give you three metrics kind of high-level metrics that you can look at to Tell how well your little poor man's time machine is running and they are One is to ask yourself the question Are you really testing integration? Like what is it you're testing and are in particular you really testing integration to are your stages actually bolted together and Three is like who's what hands have to be on deck for delivery and who's making the call? Those three questions will take you a long way down the road. So the first one Are you testing integration? So CI is the first step of CD and this work, you know codes built every time there's a commit and the Value of CI is faster feedback and so if you have pipelines where the dev instance is unattended in a broken state for any period of time then You know you may have some build automation, but you know What are you doing? I mean, what's the point of that you should have a rule that of the amount of minutes That a put that an instance should stay in a in a broken state and whoever pushes a commit Should attend to see what the result of that is and if they can't fix it if they break the build if they can't fix it In ten minutes or whatever you decide then they should roll it back, right fix or roll back never leave broken So just humble sarcastically remarked there's two words in CI continuous and integration All right continuous means more often than you think and Integration means trunk right the branch of code that you're going to deliver to production So those are two kind of really big challenges in that way If you're bill if you're working on long-term feature branches, which I think is pretty common in this community Then you're probably not doing like many many commits through the course of the day You're probably not using feature toggles And those are those are practices that you aspirational practices that you can look forward to and if you're You're testing before you merge then is that really testing integration because the merge is integration That's what merge means is integration, right? So what we're looking for in in CI or small frequent pushes Where we're test where we have fast-running tests and so Martin Fowler put it this way he said quote one of the challenges of an automated build and test environment is you want your build to be Fast so that you can get fast feedback, but comprehensive takes tests take a long time to run so a Deployment pipeline is a way to deal with this by breaking your build into stages each stage provides increasing confidence Usually at the cost of extra extra time Early stages can find most problem but problems by yielding a faster feedback while stages Later stages provide slower more thorough probing. So deployment pipelines are a central part of continuous delivery That's the Martin Fowler quote. So think about that phrase Each stage provides increasing confidence Right think of the stages of a pipeline is being bolted together What are the bolts what bolts these things together, right? They're they're the the the protocols that you put in place the quality assurance contracts that you say when something's done with dev Then these things are guaranteed We're going to move it on to the next stage and when it's done with that next stage We're going to extend our guarantee on what it is that's tested and when it moves to the next stage Then our our guarantee is broader than that So if you're testing a feature branch on an on-demand environment can be helpful but if the code path from local to production is not consistently running across the same stages with the same guarantee protocols every time then You're vulnerable to the kind of inconsistencies that we're trying to resolve in this thing and Then finally, you know, who's supposed to be involved in the production release, right? So you're the ideal that we're shooting for is that you can have you can give a non-technical Stakeholder a green button and a red button and you show them the stuff that's on stage ready for delivery And if they like it they can push green and they shouldn't need any technical assistance for that to be delivered and you shouldn't really even need people on call for delivery and If there's anything that goes wrong with it, then they should be able to press the red button and it rolls it back and they also shouldn't need People on call to do that. And so that's one of the things if like if you're starting to build delivery pipeline Really one of the first things that you should do is to write rollback protocol Very first not don't wait, right? Write your rollback protocol at the very beginning because it's the beginning of this practice all the way through and If you're if you're a small shop got limited resources You're even working solo You can still embrace all of these goals, right? Because if your pipeline is just a few bash scripts with a few well-considered tests Guarding each stage then that can be a more proper Pipeline then some overblown DevOps operation with Kubernetes running and stuff that never seems to get like Every all their ducks like lined up in a row before things go to production, right? So the important thing about having a CD pipeline is Knowing what your protocols are and applying those protocols as you move stuff across to Production and the automation assists in all of that, but your automation is not your delivery pipeline Right your protocols are more important part of it and you don't need a budget in order to set your protocols You can do that without budget, right? so I Don't know if they're I don't know exactly. We're on time. Hopefully I have time for a few questions Thanks for coming out certainly come to the code contribution sprints sprints on Friday The slides are there and that QR code goes to the slides and again I'll update the slides for the Technology that goes into the slide deck and there's a bunch of other posts on my site that are kind of related to this whole topic area So thank you in any questions anybody Yes, no, maybe so thank you