 Welcome, welcome, welcome aboard, people wandering in. My name is Michael Godak, I come from Austin, Texas. I'm a developer. I've been doing Drupal stuff since around 2001, something like that, and a dev for a long time before that. And I work for ThoughtWorks, which is a global software company. I'm just a little guy in a big company in that sense. And we're gonna talk about continuous delivery. So, but I wanted to try to get a little feedback in here, like how many people in here are, we're gonna go over some like roles, right? Roles are like dev, QA, project manager, other, right? So how many devs? You can be more than one, a lot of devs, right? QAs, dedicated QAs, not a one. And like business analyst, project manager, right? It's back in there, and like bosses, right? I mean, it's like, yeah, I'm the boss, right? It's coming up. So any other roles that I didn't cover right in there, people that just wandered off the street for coffee or anything else, right? So, and how many people here are like doing continuous integration today, right? And anybody doing continuous delivery today, CD, right? Going through, yes, all right. So, we'll just launch in, there's folks coming in here and everything, but there's a lot of material. The talk will be, there's like three kind of sections of the talk, right? The opening part is the turkey talk, right? And that's because like experts are turkeys, and this is sort of like, I'm the expert or something like, I'm not, you know, I'm just a, what an expert is a turkey, turkeys get shot. So we'll try to keep that opening part like short, but we have to go over the why of why we're doing this, right? Because it's really kind of, in most of the dialogues I have, it's kind of the most important part, because people kind of get it that they wanna do it, but they don't really have the narrative to sell why it is that it's important to invest in getting this work done, right? So we'll cover that first. And then we'll look at a particular tool, right? Or this Go platform, and not the language. And then we'll go into some, like case study stuff towards the end to try to put it together so to give you, not the way that I'm saying that you should do it, but a way that it could be done, right? Just to try to give it some context. So that's really the scope of it right here, right? So we launch, right? We'll go in right here, right? What the first question is what is continuous delivery? Right? And so continuous delivery, we'll call it like a technical practice of Agile, that's concerned mostly with getting good ideas delivered in the shortest time possible. That's really what we're trying to do, why it is that you wanna be doing this stuff. So since it's kind of about this question of time, getting stuff done in the shortest time possible, then let's go look at time. So we have the time that it takes to get a story. There's some of these terms, like there's story, requirement, feature, they're kind of synonymous, right? And we in our culture use the word story to say the requirements that we're asking a dev to deliver. So the time it takes to get a story to dev complete is called cycle time. That's your cycle time. How long does it take you from the time you give a requirement to the time you get it's done? And this, the cycle time tells you something about like developer resource utilization, but the larger context is the time it takes from the time an idea is originally expressed and a requirement is expressed to the time it's actually delivered for use by a customer. And that's called lead time, right? So continuous delivery is concerned with both cycle time and lead time, but we really wanna emphasize lead time because you don't get any return on investment on software until it's actually delivered for people to use it. So the fact that you've got something dev complete, if it's still sitting there and not delivered, there's no value in it, right? It's still just an expense that nobody's actually getting any benefit from. So cycle time, lead time. So here's an image of the product that we're talking about Go. It's an open source product. And I guess I'll, let me take a sidebar. We actually have two copies of this book right here, Continuous Delivery by Jez Humble and Dave Fairley, who were thought workers at the time they wrote it. And it's really the best book on the subject. It's an awesome book. You should definitely read it. I'm gonna give away two copies of this book at the end of this session to whoever tweets the most interesting questions at the hashtag goforcd, G-O-F-O-R-C-D, right? Or, okay. So anyway, two copies are right here, honey. Okay, so we have this idea, right? We wanna get to faster cycle time, faster lead time, faster feedback, continuous delivery. We're gonna need some tools to get this stuff done, right, to do this. But, and so we're gonna look at this particular one, Go. Not Go-Lang, but Go-C-D. It was a ThoughtWorks product. It's now an open source project, right? So now you can just go to, it's go www.go.cd is the community site. And then it's on GitHub and you can get it, install it, and it's yours to use and contribute, et cetera. It's an awesome tool. There's other tools. Hopefully everything that we present in here, if you decide to use other tools, the concepts are still the same. We wanna get you to understand the lay of the land of what CD tools are really about and how to differentiate what you can do with Jenkins with other tools to ElectroCloud or Bamboo or Go, right? But Go is awesome. But still it's like, you gotta start with this caveat that usually the biggest problem with getting your organization to, you know, transition to CD is typically not really the tools. There are usually things to have to do with like organizational change management, architectural issues, process issues, that kind of thing, right? So it's kind of like a big takeaway we're going for here is that we wanna leave you with a better sense of what you're trying to achieve in a CD transition so that you can take this question of tools, which are really fun, and put them in the context of all this other work that you have to do in order to actually get to delivery. Because delivery is not just a technical thing. It has to be the, it's really, it involves everybody on the team from customer through to dev, everything. So let's look at a CD in the context of Agile, right? Because CD is continuous delivery is born out of Agile, right? So the Agile Manifesto from 2001, the first principle of the Agile Manifesto is, quote, our highest priority is to satisfy the customer through early and continuous delivery of valuable software, right? So the continuous delivery movement as we know now was simply trying to realize that expression from the Agile Manifesto, right? Now, Agile's the term that it's like it's been so abused at this point that often it's kind of hard to know what people really mean when they say it, right? They say, I'm Agile, you're Agile, whatever. So when we're talking about Agile, we're talking about an adaptive process as opposed to a predictive process and we're concerned with making sure we're building the right thing. That's what we're talking about with Agile. Now, iteration is often associated with Agile, right? Clients often going to agree to an iterative process and principle, but still like they essentially impose this predictive model on the project, right? I want these requirements, we'll do iteration, we'll do Agile, everything, but we want these requirements in this period of time, right? And so if you've got this like predictive straightjacket on top of an Agile process, then you end up with a process that really is dedicated to prioritizing and estimating requirements, sorting the requirements into sprints, like maximizing resource utilization, that kind of stuff. So essentially what you have is like a project management process, leaving out a lot of the technical practices that were discussed in, if you go back to what Agile, the intention 2001 forward, right? It's not just the project management processes, there's technical practices such as continuous integration, automated testing, paired programming, unit testing, and continuous delivery are technical practices that are part of Agile, right? And I don't mean to state those as prescriptive things like you have to do paired programming to do Agile, but those are technical practices that really, you need to be doing parts of those to really get the benefits of an Agile process. And so typically we get into the discussion of like, why is it that you can't, on your organization, you can't get to these technical practices and usually it has to do with like, our customers won't pay for it, the customers won't pay for quality. Everybody's got their own story, but we hear that part a lot. So really to get into this question of context, we'll step back, have this short discussion about this relationship between quality and productivity. And to do that, there's this guy, who knows this guy, W. Edwards Demings, right? Demings is he was this management consultant and statistician and he went to World War, he went to Japan in post World War II period to work with re-standing up manufacturing. And he really, he just changed the world of what he did from there. He's an awesome guy. And he's really associated with this, the Toyota production system, right? And Demings, it's hard to separate those two things. And so Jez in the book in continuous delivery, they start right off. That's how I know that Demings, right? In the reading of this book, is that they talk about really the debt that we have today in figuring out the software delivery process, the debt we have to Demings work, right? We can still go back to Demings to try to figure out what it is that we need to do now. And so I'll close this initial part with this quote from Demings, it's a 1982 book. So Reagan, Margaret Thatcher, all that, and it's back in that time. And he wrote this book called Quality, Productivity, and Competitive Position, which is a pretty cool book. He talks a lot about statics and statistics and it's focused on the manufacturing process. It's really a pre-software world era at that time. So here's this Deming quote to try to put this question of quality and productivity in context and continuous delivery. And he said, Demings says, our aim is to illustrate with simple examples that productivity increases with the improvement of quality. Low quality means high cost and loss of competitive position. Some folklore, legend has it that quality and production are incompatible, right? That you can't have both. A typical project manager will tell you that it's either or, right? In her experience, she'll tell you that if she pushes quality, that she'll fall behind in production. If she pushes production, then quality suffers. This will be her experience when she knows not what quality is nor how to achieve it. A clear, concise answer came in an interview with 22 production workers in response to my question, why is it that productivity increases as quality improves? Less rework. There's no better answer. These workers know how important quality is to their job and they know that quality is achieved through improvement of the process. That improvement of the process increases the uniformity of output of product and reduces rework and mistakes, reduces wasted time and material and thus increases output with less effort. Other benefits of improved quality are lower cost, better competitive position, happier workers on the job and more jobs as a result of the better competitive position of the company. These are lessons that management must learn and apply. Reduced waste transfers man hours and machine hours from the manufacturer of defectives to the manufacturer of additional good product, effectively increasing the capacity of a production line. So the benefits of better quality through improvement of the process are thus not just better quality in the long range improvement in the market position that goes along with it, but increased productivity in much better profits as well. Improved morale of the workforce is another gain because now they see that management is making some effort themselves and not blaming all the faults on the production worker. End quote. So that's a quote from the very beginning of Deming's book, right? And obviously most of American manufacturing has missed that whole boat, right? Because they're still stuck in this other crazy model that doesn't really seem to work that well. But Deming is awesome, it's a guide, right? For where it is we're trying to get to, right? Quality is not something that you're, it's not an extra cost that you're paying for. This is your road to better productivity and better competitive position and profitability. And the software delivery pipeline is part of the structure to help you get there. So that's our constant goal in this, right? Is to get to foster a culture of improving the process, right? If you Google the improvement kata, K-A-T-A, right? There's a whole school of stuff on the improvement kata, right? What that's about, constantly improving the process. And so this is CD practice is all about getting there and it's about reducing the cost, time, and risk of delivering incremental changes to users. So that's a lot like, you know, the mountains off in the distance, right? They're really beautiful to behold and everything, but they're a little bit further than they at first appear. So nobody's gonna like, nobody gets the CD overnight, right? It's not like you get a tool and now you're CD, right? Your journey's gonna take some time. And so you need to like focus, figure out why it is you're doing this, keep that in context and measure your progress as you're going along, right? You're shooting for reduced cycle time, reduced lead time, faster feedback on all the way through the process and to keep the focus on value-added work instead of non-value-added work, which is to say, you know, to try to match your efforts with work that customers are actually willing to pay for, right? So the purpose of the tools are to support these efforts of ongoing improvement of the process of building and delivering software. And so in that sense, all of the effort to build software, build automation is like a musician's rehearsal, right? So by the time that you go out on stage, right? Then all the technical issues should be resolved at that point, right? You're going out to perform at that point. Fully automated deliveries is a goal that you want to get to. If you're not getting to it right away, it's not a problem, right? It's a journey that you're going to to get to where you have fully automated deliveries, deliveries, you press a button. Nobody has the SSH or shell into anything, touch anything, it just works. And you have rollbacks that are just as easy. So it takes the risk right out of it. So we've introduced this concept of the pipeline now, right? The software build pipeline. To put it in context, here's a quote from Martin Fowler on what we're trying to do with the pipeline, right? So we go out, where we go from continuous integration all the way through to production. And so Martin Fowler says, quote, one of the challenges of an automated build and test environment is you want your build to be fast so that you can get fast feedback, but comprehensive tests take a long time to run. So a deployment pipeline is a way to deal with this by breaking up your build into stages. Each stage provides increasing confidence usually at the cost of extra time. And early stages can find most problems yielding faster feedback while later stages provide slower, more thorough probing. And deployment pipelines are a central part of continuous delivery, end quote, right? Martin Fowler. So let's talk about stages, right? So we got a pipeline are these stages. A pipeline in go is essentially a container for stages. Stages are metadata within the pipeline. And so in our case, this kind of case study thing, we're using four stages, a commit stage, a QA stage, a showcase stage, and a production stage. You know, maybe you'd call them something different, maybe you'd have three stages or two stages, right? Commit to production, I suppose you could do that, right? Or you could have more stages depending on what you wanna do. That's not important. The important thing is you've got this flow that goes through, right? So here's what our stages look like. Now we're looking at a tool, right? We're inside go, right? This is a go admin dialogue. And we're configuring our stages, right? So we get from our concept here to actually configuring the tool. So we have commit QA showcase production stages. The idea is that every time code is committed in trunk, goes automatically gonna rebuild the system, right? It's gonna kick off that commit stage. And so the commit stage is continuous integration, right? If you have a go pipeline with one stage, then you have CI, right? You have two stages, now you have something more than CI. And go is like ready to let you grow into that delivery process. But, so first let's take the side bar and continuous integration because the term gets used in different ways. CI isn't running Jenkins on feature branches, right? Because there's two words in continuous integration, right? Continuous and integration, right? And so in order to do continuous integration, you have to be integrating. And running on a feature branch isn't integrating anything. It's just running on a feature branch, right? And so the purpose of CI, one of the things you're really trying to accomplish with CI and modern build process is if you have a stage in your process where you're scheduling time to do merges from feature branches into trunk, then that's kind of a problem at this point, right? You wanna get rid of this whole merge thing, get it down right at the very beginning, get developers committing code at least once a day or more times a day or more directly into trunk. And no feature branches only for like experimental spikes or something like that, only trunk. And then use feature toggles instead of feature branches to separate out code that you don't want to be turned on in environments, but you're just going down one pipeline. And this is kind of, this has become standard practice. This is Google, the entire Google code base is one, they, this is trunk. Everything goes into trunk, right? And they're pretty big, right? And they can manage it, right? Facebook is the same thing, Netflix is the same thing, Tesla works that way. This is the, the time that you spend in this merge process is time that you wanna get back. CI, so we have to step back. The stage is kind of a bit of an overloaded term in here because we're thinking of stages as these deployment instances, right? Dev, QA showcase, right? But in Go, the stage can be like a little more granular than the instances. So we'll drill down, right? These are like 12 stages. They're like four instances, three stages for each instance. So here we'll drill down into just one of these stages, the showcase instance, right? We're actually in Go, we're gonna build three stages. We're gonna build the software, we're gonna test the software, we're gonna release the software. We're gonna do that for each stage, right? So we end up with something, I'm not saying you have to do it this way, this is like a model that we're using, right? So we have these four instances that we're deploying to, and in each one, we're building, testing, releasing, then building, testing, releasing, building, testing, releasing, building, testing, releasing, right? So back again, here's the, each one of those rows is a pipeline. A pipeline is triggered by some change, so every change in source code kicks off a new instance of the pipeline, and every, the goal in CD is that every commit is a release candidate, right? You wanna get from a development perspective, when you push code, then you wanna have this mentality of like, this code could actually go to production and it wouldn't break anything, right? Even if it's turned off, whatever, but every commit is, so commit, the pipeline becomes like the medieval feature quest, right? Where the hero goes after the girl, right? And he's gotta go after all of these challenges in order to get there, and maybe he's slayed in the middle of the, it's the red things, right? Where you have failures, right? You're slayed by the dragon, right? But the point is to get all the way through to green, and the emphasis is on keeping the build in release ready state all of the time. And so when you get to, when you really have a CD practice, the keeping the build in release ready state is actually more important than getting new work done, right? You want your team to always prioritize keeping the release, keeping the build in release ready state, right? That's the first priority. If there's something that's preventing that, then you do that first. So here, we're gonna kinda punch around in the tool a little bit. So you have the pipeline, and then on the left-hand side, if you click on the little link there, then you get this dialogue that tells you the changes that are driving this pipeline build. So you get the committer, the commit message, and the commit hashes that are driving this instance. So you can navigate and see what's going on. This is a drill down into one pipeline instance. So the stages are across the top, across the right-hand side, you've got like a stage history, previous runs of the same stage, and in the middle there, you've got the commit hashes of the build materials, the source code, right? So you can see what's going on. And then on the left, you've got that where it says deploy content in this case, you've got a link that goes into the console. So in Go, you get this console view for every build, every run, every time you get the view in detail. And this console log in this case goes on for pages, pages, pages, pages of detail. So when something breaks, you can go through and see what actually happened on the console, right? So the way that the tool organizes this stuff and presents it to you is what makes the tool valuable. This is that stage history that was on the right-hand side. And so here you've got different runs of the same, different pipeline runs of the same build. And here you can click on that config changed link and get a diff of what actual code changes went into the differences. So if you particularly get this case where you deliver software and everything's good, and then like a week later, you find out that it's broken in some way that nobody really noticed, right? Functionally broken. And so you can go back to that release and you can go here and you can see what code went in at that point, right? So the tool helps you out with that. So once you can get all the way through, right? On this journey to fully automated deployments and you gain enough trust in your automated tests that you can actually, you're willing to click that button to do automated deployment based on what's there. A really interesting thing happens, right? The decision to release software is no longer a technical decision, right? So if you're a project manager, right? You don't have to go and ask the devs if what's done, right? Because you know how that conversation goes, right? Is it done? Oh yeah, right. Is it, I mean, is it, how done is it, right? Oh yeah, it's totally done, right? Well, I mean, is it done or is it done? I mean, right? So, you know, you want to resolve all that stuff. So the software that's on showcase is done, right? So you no longer have, this is what you're really going to, right? So you get this transparency out of the process, right? So you want to build this confidence that what's on showcase is what's done and you don't have to review it. And so at that point, the conversation, the release decision becomes a conversation really between the project manager and the product owner or the customer, right? To say, okay, here's a release candidate, right? This is the stuff that's in it. Do you want the stuff in production? Because if you want the stuff in production, we'll click the button, you'll get the stuff in production, right? If it's not what you want, then we'll keep working on it, right? So then it really, it changes everything, right? So you think about it, right? With Agile, the Agile recognizes that when stakeholders first see working software, it changes their perception of what the requirements are, right? So in Agile, we try to like, we say, okay, what do you want? Here are the requirements. We try to like build them something and show it to them so that they can say, oh, no, no, no, so what I actually want is this, right? So you still have time to just like build what they actually need, right? And that's a good thing, right? That's what part of what Agile is about. With continuous delivery, it's taking that same idea really all the way through to the delivery side, right? To change this relationship between the stakeholder, the customer and the build team. And this really implies some really much richer relationships with the customer. That's really where a lot of the value is. The value is not so much in like saving time through automation so you don't have to do the DevOps work every time, right? The real value is on how it changes your relationship with your customer builds a trust relationship. The competitive advantage of it is awesome. The alternative, right, is like you spend a lot of time like fixing things, right? Low quality, Deming says, low quality means high cost and loss of competitive position. So let's look like in more detail of the tool, Go itself, right? So Go is, it's a server and it's agents, right? So to do Go, you download the server, you install it to your server instance, to a VM wherever it is that you're gonna run and then everywhere you're gonna build software, you install an agent, right? So it's a lot like Jenkins. Jenkins has a master only mode but most of these have the same concept of agents and a master server, right? And a build pipeline is a sequence of stages that are pinned to resources. So here's the, we're gonna like drill down into the stuff, right? These concepts. How to declare build materials, how to set up build stages, how to set up jobs to run in stages, how to get agents to run your jobs, how to configure tasks and how to use trusted built artifacts, okay? And so this thing, one of the things that is takeaway that we hope to get out of this is to understand this distinction between build materials and build artifacts, right? Materials are like the ingredients going into your recipe, right? The source code is going in. The ingredients are constantly changing the recipe, right? That's why you're building this stuff. And the artifacts are the cupcakes that come out of the oven, right? After you bake the recipe, right? They're the result of the process. And so you're always tinkering with the recipe. No two batches are gonna be the same. So GO supports this concept of trusted artifacts that allow you to have these contractually consistent cupcakes within the context of a given build pipeline. And so for like a Java build, the Java binary is the contractually consistent object, right? That you only build the binary once in the build pipeline and then you deliver the built binary down the stream. So you never have to worry about whether or not somehow it's built a little different in production than it is in earlier stages, right? So in our kind of case study thing, we're gonna look at the Drupal database as the artifact, as the cupcake, right? As that we're gonna bake. So it's a little different concept than how most Drupal sites are built and hopefully it'll be a stimulating discussion. So we'll come back to that, but let's kind of take a step back, right? The first thing you do when you set up a pipeline is you declare build materials, right? You say, here's my source code. This is where it is. And then go is gonna go and pull that. Whenever that changes, then go is gonna like kick off a build, just like Jenkins would, right? Or any of these tools. And the agents are gonna take care of like pulling the source code onto your server and getting you started. And so to get the build to kick off automatically whenever your material changes, you just gotta check off this automatically, automatic pipeline scheduling. That's what gives you, once you do that, you declare your materials and you check that box, you've got CI at that point, right? Not necessarily your tests, but you've got CI. So far we've covered build materials, the triggers and stages. So next from here we drill down into jobs, tasks, and artifacts. And then you're gonna have the whole picture. So a job is still a meta concept, right? It's stage is a container for jobs. And so you declare these jobs and for them to run, go server goes and find some agent that's suitable for running a job and it hands the job off to the agent to execute, right? So servers, the orchestrator, the agents, the executor. And this process of matching up jobs to agents is just a tagging, right? So you set up tags in the agent that says my website DevBuild and you set up the same tags in the job and then the server goes out and finds that registered agent wherever it happens to be, right? So resources are effectively just tags. This is tag matching to get your jobs and your agents to match up. And here's the agent view, right? So this is a list on that right hand side of these tags that allow the jobs, the go server jobs to find the agents. So if pipeline is a container for stages and stages are a container for jobs, right? Then you'll probably find no surprise that jobs are containers for tasks. And tasks are commands to be executed, right? So finally we're getting to actually doing something, right? Up until this point, it's all like meta configuration stuff and the tasks are actually doing something, right? So if you've got a back of a napkin deployment, which is fine, right? So you go and you get your napkin out and you go and SSH into your server and you start executing all the commands on the napkin, right? And that's like your base build, right? And so the next thing is you write that into a batch script and you run the batch script. And then the next thing is you write it in Python or something like, all the go is doing is taking your napkin and bundling it all up and automating it so you're doing the same thing every time all the way through. And the task is where this actually happens, right? So we're gonna, we'll drill down into task, but so now that we have all of the components, these meta components sort of defined, so let's take a look at the architecture of go because hopefully that'll help you understand what you're looking at as you're looking at various CD tools, right? So go has got these carefully considered abstractions providing tasks inside of jobs, inside of stages, inside of pipelines with a mix of parallel and sequential processing. So pipelines and jobs are running in parallel, they can run at the same time and stages and tasks are running serially, right? And this was a carefully considered design. If you try to build your pipeline in Jenkins, then you're gonna be inventing your own abstractions, right? How you wanna be modeling this stuff, right? So now you've become the architect. The guiding point in go was this, have you ever heard of Joe Splotsky's, the law of leaky abstractions, right? That was what was driving this. And Joe Splotsky says, right, our goal is to have powerful enough abstractions, the right ones to make it possible to model your path to production and effectively and more importantly to remodel it as you learn and evolve over time while at the same time resist the temptation to continuously introduce new unnecessary abstractions that are only gonna make things more difficult in the long run because they will be leaky, right? So it's true anywhere in software or architecture, right? Is you're dealing with this question of what is the right abstraction for the job that you're trying to do and we try to use patterns in order to avoid the process of us and constantly reinventing our own probably leaky abstractions. And so go helps that go out, yep? So drilling down into this task configuration, right? We have, here's like a copy task, right? Everybody knows copy, right, CP. And this is how you'd configure it in Go, right? You have got the command, the arguments for the command on each line. You've, you know, the command obviously is gonna run on a server somewhere. So the command either has to be on the path or you have to specify the full path or it's not gonna run. And the command is gonna be executed by a user, the user that the agent is installed in. So that user has to have the rights on that system to do the things that you want it to do. They should be self-evident, but you'll spend a lot of time getting, you always spend that kind of stuff and DevOps, right, getting that stuff right is a big part of the job. So the second part of our job, this job is a task, it's a thing called to build the system, to build our Drupal site, right? And so thing is a patchy ant for PHP, it's a target-based XML scripting language and there's a lullabot primer on it, that's awesome. So, you know, these build systems are generally, they're either target-based like ant and thing or product-based like Maven and make. And the target-based systems are like easy to get started with, but they're XML, so they get like long-hared and hard to deal with. The build, these make systems or I find them like hard to like jump in the middle of a broken process and figure out what's going on. So I prefer the thing stuff, but the main thing is that Go isn't like imposing a choice or a way to do this stuff, right? The Go is an orchestrator of your tools and processes as you choose. So whatever works for you, right? The devil you know is probably the best one. So here's the task dialogue for our thing call, right? And we get down to calling user bin thing, right? And the dash F says what file the load, the build that XML, deploy dev is the thing target that we're actually gonna execute. And so this is what the thing actually looks like, right? This is this XML thing code, right? That gets down to this call that says deploy website and the deploy website, then we get down to the next target, right? And so this sets up the Drupal site, right? And it calls at the end of that red box site install. And so just to show that like there's nothing up our sleeve here, right? This is a drill all the way down to a Drush call, right? So Drush is still doing the heavy lifting in your deployment, right? This is you're just orchestrating the sequence in order this is being done in, plus you're marrying all your Drush stuff with all the other DevOps system configuration stuff that goes along with it. And then Drush is supported in Fing with this, it's an add-on called, what is it called, drush-task.php. So if you're gonna use Fing, you've gotta get drush-task.php and then you're in business, right? So you run site install, then you build out the site build stuff. This would be Drupal 7 and Drupal 8. It'll be, this is where your configuration management stuff is gonna come in, where that whole red block is today, right? This is all features and stuff like that is what we do in Drupal 7, right? We build the site out. So we build the whole site out in code from there. So, you know, there's, I could go through, I've got like another 10 slides of the detail going in, but I'm, and we could drill down into that, but probably we'd be better spent like jumping to the discussion rather than drilling through because probably we've provoked a lot of questions, right? So, yeah, how do you sell CD, right? And there's, it's kind of in a similar space of how do you sell Agile, right? And I'm not, I'm working on that, right? That's why I'm here, having this conversation, right? And you know, the tact that I'm on, right, is deming, right? You know, like, take deming as your guide to say, we're not talking about adding on a whole series of costs to your project, what we're trying to do is to get to higher productivity, right? So if you're selling it inside the organization, we're talking about selling it based on productivity, right? Because who can be against better productivity, right? How can you be against that? Nobody wants to pay for quality, but everybody's willing to go for higher productivity. So you keep the emphasis on that and then quality is a vehicle to get to better productivity, right? You got to build quality in. Now, if you're selling it to the client, then product, they're not really, they don't really want to hear about your productivity problems, right? Because if you bring that up, they're saying, oh, you've got productivity problems? Maybe I should be talking to somebody else, right? And so they're really, it has to do with responding to them on a faster basis. In that sense, it's not a question of like selling it to them, like saying, this is what we want to do and you've got to pay us to be able to do this other stuff. It's like gradually getting to the point where these are the capabilities that you have. It's not like it's a big expense. It's your time, right? And once you have the skills, it'll just be what you do. It won't cost you to be able to do these things. It's just a ramp time to get there. And once you get there, then the way you sell it is the fact that the customer says, well, we think that this is what we want. And then like the next day, you say, well, here it is, right here, right? You get to where you can deliver like small incremental changes quickly. And it changes the relationship with the customer and it changes the trust relationship with them. And then once you have the trust relationship with them, then you're on the basis where you can say, especially with a larger organization, you can say, now what we want to do is we want to help you transform your organization along the lines of these practices, right? So you can expand the scope of your engagement with the customer, right? And this is your book, right here, by the way. Here you go. So next question. Come on up. This question is for you. All right. So can you read that question, please? Yeah. What integration points does Go have with ticket systems and other ones? Whose book is this? I have a tarant, Marshall. Here you go, tarant. Come on up. It does. I'm not really like a big expert in that part of it because I'm not using, there is like an integration interface in Go that I haven't delved into. And let's see if we can jump over right here. So here's Go. So hopefully we don't, we're actually not up on screen, are we? So this is, if we take off mirroring, this is probably gonna kill me, right? Okay, so here's Go right here. And then over here in the admin section, you have, or no, actually it's, yeah, admin, pipelines. And then for the pipeline, you've got the integration, those integration points are over here, right? They're plugins, right? The tools are plugins. Go has a plugin interface, so, and it's an open source project. And so there's some stuff that already exists. I know that our own mingle project management integrates some other tools integrate, and other integrations are being built as we go. And the whole interface is there to create new interfaces going on. Kind of as long as we're along this, Go also, you know, this whole thing of the pipeline, Go provides you a GUI for configuring the pipeline, and all of it resolves to this XML, and so you can actually modify the pipeline in this context right here. So that's part of the integration of the plugins. So that's kind of, here's the plugin tab here, right? So, it's kind of a weak answer just because I'm not that familiar with that part of the system, but it's all open, everything can be done. What is done, I can't tell you. Yeah, what do you got? Yeah, in theory, we're kind of really doing different things. Like in our world, we're building, we build our servers with OpenStack, we provision them with Chef, and all that's done before we're talking about doing our Go builds, right? So at least in my, because it's a big world, there's a lot of different cases, right? And I live in my world, and so maybe I don't understand your context exactly, but basically our build is on the application level, right? So we're just going through an application build process, and typically what we're doing is orchestrating a series of steps that you would be doing on the command line. Certainly, I mean, you can be calling batch scripts instead of commands, and the batch scripts can then be contextual in terms of what they're doing, right? I'm not doing anything like that, but sure, yeah, why not? Yeah. Well, yeah, I mean, you can certainly, you can set up four Jenkins instances, right? And then kind of write some glue code to put them together to make a pipeline, but then you'll be declaring your own relationship, right? The relationship between commit and QA and QA and showcase and showcase and production are all gonna be kind of your code that's gonna make that happen. And if you wanna do artifacts, then you're gonna be using some Jenkins plugin to make it happen. You can do it, sure, there's definitely, you can definitely do it, right? So it's just a question of how much of this stuff that you wanna get done in your tool, have your tool do for you, and how much of it you really wanna do yourself, right? So we wanna concentrate on building and delivering changes in the software for our customer, not really, we wanna have the tool, the technical aspects in the tool, we wanna try to bury those so they're not really what our team is dealing with all the time, right? We wanna be dealing with the changes that we need to deliver with the customer as much as possible. And there's a lot of tools at this point, right? There's bamboo and electric cloud, and there's tools you can pay for, there's free tools. Yeah, on set updates? Yeah. Right, so how do you deal with this thing with schema changes, database updates, et cetera? And it's a really, really cool topic, it's kind of a big one. And the path that I went down or we went down was that we build the entire Drupal site from scratch with every build from site install. We start with a blank database and then we do everything in code to build on every commit, right? And then after the sites, we do that and we build everything from code, then we build our content from code as well. In other words, we use, in our case, what we're doing is we take a non-production facing site that the content editors, the marketing people, post their content to this non-production facing site and then we run a node export, we export their content to JSON, we commit it to Git. And so when we get in the build, right, so now we build the entire site from scratch, everything in code, and then once it's built, then we start importing the content from Git and so we can end up reproducing the entire site from scratch all the way through to the production parallel instance and we deploy that to production right beside the existing production site in another document route with its own database, right? So it's sitting right there, ready to go. And then, so then you have the deal, you have the, so any schema changes, obviously they're already in the new thing. If you have schema changes that affect the node import, you've got to deal with those in code. But so the thing that you've got to deal with at that point at actual release is any data that's actually production driven. So if you've got like a Facebook web 2.0 site, this doesn't make much sense. But if you've got a typical corporate property, most of the content's being driven from marketing forward, not coming from public, right? And so then architecturally, we reduce these production generated issues by moving comments to discus so we don't have to worry about comments, right? We moved web forms to Marketo so they're actually not in the Drupal database anymore. Those are, you may not want to make those choices, but those are choices we made to get our content as much being driven from our own source rather than coming from public. And then any remaining data that's in tables that only comes from production, right? That data could have changed the second before release, right? So obviously you can't, right? You have to get it from that source. And so in that case, you've got these two sites, there's separate document routes, they're sitting right next to each other out there in production. And so when it comes time to do the cutover, you click that cutover button, then you put the production site in maintenance mode, do a SQL sync of those individual tables that need to get synced from production to the outgoing to the incoming site. And then you simply rewrite a sim link to what Apache is loading for production so you don't even have to bounce Apache. So the whole thing takes a process of like the cutover takes like less than a second. And since most of the sites behind Varnish anyway, that nobody will ever see any, it's like totally seamless cutover time. And we've done this with a Drupal commerce site, right? And so with Drupal commerce, you actually have large sets of tables that are production oriented, right? So you have all the customer data, all the order data, all that stuff, right? And we could do that and we'd SQL sync it between the two site instances. And it just takes a second to do, it doesn't take very long at all to do that part. And one of the advantages, once you look at that kind of architecture of doing things, then think about what a rollback means at that point, right? The rollback means you just rewrite the sim link back from one site, because the other site's still there, right? The one you just cut away from, you just rewrite the sim link and you're back on the other site. So it reduces the risk tremendously, right? Rises the time and the risk, right? And which are the main goals, right? You wanna get the time, you wanna get the risk, burn the risk out of delivery, right? Get rid of that fear so that you're just like concentrating on all those issues should be done at that point. So there's other ways to deal, this thing with dealing with schema changes, data changes in releases, the one thing you definitely, I always wanna avoid is touching that production database in a site upgrade, right? Because you never really actually know for sure, no matter how many times you've tried it, you never know for certain what's gonna happen, right? And so then, if something goes wrong, your best case is you've made a database dump, you went into maintenance mode, you tried your site upgrade, you tried to test it out, it didn't work, and so you're going back to your database dump and your window's bigger and you're sweating and it's the fear factor is higher, right? So in this case it's like, there's no way you can go wrong in that. So we're out of time, right? But I'm still here. So my Twitter handle is August, 1914. This is however I got that. So tweet me there, I'm godeck at ThoughtWorks.com. Email me there and if you want to have a larger discussion about this, I'm more than willing. Thank you very much. And...