 The title out there is about Docker and that turned out to be a lie. I've rewritten this talk about 10 times. Each time it gets better. This is the 11th time, so I'm sure that trend does not continue and this will be much worse than the last time I rewrote it. So I apologize if it's awful. The way this will work is I've kind of broken it into five or six sections and each section will have a little Q and A afterwards. So instead of me rambling for 30 or 40 minutes, you'll hear me ramble for five minutes and then you'll hear me ramble about a question that one of you asks or two of you ask. I do not, well, I do have a timer. Let me make sure I've got that set up so I'm not going over time. Let me remember how to set up my timer. Dude, dude. I swear, phones and printers are my nemesis. They are the two pieces of technology that I will never understand. Do not turn off auto lock, never. There we go. Timer and go. All right, now I've got a timer. So anyway, this talk will be about Agile infrastructure and in order for us to begin this talk, we have to answer a question. What is Agile? Most of you are Agile practitioners, I'm assuming. You have been doing Agile work in your businesses or have been reading about Agile or have been applying Agile principles and you've probably all gotten very familiar with the Agile manifesto which is the core of all Agile practices. I have a few problems with the Agile manifesto and so I've rewritten it for you today and you can feel free to like, I don't know, throw things at me if you hate it. Make sure it's not like hard things, by the way, if you do throw anything at me. So the Agile manifesto in my mind is very confrontational, which was important, 15, wow, that long, 16 years, 20? I don't know, back a long, long time ago in the dark ages before there was light and I think that we need to reframe it a little bit if we are going to embrace Agile now where we're not quite in the same space. So instead of just individuals and interactions over processes and tools, it's individuals and interactions supported by processes and tools. Any process or tool we have should be in support of the individuals that are part of the team, are part of the organization and facilitating positive interactions between them, which is again, it's not unheard of. This is kind of what the Agile manifesto meant, but because everything was so process heavy back then, it had to be very like, gr, no, processes and tools are bad, which is I don't believe the case anymore. Working software supported by useful and valuable documentation. If your documentation is not useful or it is not valuable to the individuals that make up the team or the customers who are consuming your product or the testers or other people who are engaging with your software, then get rid of it. But otherwise, strive to write useful and valuable documentation. I believe that many teams de-emphasize this and it hurts them in the long run and in the short run. Customer delight via empathetic negotiations. The Agile manifesto is very adamantly against contract negotiation and I kind of agree. I think that the contracts like things are recipes for disaster. There's another talk about that. If you have strong opinions about contracts, you probably should be in that one. But I'm a big believer in trying to delight customers, trying to delight stakeholders by having empathetic interactions between them and negotiating in a way where we try and deeply understand our customers and stakeholders' viewpoints. And then we can come to the negotiation table and say, okay, you really need this for this event that's happening in a month. Here's what we can do in order to make that happen for you. And I feel like that's a much more positive place than like, no, contract negotiation is bad, no negotiations. Adapting to change when reality meets our plan. There's a lovely quote by, I think it's Dwight D. Eisenhower, who basically says, plans are useless, but planning is indispensable. I'm just gonna paraphrase it because I don't have that quote memorized really. And I think it's really important to spend time planning. When we de-emphasize planning in agile teams, it's good. Like if we have significant over-planning happening and we have a rigorous 25-step process, but you see like the other side of the pendulum where there are teams who are allegedly post-agile or fully incremented into agile superpowers and they just have no plan and there's no progress and everything kind of just floats around. And I feel like that's a failure mode of agile adoption. So, yeah, what are your questions or thoughts around that? Does anyone have any things they wanna bounce off me, any other than like, not water bottles, please not water bottles? The next version, you think I should just upload it? Okay, I'll talk to Ron Jeffries and see what he says. He doesn't like change. Oh, that can change, that's wrong. So, okay, any other questions or thoughts? All right, we'll move on to what is infrastructure? So most of us work on these really cool products. You can tell it's cool because it's got like a lightning bolt on it. And our products, they kind of exist in a vacuum until they reach a customer and the customer can interact with it and use it. And this is what I believe infrastructure is. So I'm currently using a stopwatch app. Oh, shit, it just moved, sorry, I swear, my bad. And the infrastructure that got this stopwatch app to my phone happens to involve things like trucks and boats because how did I get my phone? Well, it probably came on a boat. How did I get the software installed? Well, it came over the wire via iOS. Okay, so how did it get there? Like, oh, well, there's probably internet in between there and there's lots of routers and networking systems and all these very complex systems which allow the bits and bytes to flow to get this lovely stopwatch app that I can now use on my phone. And that's all of that is infrastructure, all of it. And that's a lot. When developers think of infrastructure, frequently they think of the interaction between themselves and their operations team which is I just hand them my product and then it somehow gets in the hands of customers and it's magic. And that perspective, while accurate in some ways, deeply minimizes the work that the operations teams do. And in my opinion, a positive working relationship between either multiple teams or individuals on a team are the most important part of a productive team. And it's very, very dangerous to de-emphasize the value of what operations does. So my personal definition, which again, you won't find in websters, you won't find on any website unless I happen to publish it someday, is that infrastructure is the delivery mechanism between our code or our product and our customers. And that is infrastructure. And so when we think about agile infrastructure, we have to expand the conversation beyond just, well, we have an embedded operations person which is good. And start thinking more about like, what is it that actually gets this product into the hands of our customers? Questions or reflections on infrastructure and what that means? I'm getting a lot of nods, that's good. No one is readying the water bottles, this is also good. So you might have heard of this fancy new phrase called DevOps. It's a fancy new phrase because branding is powerful. In my experience, there are lots of people who have been doing DevOps for a long time. And we just had to give it a new name so that we could sell it to organizations, is kind of my expectation. And DevOps I've seen is kind of an interesting, it can fall into several failure patterns which are unfortunate, but it can also have several very positive success patterns. The embedded ops person on your team is actually a very powerful thing. Like I would actually go for an embedded ops person over a dev person trying to learn ops on their own. Because like I said, infrastructure is big, right? Like trying to learn all of the things about routers and networking, load balancing, how do you actually get this product into the app store or get it into the servers or whatever is a non-trivial bit of work. And developers don't tend to be very good at that kind of work in my experience. Unless they've already been like, oh by the way, yeah, I run my own engine X server and I automate that on my weekends. Maybe there's a chance. But in my opinion, the foundation of DevOps is embedding operations people onto your development teams. If you can start there, then you can go very far. If you start with developers who want to do operations and then just kind of like let developers do operations, you wind up with an accidental mess. Because even though we have the best of intentions, no matter how good we are as programmers or as designers or as whatever, if we minimize the amount of effort and understanding that it takes in some other discipline, we're bound to miss things. Like you'll have developers who think that oh, management is easy and then they become a manager. Oh no, those are the worst kind of managers. The ones who like oh, management's easy, I'll be great at it, I'm gonna go do that. And then it's like nope, nope, nope, nope. You need to go get some training, learn how to listen, learn how to empathize with people, learn how to facilitate meetings, learn how to not be a jerk. Like there's all kinds of skills that take a good manager. And when you've had a good manager, you can tell wow, there's a difference between a good manager and an not good manager. So anyway, sorry, that's a tangent that I'm kind of going off on. So the core thing in my mind about DevOps is learning. So developers are really good at automation. They're really good about breaking down problems into smaller problems. They're really good at writing tasks. They're really good at making things repeatable. Operations people and operations teams are very good at making sure things don't fall over. Like that is the primary goal of a good operations team. If we push this into production, it's not going to pass out in the next hour and we have hundreds of angry people and my phone gets all buzzy and then I have to wake up at three in the morning and solve the problems, right? And so developers need to learn how to make things not fall over and how to keep things up and how to actually get them reliably distributed onto the internet. And operations people need to learn how to automate things because those are two very complementary skill sets. Often you'll find that the most effective operations people already do a lot of automation but they don't do it in a maintainable way. Like it's like, oh, we've got this 300, that we got 300 scripts inside of our bin directory and that's how we manage our servers. We just remember which script we're supposed to run and maybe change some files and then push the button and now it's deployed. And that's not a sustainable pattern. Like that's how you get like 100 person ops teams to support like 25 servers. Like, and it happens all the time. The second is about reproducibility. Being able to, so there's this notion like in complexity theory called cardinality which most of you probably know more about if you have a computer science background, which I do not. Where the more aspects of a system that could go wrong, the more difficult it is to understand the system. So if your server, maybe one of them has the SSL headers for 1.4.3 and one of them has the SSL headers for 1.4.4 and all of a sudden 3% of your users are getting like these errors caused by like SSL problems and you see a drop off in traffic and you're like, what the hell is going on? Well, that's because you lacked reproducibility in your infrastructure process. And so operations people are very good at reproducibility. That is their thing. They are adamant about it and that's why when developers are like, could we just install like, I don't know, LibEx SLT and they're like, no, why do you need LibEx SLT? Just put it in Tomcat, use JRuby, make my life easier, darn it. So that's another critical part of DevOps. The third is recovery. If you have a system and you decide to deploy a new version, how do you roll back, right? This is the thing that operations teams spend like weeks, months, sometimes even years, really refining and making sure it works because good operations teams and good operations people, their entire thing is like the developers are their clients and they wanna make sure their developers look awesome and the way that they do that is by making sure the customers are never mad at the developers and sometimes what that means is saying, sorry developers, we're not gonna push, it's four o'clock on a Friday, we're all going home. Even though you really wanna get this feature out over the weekend, I'm the one who's gonna get called when the server goes down because we didn't have the right recovery strategy. So you wind up with these, this is being a really core thing for developers to learn when they were learning how to do DevOps. And the automation inside of it is what the operation team tends to have to learn. I think finally, yes, finally. Sometimes I have to look ahead. There's flexibility, right? This is another important aspect of a DevOps role. The ability to understand that sometimes things are not going to be ideal from the developer side of the DevOps, the developer side of the DevOps, that could mean we actually just copy paste the script and run it. I know copy paste, like the copy pasta, like that's a terrible thing. Oh my goodness, as a developer, my heart palpitates a little bit and I get anxious just even hearing the words copy paste. But sometimes that's what you need to do in an operation situation where you need to spin up a server and be like, okay, is this actually gonna work or not? I'm not gonna extract some reusable component in order to like, genericize the way we install the appropriate libraries that support Ruby 2.2. Like that's not a valuable use of my time. And what all I wanna do is get this server up and see if it's gonna fall over if it gets hit with enough traffic, right? So the developers have to be flexible about there are engineering practices, like we are engineers, we solve problems and we do it well and we make it reproducible and everything is square. And the operation team needs to learn to be flexible with developers who maybe they actually do need or have a very good reason for a package or a library or something that is necessary. Like hey, maybe we do need to upgrade to Ruby 2.2. I know it's just because maybe it's just faster and it's got name parameters, but it might be worth doing even though it's not a significant change from the operation team's perspective. And so a good DevOps team and a good DevOps individual or group will promote this idea of flexibility both between engineers and developers and in operation teams. So those are the five-ish, I think that's it, yeah. The five-ish things about DevOps that I wanted to share real quick. Reflections, thoughts, questions on that. The question is, yeah, so the question was if an operations person is on the dev team or the product team, what would they be doing during that time? So operations isn't some switch that can be turned, right? It's not like, although it can sometimes seem that way if we're a developer or some other group that's not embedded into operations, there's a lot of work that goes around, especially in big IT shops, where you have, well, we need to requisition the servers, we don't have virtual machines yet. In those cases, the operations team member's job is to make sure that all of the things that need to happen in order to have a deployment environment can happen. So if you're at a big IT shop without VMs, sometimes that means filling out requisition forms, making sure they get installed with the right data centers, like following up with people, automating the installation process for the software that your server, that your software will use to run, like all of these things that they'd be doing. And because they're embedded on the team, they can learn very, very quickly, right? Learning, principle number one. They can learn very, very quickly what it is the team needs, and then they can adapt how they're interacting with the rest of the operations group. Or if you have an outsourced operation group, like AWS, for instance, I consider that an outsourced operations group. They can make sure that they're adapting their plans and the work that they're doing with the AWS or with the operation centers to your application. Does that answer your question? Thank you, sir. So I'm not sure they can actually start involving them. What's typically happening is they do a very remote kind of analysis, and then just pulling the L to start to get it. So what can the value acting from off-site, when there's a problem issue, how does that approach it? Yeah, so the question is, so there's knowledge and operations, frequently we'll just kind of like say, oh, this server went down, and it appears to be this process that had the problem, and here's the stack trace, now go fix it, right? And the question is, what are more ways that operations can contribute to that? In my experience, my preferred operation, our outage resolution strategy, is the operations team member and the developer team member are both on call at the same time. So if something goes down, they both get a page, or a phone call, or a text, or a vibrating shock underneath of their bed to wake them up, depending on how deep of a sleeper they happen to be. And they immediately get on a call with one another, like immediately, even before operations looks at it, even before a developer looks at it, because what you need to do is share the pain, right? If you're not sharing the pain between developers and operations, then what happens is this rift emerges, where operation is like, I'm the one waking up at three in the morning when your code is broken, and oh my goodness, you're taking advantage of me, and I do so much hard work for you, and you don't even care. It might sound a little bit like relationship therapy, but that's because it is all software's relationships, right? And then once they immediately get on the phone with one another, or their Google Hangout, or their video sharing, or whatever they're doing, their job is to collaboratively debug that problem. So if it's a single instance of a hundred server cluster that just got a single application deployed to it, then the ops person's job is to know everything about what the infrastructure looks like right now, and the developers person's job is to have an idea of what features had been changed, how would the code been making adjustments in order to help pinpoint that. And I frequently recommend pair programming almost on this, where they might screen share with one another and debug at the same time. And that's how I would recommend you handle these kind of situations. Does that answer your question? I saw another hand up over there. The question I have for you is, based on your experience, if you look at it, you have a specific process, if you say a jail, or a convoy, or a scum, or whatever, right? You have a specific process to measure how things have been taken from the dev under the production. But in the DevOps space, you really don't have anything like that. That's what I feel. I'm part of the past group in my team. And we don't have any specific metrics to say how effective my DevOps model. So what's your thoughts? Yeah, so that's an excellent question. I think I can address that with this. This. Ah, planning. Sometimes it works. So this is a pretty traditional agile development process, right? You've got Diana, the developer, if he's writing some code. They push the code up to CI. CI is a robot because it is a program that's doing execution, running your tests. CI then builds a package. And we'll get into what a package is and why we use packaging in a little bit. And then the ops team notices that a new package is up or someone says, hey, we need to do a release. Or if you're really fancy, you just have like a push button deploy or a continuous deployment system set up. And they use a tool called configuration management. You might have heard of Ansible or Puppet or Chef or Salt. These are all configuration management tools to shovel that package onto the waiting servers. And that's how a deploy happens. My opinion is a solid DevOps team is doing, well, for one, deploys just happen. Like, if it needs to happen, it happens and it's good and it works. And that's how you know it's effective. It's not about how many times you've had to roll back because quite frankly, that's sometimes the developer's responsibility. But how many times does a deploy happen per day or per hour or whatever? And then how frequently, just I lost the train of thought. Do, do, do, do, do, do, deploy. Oh yes, and how fast do those deploys occur? Right, so frequency and speed. So if you can get a deploy done in five minutes or in three minutes or in two minutes, and by deploy I mean the post-package stage. So after all the tests have been run and the package has been built out, can you get that package onto servers and running in production in under five minutes? And if the answer to that question is yes, then your DevOps team is doing amazing. Like, gold stars for everyone, break out the champagne, don't spray it on the servers. But like, that's success from a DevOps perspective in my opinion. So you said Dev, Ops, and then what was the third one? Tech Ops. Tech Ops? Okay, yeah, I mean the question is what's the difference between Dev, DevOps, Ops, and Tech Ops, right? So the line is beginning to blur as more people rely on platforms as a service, as more people rely on like Heroku or Amazon's Elastic Beanstalk, or Cloud Foundry which is something that most of you who are working in larger shops will want to adopt. As people get more comfortable with these tools, the line between developer and operations gets kind of blurred because there's a lot of code that's being written to manage the infrastructure, right? There's robots that are actually doing the deployments. It's not literally the operations person SCPing the files onto the file server anymore or dragging and dropping it into Tomcat or whatever. It's actually like a program that does this for you. I'm not sure, I've never heard the phrase Tech Ops. It might be just another spin on DevOps. But in my opinion, while the line is continuing to blur, and as it becomes more of a cultural thing and it's not so much roles and more about hats and what times and what you're doing at one point in time, the distinction becomes less meaningful, but until you reach that point and you'll know it when you get there. Like you'll know it when you get there because you'll be operating at a really high level from a deployment perspective. Like oh, we deploy in five minutes, we can roll back to any version of the application we want, like all of these things which are very, very hard to actually pull off. But once you get to that point, like then DevOps is a culture. It's not a set of practices. But until you have these principles and these practices in play, you're kind of like, it's not a culture until that point. Does that answer your question? Just somehow actually it's not like I said there'll go into cases. At what point should I start thinking about operations for a minute? So I actually had several slides on it in a previous version of the talk and I'll recreate them shortly. So this bucket of money, it doesn't look like a bucket, it looks like a V. There we go, that's a bucket. This bucket of money is how much an hour of your developer's time is worth. I don't know how much that is worth here in India. I know for me I valued about $150. So anything that would take me an hour, if I can replace it with something I can pay $150 for, I'll take that every time. So if you look at platform as a service options like Heroku or Elastic Mean Stock or some of these other tools that you can just kind of outsource to. Ingeniard does it for Rails apps. Rackspace has this dedicated ops team offering that they provide. There's a bunch of services which offer dedicated operations for smaller businesses. But at some point, the amount of dollars in that bucket that are going into ops or servers and infrastructure and all that, are going to outweigh the amount of dollars, this is the amount of dollars in time savings that it gives you. So if you're using Heroku and you ramp up to like 150 dinos and you're like spending literally tens of thousands of dollars a month, you might want to consider getting an operations person to help under, well first of all get your Heroku and bill under control because it's quite possible that there are optimizations that an operations person will be able to see and understand that people without an operations background won't be able to see. And once you get to that point, you can then begin thinking about, okay, so here's this service that we're running. Let's call it monolith because everyone's got a monolith, right? And the monolith service is where all the CPU and all the memory usage is being hit. What we're gonna do is automate that and bring it into AWS so that we run the servers and we can have a slightly better performance characteristics and we can have a little bit more. But I would tend to as a small, as someone who works mostly with small teams and small businesses, I would recommend starting with just the Heroku's or the elastic bean stocks or something super tiny and super cheap that you can just kind of use. And as you get traction and get customers, you can be like, oh, well I've got this great problem. People want to keep paying me money to do things. Now I can make some financial gains by cutting prices, by hiring a DevOps person to actually, or several ops people and pairing them with my developers to automate the infrastructure that we need on AWS or some other cloud provider. For large companies, it's almost always the opposite. And the advantage to using something like Heroku is the time to market. Like if you've got a large company with real physical hardware and you've got this large ops process that takes a couple months before you can get a single server, then using something like Heroku or an outsourced operations team is a very powerful tool for just like iterating quickly and getting something into production, seeing how it works, moving on, right? And once, if you do that enough, so I do not recommend this, but if you do that and your operations team kind of catches wind and gets upset, that's the possibility for a learning moment between your operations team and your development team be like, hey, if you embedded an operations person on our team and we could cut the cycle time for getting servers up from four weeks to four hours, then we would be totally happy to abandon Heroku and follow all of your practices. And that's kind of how I kind of pitch DevOps in organizations that are bigger and a little bit more process over people focused. I have more drawings. Other questions about this kind of workflow that I showed you? Can you say that a little bit louder? Yeah, so your questions are how do you handle with so like HIPAA or PCI compliance or any of these other regulations that kind of happen. If you have government regulations around the product that you're building, you need to invest in DevOps sooner rather than later and you also need to figure out what barriers you need. So like, hey, maybe you have strong barriers around your database, like really strong barriers. Like you can't access our production database unless you have like this two factor authentication system and we know exactly when you interact it and what queries you ran and all that, right? So as regulation exists and as responsibilities rise around basically consumer protections because that's what they're for, right? To protect your customers. That's when you need to invest more fully in a DevOps scheme that involves a lot more monitoring, a lot more tighter controls and kind of a more traditional operations section. But you try and cordon that into the smallest part possible, like just the database if you can help it and then keep the rest of it in very much a, yeah, we can pop into the machine and like rebuild it and rebuild it in like an hour and it's not a big deal because it's just the application layer. Different regulations will have different requirements. So I can't give you like a gold star, like here's your answer, but my recommendation is to invest more heavily in your DevOps, in your operations and in your developers like if you happen to have those kind of regulations. Other questions about this diagram here? That's great, but if you have some pieces missing over there but the rest of the things are still in place, still you know DevOps doesn't work there. At least that's a real experience what we are having. Yeah, so if I can talk about this from my golden, I don't know, chariot of awesomeness that I get to ride in because I have very small teams that just do awesome work, but not everyone is in this lovely situation where they've already got continuous integration, continuous deployment setup. And in my perspective, there's a couple really big wins, some really oopsies, don't close too much, there we go. There's a couple really big wins that a development team without a lot of operations experience can get right off the bat and that would be building a package. So a package in my definition is an artifact, like a single artifact, either a tar file, a war file, a jar, like one file that when pushed into a repository can be installed on either a machine or developer environment or whatever. So like if you're building Java apps and it's a Tomcat app, it would be a war file. If you're building iOS apps, it will be, I don't remember what it is, it takes forever to build and it's a pain in the ass to configure. If you're building web apps, like Oh, it's Node or it's Ruby or whatever, even Closure. Well, Closure has a jar and war, so don't worry about that. Then that package is going to be a tar file, which is just a zip basically of all of the files in the project. And you would integrate that with your continuous integration setup. So every time you decide to cut a release, your release is not actually shoveling into production necessarily because you've still got kinks to work out on the operation side. Maybe it takes like a day or two to take a package and put it in production, whatever. But every time you cut a release, what happens is a package is built and pushed into your Maven repository into an S3 bucket, into anything, anything. But some location with a build number, a version number, and it can be pulled down onto a server and ran if you so desired. This is, in my opinion, the biggest win a DevOps team can have. Other than getting the infrastructure for taking a package and putting it into production. That's the second big win. But this one, if you can do this, oh my goodness, everything gets so much easier because you no longer have to worry, these questions that often happen like, oh, well, what scripts should we run once this gets deployed? Well, the really cool thing about packages is if you're packaging for a Linux environment, so let's say you're building an RPM or a DB in package or whatever, is you can have an after install script that runs or you can have an after upgrade script or a before install script or whatever. There's a lovely fly attacking me. And so if your package can actually take care of a lot of those things, like what if, and this is mind blowing here, but what if when you built this package and you deployed it, it automatically ran the rake DB migrate? If you're Rails people, you'll understand what that means, if you're not, I'll try and rephrase it. It automatically made sure that your database schema was upgraded and in line with this version of the project and automatically migrated any data that needed to happen, so oh, we added a field, which is a reference to this other table, so we wrote a little script that after we install this version, it actually upgrades that database table and does the work for us. And you may not want that at the package level because a database, unless it's item potent, everyone familiar with item potency, that word? I've got like three hedge handshakes, okay, six. Cool, this is a really powerful idea. This is amazing and I learned it when I was learning HTTP and I didn't understand it, but now it's like everything. Item potency is amazing. So the notion of item potency, you all actually know it. Like you just aren't aware of that fancy pants word that I used. Let's take a very complex problem. Is there any situations in which this operation is ever has a different result? No, never, two times four is always equal eight unless you happen to be a really fancy math person who knows a lot more about math than I ever will and they'll like disprove it over a course of three beers and I'll just feel really dumb. I don't use this analogy with math people because they can beat me with it. But yeah, so this is what item potency means. It's an operation that once executed against, when executed with the set of inputs always has the exact outputs. Which is not exactly what I'm talking about in regard to the database migration. Oh, that would kind of be at this point. But because the inputs would technically change. The schema would go from a schema with these tables to a schema with these tables, right? And so the inputs would technically change but you wouldn't need to rerun it. So long as your scripts or your installation scripts are not dangerous, are not potentially destructive, you can totally automate that and make it part of your deployment process. I drew our lovely developers and operations team together because when you go and actually do deployments, like this is when DevOps is a thing, right? So I mean DevOps is kind of a thing when you're building packages because your operations team will be like, well, we're Windows, so it needs to be an MSI. Here are the build tools for how you do MSI. Here's how you integrate the MSI build tools into Jenkins or whatever your CI is, right? And your operations team will know a lot more about that than your developer team will, like in my experience. And so that's where having an operations person embedded on the team is very valuable because they can answer all those questions in ways that I wouldn't. Like I didn't know how to build an RPM until like six months ago. And the reason I know is because I called up an operations person who I've worked with in the past and said, I need to build an RPM, I'm getting stuck, help me. And we paired for half a day and now I know how to build RPMs and I use it all the time and it's amazing. Anyway, sorry, tangent, tangent, tangent. So if part of your deploy process is literally like, let's put this package on and make sure we've updated any of the resources, the dependent resources to make sure that they're in line, then your deploy process becomes very simple. The hard part is undoing those changes, right? So we talked a little bit about recovery. So let's go with a reasonable deployment process, in my opinion. So you've got some traffic coming into the server, you've got a new deploy that you wanna push out. So you decide, hey, we're gonna shut off traffic to the server. So no traffic is coming to the server so we can deploy the artifact to it. If that deployment is state modifying, right? If it alters the database tables in a way that is destructive for the rest of the application, then a deployment to this server could potentially break these two other servers. So your deployment really should be item potent and it really should not modify state of any other parts of your system if you can help it. So once the deployment happens, you can then direct traffic back to it and your little monitoring bot can be like, okay, is there any problems that are happening? Yes, no. If there are any problems, I need to go notify the Diana's and the operations person, the developers and the operations person and they will work together to solve the problem. In the meantime, let's cut off traffic here. Boop. Yep, see this. Let's cut off traffic. Oh no. I'm so sad right now. Doot, doot. Let's cut off traffic again so that the DevOps team can actually figure this out. The DevOps people, the people wearing the DevOps hat. Whatever words you choose to use to describe this relationship between people who know a lot about operations and people who know a lot about programming, like let's let them handle this and leave that server running and make sure that traffic still goes to these two or potentially you might even be like, okay, we're going to deploy a new server. Like, okay, we're going to actually do an entire server deploy beforehand. And they're kind of running out of time so I want to rush through these next bits, sorry. And the way that you do, the way that you provision servers in the land of DevOps is this notion of a golden image. Some people disagree with me on this, that's okay. They're like, well, we just used Docker and that has a containerization and then Docker can just run on anything and like, great, you can run it on Azure, you can run it on AWS, you can run it on Google App Engine, you can run it on, great. If you're using Docker, don't worry about this step. If you're not using Docker, then worry about this step. So building a golden image is where you take a server specification. So it could be like, oh, this server should have JRuby of this version, it should have lib post-dress dev, it should have this monitoring tool for New Relic, whatever, right, association. That server specification gets applied to a blank virtual machine of your choice. So like, ideally the server spec also includes like CentOS 64 or whatever operating system you're running. And then once that build completes, like this is a build, just like we build our software, then you have this pristine image that is capable of running your software. Now ideally, this is a very, in my experience, ideally this is actually just a Docker server or some kind of containerization server so you don't have to worry about changing this every time your developers need to add a new library. But if you have this golden image, then what you can do with AWS is you can provision a new instance and say, hey, this new instance of the server, I want it to be based off of that golden image, and then bam, you've got this image that you can then deploy your package onto and you're done, right. And then your deploys happen in like two or three minutes. However fast, however long it takes to download the package onto this newly provisioned server and to expand it and install it, that's how long it takes to do a deployment. And then of course to add it to the routing tables and that kind of stuff. But that's pretty negligible from a time consumption perspective. But if you have this golden image, then your life gets a lot easier. So questions about, do I have more slides? I do, but they're not very interesting, so any questions around like deployments and load balance and that kind of stuff? Item potency, golden images, any of that? I saw some people shaking their heads in potentially disgust at what I'm professing because there is some people who just are like, no, golden images are bad. Build the entire service from the ground up using Chef every time. And that's a valid approach. It also takes 20 or 30 minutes just to provision a server. So if you're trying to get into auto scaling, you're not gonna be able to do that because it takes 30 minutes just to provision a server. And that's where the golden image really comes into play. Questions, thoughts? Yep, in the back. Yeah, so this is one of those things where if you're used to like AWS or anything, it's a little bit easier. But if you're at all familiar with virtual machines, like there's this notion of a snapshot that you can save, right? And then you can build a new virtual machine from the snapshot of an old virtual machine. So a golden image is just a snapshot of a virtual machine that has almost all the dependencies that your application needs. Most of them, if not all, but not the application itself so that if you have to do another deploy of your application code, you can start up a new version from that snapshot, sorry, a new server from that snapshot, deploy the new version of your software to it and then start that software running. And if you can do that, then your startup times for deployments go from 20, 30, 40, 50 minutes to however long it takes for AWS to provision a new virtual machine and however long it takes for your configuration management software to install the package on that virtual machine and then however long it takes for configuration management software to tell your routing system, either AWS is route 53, or if you're using console or some kind of like a zookeeper or some other kind of thing to start directing traffic to this newly provisioned server. Does that answer your question? Other questions or thoughts around this? I saw two right behind one another. I think, yeah. So I tend to, most of my projects rely on containerization. So most of the client projects I do, we just deploy to Heroku. It's cheap, it's fast. I mean, it's cheap as a relative term, but if I have to choose between spending eight hours a month maintaining an elastic mean stock setup or no time maintaining a Heroku setup, like that pays off very quickly, right? No matter how you price your time. And Heroku uses containerization behind the scenes using a tool called LXC, which I can never remember what that acronym means, sorry. And so technically most of my projects rely on containerization. There are some projects that I have where the deployment structure is literally, we need to deploy a virtual machine to a desktop, like a desktop, not even a server. And in those case, I build a golden image and deploy an RPM package that can get downloaded. We tried to use Docker for that, but I kept screwing up the server so that it would not actually be able to share files between the virtual machine on the host and the Docker container. And it was just a pain in the ass, so I'm like, okay, fine, we'll just use a RPM and it saved a bunch of time and energy. But my golden images, most of the time, are just like a Docker server. Like it's CentOS 6.5, I think, is the version I'm running, 6.4, I don't remember. And the Docker name and running on it so that I can deploy a container, or sorry, deploy a Docker image as my package. So like Docker fits into this system pretty nicely because literally, Docker build, that's your package, right? Docker build is your entire packaging system. Like now you've got all of the dependencies you need and you can just shovel that out into the Docker registry, perhaps you're running your own Docker registry, perhaps not, and then your machines can pull them down. So we have three minutes left and then we need to be done. Does that answer your question or other questions or thoughts? Who owns culture in an organization, period? The answer is everyone. If you're part of an organization, you are part of that organization's culture, right? So if we're trying to profess some kind of DevOps culture and the only person who actually cares about DevOps is the CTO or the CIO or whatever and no one else does, then DevOps is not going to happen. Because even though the CIO has all the power, allegedly, you can't have like one person outweighing 300 people's apathy, right? It just doesn't work. The opposite is also true. If you have maybe 15 or 20 really passionate DevOps, like operations team and developers team members who are like, yes, we need to be doing DevOps, we need to be pairing together, we need to be on the same team, this is all about, it's gonna be awesome and great. And the CIO is like, no, this is untested and it's probably gonna be painful. Like you're gonna have a hard time there as well. So no one can own a culture really other than everyone. And it's up to each person to be the best person they can be at work and be the most compassionate person they can be at work in order to help walk along these steps to a slightly better process over however long we happen to stay at that job. One final question I think. Yeah, so that's why I consider packaging part of CI's job. Like so there's this notion of an abstraction layer, right? So if I'm a developer and I need to know about like which version of the operating system is running on the servers, that's an indication of a leaky abstraction, right? And maybe if I have to know what kind of packaging tool we're using, that's not a leaky abstraction, but maybe it is, it kind of depends on the context. And so some developers very rightly say, you know what, my job is to build these features. The more time I spend is trying to figure out like how to debug like system D processes on like CentOS, like that's time that I can't spend like building software for the clients or for our stakeholders. And it's in those situations, it's a very tough, you have to figure out where you want to draw those boundaries, right? And I really don't think that everyone should know everything is a valid answer. Like if you're like, oh DevOps culture is, all the operations people understand TDD and refactoring and like all of these fancy ass like engineering practices and the developers all know about packaging and like load balancers and configuring reverse proxies. Like that's also kind of like, maybe not. It's about like building a team, a whole team that carries these skills. Not necessarily every person on that team carries those skills, all right? It is 11.15, I'll be happy to take further questions in the hall, but the next talk we'll be starting soon. Thank you very much.