 I'm Christopher Giorve, and I go by GorgonLogic online on githubdrupal.org, Github pretty much anywhere that I'm online. Even though that's a Twitter tag, I don't do anything on Twitter, but anyway. So my presentation is titled Automate All the Things. The idea here is basically that all of programming is automation, essentially one sort or another. We're just getting to a point now where we can automate a lot of stuff that used to be very manual. So I'm going to look a little bit at the history of cloud computing going back to the late 1950s, essentially. We're going to go very quickly, right up to more or less the present day, and we're going to start looking at infrastructure as code, what it is, what the benefits are, what we're trying to accomplish, and we're going to examine Ansible a little bit closer to that. Ansible is a tool for configuration management, so similar to Puppet, or Chef, or Cfng, that sort of thing. I personally like Ansible a lot, and it's sort of the base of most of my infrastructure work that I do these days. And then I put it all together with a demo towards the end. So what I'm going to do before we get started is I'm just going to show you briefly. This is Eager. Those of you who know me probably know that I'm the project lead for Eager, which is the hosting system for Drupal, unlike just about every other conference that I speak at. I'm not talking about Eager today. This is the application that we're going to deploy on an Amazon EC2 instance. So this is an example that I just launched as I was madly debugging this before the presentation about half an hour ago. And it is running on an EC2 instance, you can see here, bear with me. So here's the EC2 instance, right above it you see the terminated instance of the prior debugging instance that I had running. So what we're going to do here is we're actually going to destroy this VM. But I wanted to show you that it works, because this is going to be a live demo, and live demos always mess up. So you're just going to trust me that we're going to get here at some point and we'll go from there. So, you know, the traditional way of working with these kind of things is to click around in the user interface and do things like this, and say instance state to terminate, and it prompts me to confirm, and now it's going to go ahead and tear down that VM. Okay? Everything that was installed on it is going to be removed, deleted, and so forth. And then later on, when we get back to the demo part, we'll rebuild all of that and get it running again. So let's get back into the presentation. Oh, just in case you were wondering, the site really is down now. So if it's going to spin because there's nothing responding at that IP address. So that's the plan for today. We'll go through it quickly. I'm going to try to get through this stuff at a reasonably quick pace. Start running the demo. Most of that's just going to kind of flow by as it runs various tasks. I'll narrate a little bit of that and then answer some questions and presumably get into a bit of discussion around some of that. So I'll try to leave about 10 minutes at the end to dig into questions and how you might want to go about implementing these things. So no presentation at a tag conference is complete without the next KCD slide. And so here is mine. This is actually a fairly accurate representation of how people think about automation. So you plan it, you've got the work that you're doing, you put in some extra work to get to write up the automation for it, eventually that kicks in, and then you have much less work to do to maintain that task. The perception, however, in terms of what this is labeling as reality, is that you start writing the code, then you realize that really you can do so much more with it and you start extending it in all kinds of different ways, and the original tasks that you were trying to accomplish with it essentially gets left in the dust as you go into that. So that's essentially where I live. I do mostly that. Part of the reason for that being that I write these kind of automation scripts to help teams deploy this stuff easily, so it's fine for me to get stuck in that space. The reality, though, is actually closer to the top in a lot of cases. It feels like the bottom, but in my estimation anyway, there's a bias that we have where as soon as we've automated something, it drops out of our attention span and we're not even aware that it's still there. So we really do achieve that time, that the drop in effective time that we're spending on it. It doesn't result in free time. We just take on more work. So that's essentially all of computing. All of computing is one form or another of this. I like to go overboard, so I have two exit cases of each slice. This is a really nifty hand you want to have because it kind of gives you an idea of depending on how frequently you're running a task, how much time is reasonable to devote to automating parts of that task. So I'll give an example here, say something that you're going to do weekly, like say, Drupal updates, weekly, maybe monthly, maybe somewhere in between. Let's take weekly as an example. If that takes an hour to run Drupal updates, or say, let's take two hours, and you cut it in half. So you find a tool like Eager, for example, that will cut in half that time. If you're doing that on a weekly basis, you can devote 10 days of time to deploying that fix, to getting that automation in place, and over a five-year period you'll save time. That's over a five-year period. So that's how this works. This is handy because a lot of times, I know for me, I'll spend hundreds of hours over the course of months working on something that saves people five minutes. But again, this is where if a dozen people do it, then that's a multiplier effect for this. So that's one of the really nice things about open source and how you can pursue those kind of what might not be immediately obvious as a productive endeavor, but it has returns in the long run. So let's take a quick look at cloud computing. Just realized, of course, that the L is capitalized, so I won't. So if you go all the way back to essentially the beginning of computing as we currently know it, it was sort of these big, massive, building-sized computers. The skill sets to run them were mostly kind of consolidated, right? You would have to essentially know how the system worked in order to be able to program it in order to be able to operate it. They were so large, so expensive that they tended to be only in sort of the government and academic. It wasn't cost-effective for basically anything else, right? So if you weren't a spy agency trying to decode the enigma machine, you weren't going to be using one of these kind of things. Then into the, you say, early 70s, somewhere around that range, mainframes really hit their stride. Banking, for example, adopted computing wholeheartedly and was thus able to lay off dozens and dozens of people whose job it was to keep ledgers, right? So that's the kind of automation that was happening there. You started to see a lot more specialization where the people who were able to interact with these systems still needed to write programs, and they were often punch cards was how we were entering that data at the time. I say we, even though I had a great beard, it was way before my time, right? But you started to see a lot more specialization. You had engineers that were working at IBM and Craig or building these systems. You had software, a software industry that had built up to write software that was being run. Then you had engineers that were maintaining these systems, but then you had a whole level of financial analysts who really didn't need to know much more than some specialized languages for how to query the data and how to enter the data. So you started to see a fracturing of the workforce and specialization came in. Eventually the PC was released in the early 80s that eventually became these sort of specialized rack mount servers and people started to run a server with email running on it, right? And then another server with a file share, various services like that that became common place and then eventually everybody had a desktop. Everybody was connecting to these services. Systems administration as a profession was essentially born and people would specialize even more so, right? So they would specialize in blade servers or something like that on a hardware side and they would specialize in Microsoft Exchange and how to configure and manage that kind of stuff. And so you got more and more of that specialization. Obviously the tech industry kind of shattered into shards all over the place of different sort of very micro-specialized software. That in 9-11 happened and I was actually selling this kind of hardware and software at the time and what happened was people got super paranoid about keeping their computers in their offices. They figured if anybody was going to target them they were going to target the office but the data was where they felt there was a lot of value. So people started to move their data businesses, I should say, started to move their data into data centers and then look at distributing that so that you had no single point of failure where you could shut down a bank by blowing up its main headquarters. So here again you now have even more specialization. Now you have companies whose job it is just to run the data center. You have people whose job it is to maintain the HVAC systems to just keep everything cool. From our perspective as systems administrators and programmers, again it got even further removed from the hardware, from the underlying software. A lot of times we would be just logging in via SSH, SFTP-ing stuff. This is kind of where Drupal really started to kick in. The web started to really develop at the beginning of this stage. And then finally we are at where we are now, give or take now. I mean it is not like it is evenly distributed. But where most of our interacting with the systems that we run our software on is completely abstracted from us. Some systems are super simple like Linode where you just click a button, you get a new VM and you can SSH in and there you go. Or very complex systems like Amazon where you have got hundreds of different systems of, again, even further specialized types of systems. And computing has really become a utility but what we are starting to see is now a reconvergence of some of the skill sets. So the DevOps movement, which I am assuming everybody is familiar with at this point, is looking at how to do sort of more cross-functional work. How to have teams understand what each of them is doing instead of having these strong silos that are dividing everything. This all happened starting around 2006 and largely as a result of the .com bust. So everybody thought that everybody was going to be buying their pet food online. Now of course some people do but at the time it was way ahead of us, ahead of the curve. But huge amounts of money would be important too. So people had data centers full of racks of servers and then nobody to buy their specialized widget service online. And so they went out of business but all these servers were living in all of these data centers. And so some smart people bought them at a discount and started selling them as infrastructure as a service. So that's where Amazon started and a lot of the, there's a number of services that started around the same time. Basically just taking the commodity hardware that they had and renting it out in some fashion. So it started off as things like co-location at the data center level. But then people put web front ends on it, put an API on it and then everything just kind of exploded. Does that make sense? That reconvergence started around then too because the ease with which you could launch these things. You still have engineers or a racking servers and all that kind of thing. But the convergence of being able to click a button and get an infrastructure system in place in which you then install your software. Also converged nicely with CF Engine and then Puppet and things like that that allow you to start automating a lot of those tools. So understanding the entire stack becomes increasingly important instead of just your layer, whatever that might be. So there's lots of benefits with cloud computing. There's reasons that it's become the thing that it is. Scalability being one of the big ones that people promote a lot. Unfortunately, it tends to be sort of overhyped in a lot of cases. Most people don't need the kind of scalability that Netflix has. Netflix has to have a system in which when everybody comes home and sits down after supper and turns on their TV, suddenly they have 10 times more capacity requirement than they do at 9 o'clock in the morning. So that is a huge difference in terms of how their business can operate and keep costs at a low point. But very few use cases actually require that level of scalability. The problem that comes with it though is that you click a button, you get a new server, makes it really easy to rack up a huge bill. Because if my kid were to get a hold of my computer and just mash the mouse for a bit, I could have a thousand servers running and not know it necessarily. And when I say my kid, I also mean the sys admins that I work with at work. Developers, whoever. So the other benefit is of course flexibility. You can get all kinds of things in place. You don't have to invest a huge amount of money in hardware. Go rent a space in the rack, spend all the time racking it and putting it and installing everything. It's really a matter of minutes to get this kind of stuff up and running. So it gives you a lot more flexibility to do different things with it. The corollary to that to some extent is the fact that with that power of being able to do all of these things, it's much more complex to do any particular thing. In some cases it is just to click a button and you get what you want. In other cases though, it can get really, really hairy. And of course because everything has an API in front of it, it makes it really easy to automate. And so tools like Puppet, like Chef, like Ansible, have made it a lot easier to deploy not just a single server at a time with a click of a button and fill out a form, but a whole fleet of them in a matter of minutes based on some code that it had. That's great if you know how to do it or if you have the time to invest to learn how to do it. It's hard to hire people to do it though. And so people like me get to charge a lot of money to do it. But for businesses it's a real challenge because you get a, not because they can't afford it, but because they just can't find people to do it. To do it responsibly. So let's turn to infrastructure as code and look at what it is and kind of why we're doing that. So essentially infrastructure as code means that you write up a description, a declarative description of what it is that you want. What is it you want to have and then you hand that over to a tool like Ansible, in this case this is Ansible, and it will handle actually running that. So Linode is just like AWS or many other cloud providers. It has UI, I can log in, I can click around and I can create the things and I can check off the various options that I want for any particular system. One of the nice things about how infrastructure as code works is that I don't necessarily need to check off the same box every time I'm launching a new system. I can define what I want these systems to look like in general and then I can just spin those up by running one command on the command line as opposed to having to repetitively ensure that I've got all the right check boxes checked off. One of the challenges with that is that it's not very discoverable. If you're not familiar with how to go and look up what the options are available for Linode, you're not necessarily going to find, oh look there's a new check box that gives me the option to set up automatic backups or whatever when I launch my system. Documentation behind this is incredibly important. That's one of the reasons that I like to actually put in a link to the official documentation within my document. One of the main reasons for doing this is to avoid what we call snowflake servers. So snowflake servers are unique servers, snowflakes being unique. As soon as you log into a system and run an app to get installed, whatever, the next time you deploy a system like that, it's not going to have that on it. You're going to have to do that part again. So if you can write all of that into code like this, then you never have to redo that. But the more you go in and you change a confile and you do something along those lines, the harder it is to maintain control over that system and to understand how it's going to react in different cases. So when you end up calling a system in to debug something, they're going to have to go in and dig into it as opposed to having a system that works a lot more like software. The other part that's really nasty about it is that you have your documentation in line. So if you're trying to be really responsible about how you manage your infrastructure, you'll be clicking around in the user interface for your cloud provider of choice. But you're also going to have a document that says which checkbox is to check off, depending on what you're trying to accomplish, so that you can hand it off to the new guy and say, here, go get me a MySQL server or something along those lines. That's a disconnect between the way that you're doing it and why and how you should be doing it. This is one of the nice things about infrastructure as code. You can put the documentation right where it's required. And essentially, because all you're doing is saying, give me this plan at this data center with this distribution, it's essentially self-document. So you don't need that extra layer of documentation. It makes it more discoverable within itself of what's going on and what you're trying to accomplish. Another nice thing is that it really is code. It's just text file, usually. It's usually YAML or something along those lines. Puppet has its own DSL, Chef has its own DSL. One of the reasons I really like Ansible is that it uses YAML. YAML is also something that you can generate so you can do various things to create the configuration that you then run, which can allow you to further abstract things. I don't recommend doing that. That's where you get into that really high curve of spending all your time debugging, how you're compiling your variables and stuff. Never mind. So virtually controlling things is incredibly important when it comes to this. This is one of the ways you would control costs, for example. If you're ensuring that people are committing code before they run this, you can see who created infrastructure at what time and give them an opportunity to write a little note about why they're doing it. It might link into an issue queue, for example, with the request that they're satisfying by incrementing the number of servers that you have running a particular application or whatnot. And you can then track it back. Get Blame is wonderful for this kind of thing because it'll show you who is it that made whatever change that may have impacted your system in a given way. So it gives you a lot of flexibility around how you can go about managing this and kind of ensuring that things are running the way that you want. One of the practices in infrastructure in code in general is to keep things, keep changes to us, like keep them small, right? Don't work for two days and then write a commit message, you know, yellow, and then there you go, right? If you've got, you know, eight pages to wade through to figure out what this change was that Mike made two weeks ago, right? Because you know that Mike made the change. It's hard to figure out what his thought process was, what was it that they were trying to do. So making these small changes is important on that basis in terms of just being able to understand it, but also because you can then atomically say this is the wrong thing, so I'm going to revert that and change it. And then one of our favorite things, of course, is testing. If you're building these things to be reusable, you can group configuration together into something like modules, public modules, in ansible, we call them roles, and by doing so, you can then run automated tests on them, right? You can define what your software, what your system is supposed to look like and then run tests against it broken by the latest change that you've made. Now, it requires quite a bit of thought to kind of keep straight what's specific configuration versus global configuration. We'll look a little bit at that later. But this is something that I always recommend since infrastructure as code really does make it code, it's the kind of thing that you can dump into Travis and have it build such a system and then tear it all back down and tell you everything's okay. Or leave it up if something goes wrong and you can debug it. So, let's take a look a little bit more in-depth at ansible and how ansible works. So, again, ansible is in YAML. YAML is my favorite format for configuration. Drupal has adopted it as its configuration format, which I was hugely thankful for because they were considering XML previously, which is terrible if you've ever worked with XML and not a great language for humans to interact with things. Because YAML looks just like the kind of to-do lists that you might type into a word doc or something like that. It's natural in the way that it works. The other nice thing about ansible in particular is that you see that name line at the very top? That's optional. But it gives you the ability to describe what it is you're doing in line. So it gives you a description that you should obviously make a habit of. That actually gets displayed when you're running ansible. And so you can easily track down where this thing is. So if that part fails and it just means that it's going to be red instead of the nice green or yellow that it would normally be, it's easy to track down the part of the program or the code where this is breaking down for you. So it makes it really easy on that basis. It takes discipline to include it because it is optional but it pays dividends like you would believe. So let's just take a look at what the ansible stack kind of looks like. So first off we would have ansible. Ansible itself is written in Python and depends on a number of Python modules. The easiest way to deploy ansible in my opinion is using pip. So pip install ansible. One of the nice things about pip is that it allows you to install different versions of ansible whereas if you're using something like the Debian packages you can't go back and install the same one as you had last week. Plus you can install user-specific ones. So if you say pip install dash dash user ansible then you can override the system default with the version that you want to run. And that's particularly useful when you have a distributed team who may be using some advanced features of ansible that are only available in the latest version. And so you can make it easy for your team to stay up to date and keep on the same version so that you don't get disparity in terms of what the capabilities are. Ansible is a pretty fast moving project. So you get a lot of new features very, very frequently. So if you're taking advantage of that and somebody hasn't updated in a month and then they run the code and it breaks that's where I'm coming from with this pip install is a really great way to do it. I actually run it off of git because I contribute back but that's a different because I really do dive into that awkward curve of wasted time. Anyway, so ansible has sort of a core functionality and then almost all of the functionality that you're actually going to ever interact with so every time you're writing one of those clauses saying launch me an AWS EC2 instance you're interacting with a module. Modules generally are written in Python but that's just because it's kind of a community thing. You can actually run them in shell script or you can write them in sorry, write them not run them you can write them in any program language you want just have to kind of standardize the input and output. There are some benefits to writing them in Python but there's a huge number of modules we're going to see how many there are for cloud alone. You can distribute your own so you can create your own modules distributed along with roles. So roles are where you start at the level of where if all you learn is how ansible works then you can do an awful lot with roles. Modules require some Python knowledge or other programming knowledge. Roles is a way of combining YAML written for ansible into coherent groups. So you can create a role that will consistently install minus QO the way you want. Thankfully there's galaxy.ansible.com which is essentially like Drupal.org but for ansible where people share pre-built roles. Excellent resource the vast majority of software that's out there can work with roles that are out of the box available. However it's worth building your own roles. Whenever you have any configuration that you want to be able to share build it into a role. I'm not going to get into detail about how to do that the ansible documentation is really good for it and you can deploy your own get based ones really simply. And then this is where I was saying there's that difference between the global configuration and then the specific configuration. So how I want to go about running EC2 instances the latest of Ubuntu 16.04 the latest LTS or whatever as the default image that I might build into a role that's kind of like a policy that we want to operate on. The fact that I want four of them with whatever host names and so forth that's at the configuration level. So you can keep that difference at the role levels where you would do your testing at the configuration level that's just deploying stuff that's just the actual definition of the infrastructure that you want to have. Does that make any sense? Feel free to speak up if you have any questions out here we don't have to wait for the end. So just as a matter of clarity when we're talking about cloud modules within ansible the vast majority of ansible projects don't have a concept of providers but under the cloud ones which we'll see in a minute it's grouped under Linode, OpenStack Azure, AWS so forth and so providers and companies that provide infrastructure as a service it can also be things like OpenStack where it's an open source project that you can interact with using these modules. The modules themselves are the code that you're running and those are that clause where you describe what it is that you want to operate on. So for cloud projects there's actually 380 at last count. This was a couple weeks ago that I wrote this slide so there's probably more now and let's actually take a quick look at that. So we go to docs ansible, here's our list of cloud modules and here I'll make this a little smaller so here's Amazon Amazon alone has 99 modules we did last time I looked then you've got Azure various other ones, cloud stack digital ocean I mean there's a lot so depending on the platform you want to build on chances are that ansible already provides it. If not it's very possible that there are open source available modules that you can deploy yourself to handle them. If not you can hire somebody to build them that's actually relatively straightforward we're not going to get into a lot of detail about it but the principles are often the same throughout a lot. So there were 25 cloud providers and 380 modules there's a vast array of modules now the way that this is split up is that in Amazon for example you're going to have an EC2 module then you're going to have an RDS module then you're going to have an EC2 fact module that actually just queries the EPI for data about your EC2 instances so that's kind of how they split that up is by functionality, by system that they're running. Linode for example has one module it's the linode module it allows you to launch VMs and that's it. So there's a wide variety of different ways of operating these things and how they've been built so you need to be familiar with your provider it's not like this is going to replace knowledge of how AWS works AWS is a huge system with not even one system it's dozens and dozens of services that they offer if you're not familiar with how it works this isn't going to solve it for you so you still need to have that knowledge but it really does make things an awful lot quicker so what I was saying there's a lot of commonalities between how the providers work a lot of that has to do with how to interact with their systems so there's always an API that Ansible is going to be calling out to to actually run this stuff and the reason that this is particularly important is that usually I work with teams where there might be three or four or more people that are going to be interacting with the system and we need to secure their access to it and ensure that they have all and only the permissions that they need to do what they're authorized to do and so that's kind of where we get into this so one of the first things you're going to do when you're building this stuff out is you're going to get an API key from your system or it might be multiple API keys from your provider and those are what Ansible is going to use to authenticate with the API so it's going to call out to the API it's going to provide that key and usually what that's going to do from the API standpoint is it's going to be able to identify what account you belong to, what user you are and then it'll be able to look up what you're authorized to do and ensure that whatever it is that you've told Ansible to do on your behalf you have the right to do and it'll block you anywhere along these lines before it actually triggers the creation of that resource the change in that resource that you've defined or the deletion of that resource depending on what you're trying to accomplish alright, so let's look at what this actually looks like so that's legible, everybody can read that so let's take a quick look what we have here so this is all in a kit repo there's some stuff that we don't want to keep track of here there's some extra tools that I have in here using GNU Make and a project that I've called drumkit that just makes it really easy to write new GNU things so that I can install Ansible from Git for example really easily so that's just some tools that I use then we have the Ansible config file which just tells Ansible how to do certain things like ignore deprecation warnings and things like that in this case we've got the two keys that are needed for interacting with AWS this is something that's kind of specific to the way I do things with drumkit you can do various things but the way that I've set it up is that I load that into the environment by doing this space dot which is sourcing file and what it ends up doing is loading those keys into the environment and now they're available for me to run Ansible with and Ansible will pick them up from the environment and authenticate using that then we have cloud dot yaml that's an arbitrary name this is where we define the actual infrastructure that we're going to build then we have that D which is just that script to load things into the environment debug dot yaml in this case is going to query the API and pull down data about our systems that we have and again this is stuff that is largely customized and running from some roles that I have there's a couple of files that we deploy to sites, to servers mostly to turn off some nginx configuration that's problematic then we've got a definition for our hosts inventory which is mostly not used in this case, a couple of requirements things, the requirements dot txt are the pip modules that we need to call in order to run this system requirements dot yaml confusingly named very similarly is a list of the roles that we want to run then we have the roles themselves this is where the .getignore comes in, I don't actually commit the roles those are separate projects so let's take a quick look in requirements dot yaml just so that you can see what this looks like it's similar to a drush make file for example, if you're familiar with that tool you're basically just saying the various roles that you want what version you want and then when you run in this case it'll be ansible galaxy install dash r tells that we're using a requirements file and it'll go in and see that everything's already installed so we're not going to be doing anything further here if those weren't installed it would go and fetch them and download them and deploy them to the right place that is defined in our ansible config file so there's a roles path let me see that just says put it in the roles directory it's pretty straightforward and if we look in the roles directory itself that's where those got downloaded right and so that's basically the code that's required let's take a quick look at the cloud part so I actually have it running in a different screen here it is so this is a yaml file it's pretty straightforward we've got again a name this is something that's going to be you're going to see with just about all of the cloud modules normally with ansible you're configuring some server that you've installed so you're going to tell it go to example.com or 192.168.0.5 or whatever that's the host that you're going to be running this stuff on in the case of cloud modules you're running on local host you're running on your local system and that's what's going to call out to the API so there's not usually an intervening system you can actually run this off of a server within your cloud and there's some benefits to doing that potentially as well and then telling it to use this role which is a role that I've recently created that allows me to build out cloud infrastructure in a way that I like it's particularly complex this is where that upward curve of wasted time comes into lately this is what I've been spending my time doing what it allows me to do though is define my infrastructure in these variables and it allows me to look them up with relatively human names as opposed to things like EC2 IDs if you've ever looked at these they're like i-aj ridiculous unique identifiers that you're never going to be able to remember you have to go look it up and you would have to enter it into your configuration in order for the system to know that that's the system that you want to do something with change the type so I prefer to do that with human names and I mean they're not very human names but it's certainly better than arbitrarily changing unique identifier so in this case what we've done here is we're saying build me an EC2 instance with using this AMI put it into the dev group which is security groups we're going to assign a public IP because we want to actually be able to interact with this thing eventually we'll put a load balancer we are not going to I'm going to put a load balancer in front of this and remove that configuration the instance type in this case I'm actually looking up my user and using the key that I have that I use to deploy an EC2 instance so that makes it really easy for me to then be able to SSH into it I've got the subnet I want to deploy it to there's a bunch of other stuff I could add here this exact count and count tag are ways for us to only ever launch one of these I want to be able to run this command that I'm about to run repeatedly and not have it launch a new VM every time I want it to launch exactly the VMs that I've defined in here and make it thus idempotent this is actually exactly the way we want it and so I'm going to now run ansible t-book clap and what it's going to do is it's going to gather some data about the system so it's looking up these facts this is that stuff that's in that role right so most of the time once I've written this I no longer have to worry about it too much and then it starts to gather this data about the systems that we're looking at in this case it's just some debug stuff because it's a work in progress it's actually provisioning that instance that could go by very quickly or it can wait the reason I'm telling it to wait is that later on I want to be able to SSH into that system so I want to wait for it to actually be booted before proceeding and so that's an option that I have set so that's where that wait goes for and we can see that it made a change if we run it again it's not going to pause at the point of provisioning because it's going to know that that system has already been provisioned everything will be green the way Ansible works normally is that light blue is just a notice that it's doing something that you can usually ignore the green is everything's good we didn't need to make any changes yellow is a warning something happened here pay attention it's not a bad thing just means something something occurred it's usually a new configuration that you're deploying or something like that and then red is the one where it'll be a debug issue where you're going to have to dig into what's going on there and so now if we switch back to our EC2 console and refresh unsurprisingly I need to re-log it so there's our new instance now what I'm going to do here is I'm just going to copy that since I'm here the IP address I'll use that later and so now we've built our system right now we want to build our system and so in this case let's take a quick look at what that Ansible looks like in this case probably it's like and this is again it's going to use the cloud module this time it's not going to provision any resources it's just going to gather data so that we have all of that information available to us when we're running our configuration so that if we need to look up the hostname for example we know where to go look for it then it's going to run some tasks so right now all it's doing is adding these instances to a dynamic inventory so an inventory is really just a list of servers that you have and metadata about them Ansible provides dynamic inventory scripts which are usually written in Python again that query the API directly and then put a bunch of data I'm doing this myself because I can basically I like to have that human naming aspect of it and normally what it outputs is keyed by that unique identifier that changes when you tear down a system and rebuild it so that's one of the main reasons that I operate on this basis then there's some basic stuff that you need to do to all EC2 instances before you do anything further especially do any kind of eager installation and then you do the eager installation itself I'm going to launch this and then switch back and we can take a look at that while it's running so here we're going to go Ansible playbook in this case and so again it's now looking at the facts and it's just going to go through and in this case it's going to connect with the server via SSH and it's going to do a bunch of configuration on the server itself so in this case it's going to ensure Python is installed it's going to find out some other information about it it's going to set the hostname just things like that then it's going to go on and install nginx php, mysql and so forth so let's let that run for a bit and go back over here and just take a quick look at what's involved there so if we look in hosts types EC2 now what I'm not showing you here is that there's a bunch of other stuff that I had done previously where I configured the VPC the virtual private cloud subnets, security groups various other required infrastructure to make all of this stuff work particularly well out of the box AWS when you provision a new account we'll provide you defaults for the vast majority of those things my preference is to actually manage just about everything so that if the default changes for example we're not cut with a change that's not controlled so in this case let me get back in here so for all of my EC2 hosts this is what I'm going to want to run I'm going to want to define things like an admin user and my public key and it's going to deploy this to these systems such that I can SSHN as me from that point forward just little things like that sets my password in this case Python is really the only requirement Python 2.4 even is the only requirement for Ansible to run on the remote hosts you don't have to have Ansible installed there's no agent required or anything like that everything happens over SSH so however Ubuntu systems don't have Python installed by default in EC2 images so we need to do a bit of a trick here where we're using the raw module that's the module that we're using here the raw module means just run a command over SSH period don't be smart about it just do this thing and so that's how we get bootstrapped into being able to run more complex stuff gathering facts is done by querying the system and saying what's my IP address what's my hostname, things of that nature that then become available later on in the configuration that we're using in this case I've commented out a bunch of things here just to save a bit of time but normally what I would do is run this admin users thing which would create that user I mentioned I run NTP you want to have your systems keep up to date with the time it's really simple obviously but it doesn't come out of the box and so by doing this that's all you ever have to do and now all your systems are always going to be up to date on the time it seems small but if your systems time drifts past a couple of minutes you can have really really hard to debug issues as a result of that like I spent days before I knew that this was the kind of problem like a class of problems that you could really have debugging this stuff because the warning or the error is never your systems time is out of date and I've seen that once on one system most of the time it's something completely unrelated and it can be really hard to track down then we have some security stuff like a service server in some way we want to turn off root SSH for example things of that nature so this is the kind of stuff that you set in place as a policy essentially saying I want to run this on all my EC2 instances and then it's kind of fire and forget from that point forward you never have to think about it except when something breaks and then you go fix it but you only have to fix it the once for that particular bug and then it does some tasks in this case it's doing that hostname setup stuff this again is the next thing to target building into a role and then just making it that one line that's done over here let's take a quick look at what's going on on our other side so here we're actually almost done we're actually almost done having our system all the way set up it's just restarting some web servers and MISQL and there we go so in this case it made 52 changes so it did things like install various PHP really called modules extensions I don't remember anyway and then it went all the way through installing eager and so what I'm going to do now is just let's see hosts file normally what I actually do with this kind of stuff is I create a root 53 entry so that I don't have to do this right that the new server has a hostname that's available to me that gets a little bit complicated because usually I do that within a VPN and I'm not going to try to do that from a conference center so in this case what I am going to do though is simply paste in the IP address let's get this again we can SSH into that system now I hadn't set it up to create my user so I'm going to have to do it as the Ubuntu user but it did use my key right of course the new system so SSH is at you that wait a minute maybe there's a man on the middle attack so this is pretty straightforward just give me a minute so in case you haven't noticed Macs are not my system of choice I usually use Linux I'm a little unfamiliar with how to operate some of these things in an effective manner so bear with me there alright so let's SSH in it's up and running we can see that mySQL is running we've got a couple of PHP FPM processes running and so forth so that's all good in this case because it's eager I'm going to become the eager user and then I'm going to get a one-time login link and let's see if this is working now there we go so we're back up and running having just destroyed it and rebuilt it all the configuration that I like and obviously you can extend this in a bunch of different ways so the current project I'm working on is a high availability system so before it installs eager it mounts an EFS mount where eager is going to be installed the next steps on that are going to be we already have an RDS system defined then we're going to deploy a couple of other servers we're going to mount the EFS as NFS on their systems and then we're going to create a load balancer in front of that, split the traffic between those systems there's no single point of failure in a system like that it's not particularly high performance we can then start looking at layering in things like varnish systems in front of it most of the time when I do that kind of thing there's multiple layers across different availability zones affecting performance in this case we're going a little bit smaller scale and in this case in particular there's just one system that we built but you get the idea I think based on this of how easy it is to move forward with that and I think that's pretty much all I had in mind to talk to you guys about today so I'll just put this back up and answer any questions right now so anybody have any questions? did you follow? it's fairly straightforward okay anybody plan on actually doing any of this stuff? is this something I mean I mean it takes some getting used to, there's a learning curve I mean this isn't it's not it's not necessarily like I say it doesn't make it easier to manage AWS from the standpoint of like if you don't understand how EFS works it's not going to do that for you however it makes it much easier to do it a second time once you've figured it out the first time you don't have to figure it out again that's what the cold infrastructure is code part of it and when how and why did you start using it? what's the breaking point do you want to automate this or are you working in-house on your own stuff, on your own servers or are you doing it professionally and then you know what you've got to make so I used to work at a company sort of co-op called Kumbit who were the agency behind EGR at the time and one of the things we were trying to do was kind of monetize EGR in some way that would allow us to continue working and developing it and so we decided to offer hosted EGR so you can just pay us, we'll run the server for you, we'll install EGR, we'll keep it up to date and then we can do various other things like keep all of your Drupal sites up to date stuff like that though EGR makes that really easy so at that time Puppet was what we were using for that and I built out a system that made it relatively straightforward for us to launch VMs and install EGR and maintain EGR on at some point I just gave up on Puppet and I like Puppet there's some really interesting things especially from more of a computer science standpoint because it has these directed cyclic graphs and stuff like that but it should have the right term for it anymore the problem is that with Python you have to be explicit about dependencies so if one clause depends on some other clause or one resource depends on another resource you have to specify that it'll run things out of order the proper order of dependencies are and then run them the challenge is if you're getting into something really complex with that is that you can end up with these loops of dependencies so it can't figure out which thing to run first because it thinks there's a loop when I can tell you 100% there was no loop there but I couldn't convince it otherwise and so eventually I just sitting on a script I need to be able to move forward the thing I like about Ansible in particular that's why I started with this kind of automation the thing I like about Ansible is that it's sort of driving principle is radical simplicity this stuff is complex enough without having to figure out how your configuration management system works Ansible is so straightforward it's like having super powered bash you tell it once how you want to do stuff and you'll just do it from that point forward and it's really easy to read you can read it almost like English or French I mean if you wanted to do it that way there's obviously the name thing but the way you pass in options are very clear after once you get familiar with what it is that you're operating on and so it just kind of snowballed from there at that point I started to learn a little bit about how roles work that made everything even that much easier and again it uses YAML and I like YAML so much that it makes it really easy to do these things and now I'm doing some stupidly complex stuff and I'm going to have to refactor into Python because it's stupid to do it in YAML but it's a good sort of prototyping way of thinking through a way of trying to accomplish that's kind of where the cloud some of that cloud role comes in basically from there once we were doing the services it supported some of that and then from there I started people would hire me to install an eager system a high availability eager system for them so there's so many times that that's new and exciting and then eventually it just becomes let's just do this easily and quickly and that's what Ansible provides so I mean as you saw 15 minutes at the end of our thing to go from nothing to a fully configured eager system and that's in terms of that initial slide where we were we were looking at how much this one I install eager a lot if I do it by hand it might take an hour if I'm launching the system beforehand and then deploying an SSH key and creating a user that's the pain so if I can say it myself in this case at least an hour you know it's definitely worth investing a bunch of time in doing this and then I'm a huge free software advocate so I make all of this stuff available to everybody else to use and thus making it easier for people to adopt eager I mean I do this with a lot other stuff too but that's kind of my bread and butter that kind of stuff the advantages of Ansible I've heard about Ansible Tower it's a UI for Ansible I'm sure it has some benefits but I would recommend you check it out I don't use it for whatever it's worth I'm planning on the next generation of eager probably having an Ansible engine as the default engine for actually doing stuff instead of writing in PHP how you write a V host and stuff so I'll probably be looking at tower a little bit closer just to make sure that we're not reinventing the wheel completely then but it's a UI for Ansible as far as I can tell if not I think we're coming up on the top of the hour thank you very much