 I'm going to talk about Chef and Puppet and DevOps and Automation. So a couple of things. I am a little bit nervous and I'm also standing on a raised platform. I tend to walk in circles a lot and we might see some interesting acrobatics here. So be prepared to laugh. So let me, if you haven't seen by the way, they're using raspberry pies to run those slide shows. It's kind of a cool little trick. So oops, of course it's not gonna work right out of the box. There we go. So configuration management, DevOps, Puppet, Chef, all of these things are, they all kind of mean the same thing. They mean a lot of different things to some people. So what I'm gonna do is spend a few minutes with context. Context is important. So I wanna talk about what the universe seems to think that all of these things mean all the different concepts we have about what DevOps, what configuration management is and then what I think it is. Now, maybe you wanna argue opinions. If you wanna argue by the way, feel free. Except for Steven Frost, I really, I think we're argued out. So, but in all seriousness, if you do have questions, please feel free to stop me. I really hate being a talking head. So if you got questions or you have comments, you've had anything to add, a lot of you probably know this stuff significantly better than I do. So please feel free to interject. Who am I? I'm Scott Mead. I'm a senior architect with OpenSCG and BigSQL working on BigSQL. I have been in the Postgres universe for about 12, 13 years now. I was one of the original employees of Enterprise DB. I was, my nickname was Configure, Make, Make, Install because I just did the builds all day long. And now I'm at OpenSCG. OpenSCG, the open source consulting group, we provide expertise to enterprises using open source. We specialize in data management. The cool thing about being the open source consulting group is I love Postgres, but it's not always the right hammer. So I get to use Cassandra and Hadoop and Bob's Backyard, whatever database I feel like at the moment. So it's kind of fun. BigSQL is the world's most complete and developer friendly distribution of Postgres. It's something new that we've launched here at the show. Check it out, it's really easy to use Sandbox. That's the end of my shameless plug. So DevOps, configuration management, rapid deployment, what are all these things? What is DevOps? Is it a development and an operations team? It's developers, it's operations. That sounds like exactly what we already do. So how is that different other than just putting a slash and making the word cooler to say? Is it developers that are operations people? Maybe. Is it operation staff that are developers? Maybe. It's always really easy to say, okay, well, dev guys are gonna be ops guys. That's, it's not realistic. I've been on both sides of that fence and when you're a developer, you're working all kinds of weird hours, you're out at Starbucks thinking about this algorithm and then you get a phone call that, you know, Prodwebo 4 happens to be down because of a bad NFS mount or something. Like, I don't, you're gonna run out of time in the day. There's only 24 hours in the day. And the same goes for ops guys. The ops guys are focused on operating the system, making sure it stays up, making sure we're hitting our metrics, making sure the system is secure, that we're patched. I don't have time to write all the code. So is it possible that developers are operations and operations are developers? Cause that's what dev ops kind of feels like, right? It's like, ah, it's one guy sitting in his dark room and he pushes a button and bam, Amazon web services happens automatically, right? It's not realistic. So what is dev ops? I think of it as kind of two specific areas. It is development and it is operations. Now, I'm gonna start with the operation side because I grew up doing operations. I was a Windows admin back in the day. I know, I know, I know, don't throw stuff. But what is it, when we're in the context of dev ops, what does it mean to be an operations person? What it really means is I want to define what I want on a system, not how to do it. So I wanna say, you know what, OpenSSL should be installed and it should be updated to the latest version. I don't wanna say I have to go download and install or I have to run the exe file, I have to do this or I have to go yum install or app get install or whatever it is. If I'm a developer, what does dev ops mean? It means that I've got code that I wanna get out the door. I should be able to do that often. And there's a lot of implicit undertones in the word often, quickly and correctly. It needs to build, it needs to pass regression, it needs to not introduce billions of extra selects against the database and all kinds of other things. So there's really dev ops, we kind of, some people smash them into one concept but it's still two. What's it really boiled down to? So let's think about it from an operations perspective for a second. If you're in operations or if you've ever worked with operations, just out of curiosity, how many people would consider themselves like mostly operations? And what about mostly dev? Uh-oh, look out, look out boys. Devs are on their way, I think they're taken over. So when all of these developers say, hey I need another node, what happens? What do we as an ops do? It's like okay, first the magic, the node appears. We called Dell, when we called them they started mining for the ore to put the thing together, they got the sand together, made the silicon and they shipped us a box. We get that thing racked, that takes who knows how long. But now this is where your standard view of operations begins, right? I gotta get the host names, the IPs, authentication. If I'm using corporate network, I'm using Active Directory or LDAP or just my Etsy password file, whatever needs to happen. I've gotta do updates. The standard packages we have, maybe I've got a monitoring tool that needs a bunch of prerequisites. The monitoring agent, maybe I've got standard mount points, security policies, SE Linux is horrible. So that took me half an hour because I didn't remember what the context I needed on slash data was. And the reality is it's half a day of work. It's a lot. It's a lot of commands and it's very error prone. This is when I was the IT intern back in the early 90s, we didn't really, didn't do much other than sit down and do this when we got a new server. And that was okay though because we only had a few servers. So then once that's ready to go, okay cool, we got a shiny new box, it's ready to go. Now what do we actually have to do? I've got to get any dependencies for my stack. I've got to get the database, Python and Perl, Postgres installer puts home directory and the data directory which doesn't make sense in the context of SE Linux. So I have to move that thing and then fix a bunch of configurations, install the backup script, get the backup job, get the SSH keys, do the testing, doesn't work, developers aren't happy. It's another half a day. And then, oh yeah, the firewall's got to be open too. So there's a lot of things that go into just getting a new machine, a new node ready to go. We say okay, well, automate it, right? I'm just standing up here complaining about how much time I'm running commands. It's like well, if that, well, I'm actually a terrible operations person, right? So let's script it. Like we have bash, we have command prompt, we've got all these different, we have the Windows CMD, we can script it, not a problem. So okay, let's script it. So do we use Red Hat? So we do a YUM install. Did I, anybody upgrade to Fedora, like the latest versions of Fedora lately? YUM is all of a sudden something called DNF. So now all of a sudden my, all my YUM install scripts are completely replaced by a new tool that behaves differently. App to get install, if I'm using Ubuntu, right? What about that? Now all of a sudden I've got, you can just see in your head, you get one shell script, maybe it's 150 lines that does all of this stuff, right? Maybe it's 200 lines, maybe you're a better scripter than me and it's 20, it's possible. But now all of a sudden I have an if statement that says if I knew Ubuntu do this instead, or Pacman, okay. Next step, go ahead, install open SSL libraries. I dare you. All right, on Red Hat and CentOS, it's open SSLDevel, on Ubuntu and Debian distributions. Today it happens to be libSSLDev, but who knows what tomorrow brings? It's a brave new world. So, and even if we are going to kind of if def our code out, so we have multiple sections, how do I do that? I gotta sit down for, now I need to go download Ubuntu 12, Ubuntu 14, CentOS 6, 7, BrutSystemD, so there's a whole new bunch of stuff. Now I got four platforms. I've gotta figure out, how do I tell the difference in my shell script? And I just ran this last night and whoops, I just said okay, what about catting Etsy release and pulling the first line? Well on CentOS it gives me a nice capitalized, stylized, productized name. On Ubuntu I've got a bunch of extra stuff in there. So already I'm having, now am I gonna make the exception? What am I gonna do? This is a pain in the neck, right? So that, scripting it, really it's an option, but it's an expensive option. It's a painful option. So let's talk about some tools. So I'm not trying to convince any of you to use any of these tools. I happen to love them. I use them a lot. There are the core of what we're really talking about is Chef and Puppet. You notice that's at the bottom. But to get us there we need some frameworks, right? So I'm gonna use VirtualBox. You can use VMware. Doesn't really matter. It's a virtual machine. I'm gonna use Vagrant as anybody in here not, who has used Vagrant? Just so quite a few people. So I won't waste too much time on context with it, but we are gonna talk about it. Essentially for those of you that have never used it, it's a way to deal with virtual machine templates. How do I deploy quickly a VM template? Get it up and running so I can use it. And then Chef and Puppet. Chef and Puppet are gonna do a lot of the workforce and we'll get to those in just a few minutes, right? So if we were in real what now? Okay, other thing, just so you know, I could have done a whole huge presentation on both Chef and Puppet. It's gonna take forever. And I don't really like Puppet that much. But that's just my personal taste. At a high level, both of these guys behave basically the same. You've got a Chef server, you've got a Puppet server, you've got, they poll, you've got clients, the way that they work, it is distinctly different, but they're very, very similar. And they're really at a high level doing the same thing. So what do we have? If I was gonna deploy this in my server room or what we have back in the OpenSCG server world, we gotta get repository. That's holding all of our cookbooks for Chef or manifests in the Puppet world. We have a Chef server whose responsibility is to talk to network clients. And then on our nodes, we have a Chef client. Chef by default polls every 30 minutes, 1,800 seconds. So the idea is that those Chef clients are gonna talk to the server, they're gonna get their state, what the desired state of that machine should be, and they're going to do it. I don't have all of that here, and if I were running that many machines in a VM, I would screw something up. So I've loosened it up a little bit. I'm using this tool Vagrant, which I'm gonna show you guys, which is basically going to simulate my server room for me. It's gonna, I think of a Vagrant, somebody in the dark server room who put servers in the rack and then disappears back into the dark corner again. That's kind of what Vagrant's gonna do for us. And inside a virtual box, I've got my Chef cookbooks, and I have the Chef client, which we're basically gonna run in local mode. So we're not gonna be looking for a server, it's gonna directly process a cookbook file for us and start applying some things, okay? Anybody not familiar, is everybody familiar with virtual box and VMware for the most part, virtualization, okay? Again, I'm using virtual box just because it's free. It really scares me every time it comes up and the Oracle logo is there. Sorry for that. But you could use this with most hypervisors. So let's do, again, basically what Vagrant's gonna do, I personally think it's easier than using virtual machine snapshots. I could go into VMware. I can build a Red Hat machine, I could build a CentOS machine, take a snapshot and then always revert to that. But I always feel like with a snapshot, you don't actually know, what did I put in that snapshot? Did I write it down? Did I actually install this? What versions in there? There's a lot of unknowns when you're talking about virtual machines and templates and how long's I've been sitting on my file server. Cloning in virtual land can be a little wacky. If you've ever done, I did a linked clone on VMware Workstation the other day and then I realized there's all these crazy hard links floating around on the file system and it kind of makes a little bit of a mess and they become very dependent on one another. So we've got woes on woes on woes. Just I don't like them, I stay away from them. Basically, this is what we do when we copy a VM, right? We're gonna make a copy of my VMware template. Oops, I'm out of disk space. That always happens to me. I'm always out of disk space. So we go make disk space, we re-copy it, we start the VM and then VMware pops up. Did you move it or did you copy it? Well, I think I copied it but maybe I want to say moved it and then because of that, the MAC address of the network card gets screwed up. The VM gets booted, network interfaces are gone. Now all of a sudden I restart it, kind of hope a little bit. The network adapters back. I end up on Google for half an hour. How do I delete an ethernet? It's just, it's a never-ending pile of stuff when you're deploying virtual machine templates and you're copying around. So at the end of it, after three hours of doing that, I realized I actually needed Ubuntu and I started again. So what does Vagrant do? Why is it any better? So what I have here, I'm gonna make this a little bit bigger. I'm gonna try and keep this to the top of the screen. If anybody wants me to slow down, let me know. If anybody has questions about this, I can give you scripts of it after we'll try to post them. Basically, I've already done some setup work with Vagrant. You're really defining what the base image, what's the template virtual machine you're gonna use and a couple of different settings. So I have here, I wanna do some port forwarding. If I'm in virtual box, I wanna do some stuff with DNS. I want two CPUs and two gig of memory. So whatever that box that it pulls down, it's gonna get those settings in a virtual machine. So all I have to do to actually deploy this thing is I have nothing running right now. Just type Vagrant up. Vagrant is going to import. It's gonna do that whole slide of stuff that I was just whining about a couple of minutes ago. It copies the machine, fixes the MAC address. It's gonna deal, it pops up virtual box for us. It deals with any of the problems that we were having. It adds network adapters. It sets up the number of CPUs, gives us RAM. And then the really cool thing that I love about Vagrant is it actually, SSH is into the server and does some cool stuff. So you'll see in the bottom we're booting there. I'm just gonna try a couple of times. Don't fail me now. There we go. It gets in, it fixes up the VM, sets the hostname. And now I have it actually installing ChefForce as well. But basically when this is done, I will have started at a known point, which is a CentOS 6.7 server-only install. And when I'm done, I will have done all of those things and I can see what they were. They're done in the same order every single time. There's no question about what that machine is by the time this is all said and done. Now the only trouble we may have is that I'm hoping the Wi-Fi doesn't give out while it's installing, there it goes. Now what I've asked Vagrant to do, and we're gonna move into Chef a little bit, I've actually said, okay Vagrant, when you're done building that template, I want you to actually run a Chef manifest for me. So what is a Chef cookbook? So what is that and what is it doing? So basically I told you already in Chef and Puppet, and if you're a little bit older of a war horse, kind of like I am, you might remember CF Engine, the idea is to defa... Oh, that's kind of dark. Yeah. The idea is to define the state, the end state. And so what am I saying? As I'm saying, dear Chef, I would really like a file that is slash ETC slash MOTD. If you're, again, an old war horse like me, that's the message of the day file, which we don't really use much anymore. But I want you to create the Etsy MOTD file and I want that content inside of it. There's a lot that's going on here. If this was a shell script, that might be an echo. If it was on Solaris, I might have to do a print F and then a slash N at the end. There's, if I was on Windows, who knows. So the idea is that this right here will run other than the fact that I'm hard coding the Unix pass, this will run on any platform. Chef will take care of the details for me. I'm telling it what to do, not how to do it. It knows how to do all of that. So when we go back to our vagrant window here and we see when Chef fired up says, okay, I've been told to update the Etsy MOTD file and, you know, hey, welcome to my talk. And the other thing that happened here that I really, really, really enjoy is the SeLinux security context was dealt with for me. I'm sure somebody in the universe is an SeLinux expert, but I haven't met one yet. And getting the context right is always trouble. And, you know, about three to five years ago, I was the first one to jump up and say just disable that thing. Nobody uses it anyways. But now in the security landscape wherein it probably makes sense to leave it there. So you can see Chef is already taking care of me. Puppet does very similar things, right? Chef is taking good care of me there. Okay, so that's a very simple version of I did a vagrant up. I got a machine from absolute zero. Now I'm a hero. It's completely deployed. That seems simple. It was quick, right? It took, Chef took three seconds to do that. It took us 30 seconds or so to get the machine up. That's pretty good. What about if we get more complicated? Let's just, I wanna make sure I don't go completely off the rails. Let's talk about these a little bit. So Chef and Puppet, everything's ruby. I hate ruby. And that's what it is. In Chef, we use cookbooks and recipes. In Puppet, we use manifests. Again, the idea is you're saying this is what I want. Don't bother me with the details. Sometimes that's not a good strategy in life, but when it comes to computers, I like it. The other thing that these guys do is they provide a very consistent systems information API. So before, I was trying to figure out what is, what machine is this? Is this a CentOS or an Ubuntu machine? And I'll show you in just a second that they do that. The other thing that both of these systems have that's really nice is a nice file management system. So you've already seen the EtsyMOTD file was I could dump content into it. But we have templates. We can modify bits of a file. We can have it replace whole files, whole directories for us, and we're gonna do some of that in just a second. In Chef, which again is my personal preference. It's the one that I'm basing the majority of this demo on. You've got client server or solo architecture. This is, the next thing is the real reason why I always choose Chef over Puppet is because order matters. In systems, in deployment, in packaging, in dependencies, the order of operations is really, really critical. It looks procedural. It reads top down. Puppet, you have to define the order separate of how you put it in the file. That's why I stay away from it. Underneath the covers there is a tool called OHI that actually gives you systems information. So if you run, if you have Chef installed and you run that command OHI, it will give you JSON output of what that system looks like. Now this is a very small piece that addresses my complaint from before, my Etsy release. I've got, this is Linux. I got the kernel version. What's the Linux standard base? What platform it is? It gives me a platform family. It's all red hats, so we're all in the same family. I even know now that I'm a virtual eye. I'm a guest, virtualization. I'm a virtual box guest. So that's always kind of a hard question, is am I a VM or am I a piece of hardware? And knowing now that I'm on virtual box as a guest, that all of a sudden I can say, well install virtual box tools. I can't do, I don't even know how to get that. I actually have no idea how to get that information out of a Linux machine. So OHI does that for us. Now Puppet, by the way, if you run OHI, the JSON file that spit out of my VM with two CPUs was over 2,000 lines long. So the amount of information you have, all the disks, the memory, the brother of the guy that put the chip on the whatever, it's all there. Puppet, it's a little more terse than that. Very, very similar. It uses something called Factor instead of OHI. The reason that I don't like Puppet, the number one reason is because when I go through in Chef, and I say okay, let's look at a more complicated, let's look at the web recipe here. I say package, HTTPD. Service, HTTPD enable, and file create me an index.html, right? I'm saying do those things. Chef is going to do them in that order, in the top-down order, which is what I would expect being a procedural bash scripter. That's what I would kind of expect. In Puppet, every single time it runs, it will give a different order unless you explicitly define that order later. And you do it, the syntax would be something, and again, not necessarily a Puppet syntax expert, but it's something like package. I also am not a spelling expert, apparently. HTTPD, then followed by service, HTTPD, et cetera. So you end up all of your manifests at the bottom, you have this long thing, or inside you'll have, you could say, the service HTTPD comes after the package, HTTPD. It gets very hard to read, very hard to maintain. These manifests can be thousands of lines long, trying to parse out if the order is correct can be complicated, right? So that's why I personally, now, by the way, I did tell you guys you're allowed to argue with me. Is anybody want to argue with me? Okay, we got one, there we go. I don't actually want to argue with you. Sure. It's really an ordering, it's a dependency on the order. But there is, with Puppet, and you can do this with Chef as well, you can say the service HTTPD depends on etsy, HTTPD.conf. So if that file changes, Puppet knows enough to restart Apache, right? Which is cool, until you accidentally make a change to HTTPD.conf and then your web servers all go down automatically, right? So we'll do that too, that's fun. I've done that by the way, just all the bad stories that you're about to hear are actual real life, Scott did this. So it's a fun exploration through life. But anyways, this is a slightly more advanced cookbook than our etsy, MOTD file. But again, I'm not saying do a yum install or do an apt get install or whatever. There is one thing buried here that I'm gonna get to which is package names. So I'll show you how we deal with actually, yeah, that's the next slide. So let's talk about it. Yeah, ah, yes. Yeah, so that's why this is important. So with OHI or with Factor, I have the ability in my manifests and in my cookbooks to actually say, what platform am I on? Am I on Windows, am I on Red Hat? Now, the nice thing about Chef is it gives me this concept of platformed family so I can say if it's Debian, if it's Debian-based, if it's a Red Hat-based, but Windows would be an option there as well. And I'll actually show you an example. One of the cookbooks that my colleague's been working on lately has where we say on Red Hat, you need free TDS-Devel on Ubuntu, it's something different. So I'll show you that. But this is basically the way around that. You get a very nice, easy to use array of, a hash basically of all the things that are going on in the system. That's basically that 2000 line JSON file exposed nicely in code. Puppet has something very similar. I don't like the syntax as much, frankly, but that's, again, just me. I'm not bashing Puppet, it's a great tool. I've seen it do amazing things and you get to feel like the master of all your child nodes, it's great. But it's, I don't like the syntax quite as much. So here, actually, this is the code example. Peter, I believe did this one. This is the OpenSSL question from before. Go ahead, I dare you. Now, I don't have to worry about doing this in a shell script. I can just say, when it's Red Hat or CentOS, or I could use the platform family, the package that you're gonna install is OpenSSL-Devel. Other, oops, keep smacking that remote that's in my pocket, me and my flappy arms. Or on Ubuntu, Debian-Base, it's libSSL-Dev, right? So it's now on Windows, just do nothing, right? So again, where do we come from, where are we going to? We're talking about operations people being able to define their job, do that day-long pile of stuff just to get a server to a developer in code. Maybe I take a week, maybe two weeks, maybe a month to write this thing. But when I'm done, I can do it 10,000 times. I can do it 100,000 times, and I can do it in seconds, right? So let's run that slightly more advanced example again slightly, oops, where do we go? Not at them, fire up. Okay, so again, what I wanna do here is in my base Vagrant VM that I pulled up, which is the basis of the base CentOS 6.7 server, I want to install Apache, I want to enable it, and I want to create an index page. And I really don't wanna type anything because I am that lazy sysadmin. So this is the other trick with Vagrant. You can SSH into your VM without actually knowing it's an IP address, very cool. But we'll become root. I'm gonna head to my cookbooks where I have all the ones that I set up for you guys. And this is the web. So I'm gonna say ChefClient, and this is called zero mode. That basically means don't try to talk to a server because it doesn't exist. And I want you to run the ChefWeb cookbook right now. So it's gonna fire up, and again, no config file found, we're just using command line options, fires up the client, and it found ChefWeb. It compiles the whole, it converges the whole thing. And it sees, I gave it package install, well, I can't even speak that fast. But I just told it, install HTTPD. Chef figured out we are on CentOS. We can use yum, I'm gonna do a yum install of HTTPD. It ran the service command. If we'd been on 7.1, it would've used system CTL to get the system d-side working. And it created the file for us. Now, that took me way longer to talk about than it actually happened, so it took 13 seconds. And if I pull up my browser, it's running in a VM so I had to tunnel it through, but there we go. That's my hello world. That's actually the web server that we just deployed. So I don't have to worry about any of the manual stuff, it's that easy. Let's take it to the next level, right? That's neat, but we're talking about serious stuff here. This seems really easy. What if we wanna use that, at this point we've built our server, we're the operations guy, we handed over to the developer. The developer has an application. We don't, I mean, I would like to develop index.html's all day, that would be fun. But we have a real app that we wanna deploy. How do we deploy quickly? My slides are a little new to me, so let me make sure I'm not too far ahead or behind them. No, I'm doing okay. Again, we've been defining what to do from an operations perspective. Now I've got a basic machine that's got the tools that the developers need to actually deploy the application. So let's go ahead and do that for a little bit. Let's deploy the core product or let's deploy our app. And how do we use Chef to do that? So what I've done is I spent a lot of time realizing that it was gonna take some time to get a big heavy app manifested out. So I picked something that we all know of, whether you love or not, which is PHPPG admin. PHPPG admin, we can run inside of HTTPD. It's not really hard to configure, not hard to deploy. It's a bunch of PHP files. But the idea is all I have to do is, again, tell Chef or Puppet what I want it to do. Don't worry about how it's gonna do it. What I want you to do and let it go. The new thing that we see here is this remote directory. Basically what I'm saying is there is a directory of stuff that's remote to the machine, the file system. It's somewhere else, it's not in that machine. And I want you to put it at var www.html.phppgadmin. And the source is PHPPG admin. Now we talked about how Chef and Puppet all have file management systems inside of them. The way that Chef works, one of the ways that Chef works is by keeping this right inside the cookbook itself. So if we look, it's very hard to see. Let me pull up a window. We'll go into Chef app, which is my cookbook. Of course, just rebuilt the Mac. I don't have the tree command in there. Right inside the cookbook, I have this location called files. Inside of there, I've got a little bit of a directory structure, but in files, default, I have PHPPG admin. It's extracted, it's ready to go. I could take, if I'm using a much more advanced application of a J2E application or something more complicated, I'm probably using bamboo, which is gonna do a build for me, then place it in an artifact and then Chef can pull it in through here. So basically, Chef is going to say, just deploy the app for me. So when we go up, back to where the cookbooks are, and I'm gonna say, okay, I want you to run the cookbook called Chef app, and this is actually gonna deploy PHPPG admin completely for us. Now this takes a little while because PHPPG admin has a good jillion files in it. The other thing here, as it, before it started to go, we had already installed HTTPD, and Chef didn't really mind that was okay. It actually, I had defined it again in a different cookbook, said, I need HTTPD, so I know by looking at my code that this application depends on HTTPD, the Chef isn't gonna throw a huge fit just because it's already there. It might update it, I didn't specify the version, so it'll bring it up to date when it runs it. But it says, hey, HTTPD is there, it's already installed, not a problem. So now you're not doing extra if installed, don't run any of those kind of wacky commands. Yep, yes, so you can specify the version number in all of this, so with the package there is a, believe it's HTTPD equals, and then the exact version number that you want to prevent it from doing the update. Yes, so the other thing about these recipes is that I'm using very simple things, but this is just Ruby, so this is a more complicated Hullabaloo that I wrote a while back. Just use variables, so I've defined a couple of variables at the top, and then I reuse them, I do string concatenation with the plus sign and all kinds of other things you can do later. It is code, so you can let it run time, figure out some more of those things without really having to bake it all in and hard code it. There's also a concept of attributes where you can define things outside the actual recipe itself, so it's more a configuration entry, it's more the advanced chef stuff, but it's definitely possible, easy to do. Any other questions? Absolutely, so in the next one that we're about to run, we have our default recipe, which is just calling, I want you to install Postgres, and then I want you to set up the DB, so I have two different recipes, so you can include others, and you can, if you realize that the differences between Ubuntu and CentOS are too great, you can, if CentOS include the CentOS recipe, if Ubuntu include the Ubuntu recipe, because it does get messy otherwise, so you can definitely compartmentalize them. Now this is all within the same cookbook, but it's very easy to go back one level and call other cookbooks as well. But again, while we were talking there, a bunch of stuff happened in the terminal window, and now, without really having to think too hard about it, PHP, PG, Admin, well, apparently I screwed something up, I missed a dependency somewhere. Yeah, I missed a dependency, oh, I know what I missed. So this is the one question that we had before, which was what, HTTP was already installed, and then I said install PHP, right? I need to restart the web server, but as a grumpy old sys admin, I don't like services to restart themselves, so I didn't let Chef go ahead and restart HTTBD for us, so all we're gonna do is now, now in our maintenance window, we've done the Chef, everything's there, I'm gonna restart it real quick, it comes back up, I do a refresh, and boom, PHP, PG, Admin is deployed. Your app here, right? So, again, this is a simplistic view of it, you can get really complicated, and I don't wanna try and teach everybody Ruby, cause I'm not really qualified to do that, first of all. But it's actually not that bad, if you don't think about the fact that it's Ruby and you're just defining something, it's kind of a definition language, I want this to happen, make it happen, make it so, said Captain Picard. All right, so, that's an application server. Now, when I was running this last night, I was actually, the other thing that I would do is, with my Vagrant, is think about, this is a web server, right, so this is running PHP, PG, Admin, if this were cool lickety-split.com, I might have 20 of them behind an elastic load balancer or an F5 or whatever I've got going on, right? It's a web server, probably very little state on that thing. I expect it to die, I expect it to have memory issues, I expect it to eventually go away, and that's what happens when I do a Vagrant destroy, Vagrant's going to basically tell virtual boxes, shut down the VM, and it nukes all the files that we just did, so everything we just did is completely gone, right, everything. But the real aha moment for me with Vagrant and Chef and any of these tools was that, uh-oh, but I just went through, if I were to stand back and think about it, this little piece took me about an hour and a half to get set up by the time I got all the dependencies right and everything, but if I go in, I'm just going to tell Vagrant, I want you to run the Chef app cookbook after you start up, say Vagrant up, and my cute little web server that we just blew away, all that work is completely gone and nuked into the ether of RM minus RF, it's going to come back because Vagrant's going to build the machine, it's going to call Chef, which is going to deploy the entire stack for me, and when I'm done, I can go to PHP, go directly to the PHP admin site, and that's it. So as a systems administrator, where it, I mean, like I said, I'm a bad systems administrator, which is why it took me an hour and a half to do that last night. Maybe it was the party as well, that could have been some of it, but I'm able to go from zero to hero in when I did my timings last night, like two minutes and 13 seconds from absolute base centOS, and the other thing that's important is I know by looking at what Vagrant's doing and by looking at this manifest, I know, oops, by looking at this manifest, or this cookbook, this recipe, I know exactly what happened, I know exactly what order it happened in, and I know it's right, and I know that it happened exactly the same way on every single server that I've deployed. There's always the mystical things in the life of a systems administrator where I don't know why that server's misbehaving, there must be something wrong with it. It's usually because you manually edit a config file, manually edit a config file, and there's 50 of them and you forgot two. It's very normal. So that's the beauty and power of this thing is that it just does all the work for you. I didn't have to write any shell scripts and away we go. Now, this is a Postgres conference, so I'm talking about apps that we can throw away. I say I'm a systems administrator, but I learned from, actually there's a couple of you in here that have taught me this over the years. I'm looking at you, Craig, specifically, how to be an extremely grumpy old man DBA, and that I am. So it's like, okay, great, your web server, we can throw it on the floor and be done with it. That's great, congratulations. I can't do that with my database. I care about my data, right? And even if I'm doing backups, if that database server were to go away, I can't just restore it in 10 seconds, even if I'm using the hottest EMC hardware or whatever I happen to have. They're not disposable. They're not like web servers. They really are, databases are. Now there's some cases, maybe a session database could be, or maybe QA dev regression, those kind of things. It kind of makes sense to be able to iterate hard and fast. But the other thing with databases is the configurations of these things are carefully tweaked. I protect Postgresql.conf, like it is my children, probably a little bit better than I protect my kids. They bump into walls and stuff. Postgres.conf doesn't do that. So, we gotta be careful. And this is where as a DBA and as DevOps, there's a line in DevOps. I like to always think of it as the line that we sometimes have to draw because I could go really crazy with this and I will in a second and I'll show you, hey, let's just go from absolute zero to installing Postgres, deploying the schema, getting the application hooked up, wired up, and bam, we're ready to go. But you gotta be careful. You really do have to draw a line. One of the things about DBAs, and this is an ongoing, I wouldn't say fight, but a friendly battle, is in Chef, in Puppet, in CF Engine, you have, it's a highly templatized configuration, ooh, five minutes left there. It's highly templatized. So, what I'm actually doing is defining Postgres configuration options inside of Chef. And again, back to our question before, if we're linking those services together, and if I pull up the cookbook that I'm actually thinking about, which is the publicly available one from Ops Code, this is their Postgres cookbook, works beautifully, you should use it. But if I search in that source tree for restart, and then I look at the change log, all of a sudden I see, hey, look it, we added a new feature. Restart Postgres service immediately on config change. So it knows when shared buffers is changed that that needs to be restarted, it's gonna do that for you. Line, wall, line, chains, all kinds of dogs, everything. So you gotta be careful with it. And some people will not let the Chef client run automatically or they'll stage their changes carefully. But a git push shouldn't restart my database is essentially what I'm getting at. So if I was Mr. DevOps in the corner here and I do a push and everything's automated and polling, I'm gonna restart my databases and they could be down for quite a while because of that. Now, let's hope that we can do this quickly. Of course, I can't type fast enough. Chef client minus Z, minus O Chef DB. Oops, that's not gonna work, that's gonna work. What this manifest actually does is go out to PGDG, it installs the repo file and then starts installing all the Postgres components for me automatically and then, oh, there it goes. When it's done, it's gonna load up a fun little schema that I did, which is an invoicing schema. So it's actually gonna deploy the database schema for me automatically just using a Psequel on the SQL file. So we should see that. The develop package likes to take a second. There we go. It fixed up some config files, starts the database, deploys the schema and now I can SU to Postgres. And I can go into my invoicing database. I've got my tables and my seed data and everything's there for me. That was from just a couple minutes ago. We had a blank, no, nothing machine, right? Here's the thing to be careful of and I know I'm getting close to my time. The reason that I'm skipping over some of this, here's the really scary thing as a DBA. I've just deployed a schema, that's great. A few weeks later, I go through and I see, where's my schema? Here's invoicing.sql. And I get down, we'll do a search for, I see that, oh, the primary key on the invoices table was disabled by, and Craig, I know you can't see it, but yeah, it's your username. The primary key on invoices was disabled for testing by Craig, I guess. And okay, I'm gonna re-enable it. Okay, and I'm gonna save it and I'm gonna do a get push and not think about it. Now next time we do a deployment, we're okay. Well, what happens when in 30 minutes Chef Client runs again? What's actually gonna take place? Let's see. Again, this is the danger. It's gonna run, it says, okay, all that stuff is already installed, it executed it. Everything looks fine, right? What could be wrong? Well, what happened is, if we go into Postgres, we go into invoicing, and let's take a look at the table. We don't have a primary key on it, so it ran. There's no primary key there. The primary key should be this invoice ID. So let's just say, okay, select the invoice ID and the number of instances of that same, which should be the primary key, it should be unique from invoices. And we will group by one, order by one. And what do we see? Now I've got, I've duplicated all of my data. Because we had in-source control, somebody commented something out, I re-commented it, didn't think that a get push at the end of my day, went home, now all of a sudden I've duplicated all my data and I still don't have the primary key there. And the automation system just did it happily for me. As a DBA, schema management still matters, right? So I've seen people do things, my favorite was the customer that was doing some automation where they were comparing these two tables, the public that invoices table. There's one difference here, and it's actually not a difference. All that happened is this customer ID field here is in position two, is an attribute, here it's in position four, right? The automation decided that in order to resolve that difference, the right thing to do was drop table invoices and then create new table. And it did it happily. And I remember that day because they had a little Christmas tree with lights that would change color based on Nagios, and it went red. So I'm pretty sure I'm out of time. This is another one, I just ran this as one, this is the same tool, and I'm not picking on the tool APG-Diff, it's an interesting tool to use, you should use it. Basically what it can do is compare two schemas and tell you what you need to do to go from A to B. This is a massive danger right here. If you look at this, it's going to, if we just pipe this into the database, it's gonna drop a constraint, it's gonna recreate the same constraint, and then it's gonna add a primary key to my customer's table. Now if that were production, and that customer's table were huge, I just shut the whole system down while that thing's being rewritten. So if that's a hundred gig, I gotta wait for bits and bytes to do their thing. So automation is awesome, automation is fun. When you're a DBA, you gotta draw the line somewhere. We still have developers, we still have operations. Automate what you can, it lets you focus on the business. Just everything in moderation and you'll do well. Thank you everybody, appreciate it.