 Okay, I think we're going to get started again. Our next presentation is from Rahman. He's going to talk about creating a mature system with Puppet, as opposed to an immature Puppet system. Is everybody, okay, that sounds pretty decent. Hello, everybody. Welcome to my talk. I finished writing it about an hour ago, so it might be a little difficult. There is no Puppet for PowerPoint. So we're going to talk a little bit today about building a mature Puppet system, which I really think is more of an enjoyable Puppet system than anything else. Something that's not frustrating, something that lets you release code and not really have to worry about it. So I wanted to say thank you to Scale and Puppet Labs. Actually went to one of the early scales a long time ago, and then actually had lunch with Dan Bode at Scale 8, and that's actually how I got introduced to Puppet. So hopefully somebody is getting introduced to Puppet today. And I like Puppet because it lets me do my job and not get woken up in the middle of the night. I'm Ramain. I've been a system man for a long time. I worked at a couple of big companies, a couple of small companies, and I've been a Puppet user for just a little bit over three years now. And how we got involved, and I worked for Snap UTV. We're hiring like everybody else. Go check out the website, see if you want to work there. We like video engineers or people who do video codecs. And so Mature is really what I think is operable. Your team can use it. They can help you write Puppet code. You can help them write Puppet code. Other teams can use that Puppet code. It does generally what it's supposed to. You don't get surprised by it. There's not like, oh, somebody turned off the Puppet server. A code just went out and servers are starting to check in and take the whole thing out. Does anybody here actually managed to call us a site-wide outage, other than me? How many people have done it more than twice? So a couple of people copped to it. Either that or you're not using Puppet. Or you did it better than I did the first time. Flexible is a big thing. Probably a lot of you early on, like, oh, I've got this code. It only does one thing. Oh, you want a second thing? Hang on while I rewrite that. And four hours later, you're like, OK, now we can release. So that's also another big problem. Learning how to structure modules and how to write code makes it a lot easier once you get involved with it. And in full or really less frustrating of, you're not scared of the system. You're not like, OK, we're going to turn Puppet on again. I hope nothing goes down. That's kind of what we want to avoid. And how we're going to get there is a little bit of a process of we always do this. We always run tests. We always test it in stage. We promote it to production. Basically all the things where you don't have to think about it. Technique, we will do these things. We've figured these things out. We use data code separation. We're not constantly rolling code to make minor changes. Documentation, which nobody likes, boring, boring, boring. But it does help the other people on your team. If somebody gets paged in the middle of the night, they can go in and take a look at what you've been doing. Like, oh, OK, I get it. I can make a minor change and fix this problem. And finally, experimentation or the ability to have failure without taking your entire system down. And it really comes about testing or having a good dev environment that you feel confident in. It reflects your production environment, that sort of thing. So are you ready to write code? This tends to bite a lot of people. If you've been writing shell scripts, you make minor ad hoc changes. And what you don't realize is, with the puppet system, you are now writing code. You should really treat it like you're developing software, because you are. You really have all the same problems as the guys who write site code on your team. And you need to approach it that way instead of like, oh, I'm going to log into the server and make a couple of changes and get out. It's the set up your environment correctly so you're ready to write code. And all these things, this is part of the process to let you write code, have it check for you as you go without constantly having to reinvent the wheel or like, oh, I don't know what I'm doing anymore. So first, we're going to start with your editor. How many people will use Vim most often? All right. I don't know. It's almost everybody. Emacs? Anybody? All right. You're terrible people. Sublime, TextMate, anything like that? Which one? TextMate now, OK. I couldn't actually find any examples, so maybe you know some. But I just find some for Sublime. That generally seemed to be Vim was the most mature out of all the editors I looked at in the plugins that go with them. There was some Emacs. That seemed to be fairly close. Anybody using Eclipse? One. Gepetto, yeah. So if you use Eclipse, Gepetto is very nice. It's kind of an integrated environment. There's a lot of checking for you. But that's pretty much it. So syntax highlighting, there's a number of good ones for almost every editor out there. It will stop these problems from happening. Oh, I forgot that comma, missing curly brace. We've all got it, you know you have. Code snippets, actually, TextMate, stolen from them, ability of your writing puppet code, you type in file, curly brace, you hit tab, and it completes the rest of the stanza for you. It's actually very slick. You don't have to remember, like, oh, how do we normally do it? What order do we put stuff in? Makes your code very consistent. And you can go in like, OK, I remember source syntax. I remember content syntax. You don't have to constantly refer back to code you've already written. How many people are actually using puppet lint? That's it. Three, four, it looks like a four and a half. All right, puppet lint pretty much eliminates most of the previous problems. Because it will check for commas. It will tell you how to quote your variables. It just keeps you from making weird mistakes. It'll check your spacing. You don't have any hard tabs that you've put an insurer in that you've put a default in. Really, I would get to the point where you're just running it by default. Because it does help you enforce code style. Does anybody actually have a code style for your puppet system? Has anybody even talked about this internally? No. So what happens is this side of the room starts writing puppet code. And they decide they're going to do it one way. This half of the room writes it another way. Three months later, you guys get together. And nobody can reuse each other's code. Or you're confused. Or he starts checking in the way he likes it. You start checking in the way you like it. Try and settle on something early on. We always do it this way. We put an insurer at the top. We put stuff up, send the other. Puppet lint helps you enforce this. If you have anything that puppet lint doesn't support, it's pretty easy to write a plugin. Generally, it just makes your code more consistent and therefore more readable by more people on your team. How many people are actually the only person who writes puppet at your company? OK. So that was less than 10, which is actually kind of surprising. I was expecting more people to be kind of single system administrators. So this becomes actually very important for the rest of you because you will have to share code with the rest of your team. And how many people validate their puppet code before it goes to they check it in, push it to stage? Again, less than 10. You're all horrible, horrible people. This is, again, I was going to check a lot of your syntax errors this time. This is not just a style like, are you quoting the way we like to quote the stuff and the other. This will actually catch compilation errors before they make it out into a server. It does, it won't catch everything if you've misnamed a class that you're loading from inside this class. It won't go figure that out and find it. It doesn't work on templates. But again, it is a defense in depth of, are we checking our syntax? Are we checking style? Are we making sure that at least this PPE can compile? And this gets you 90% of the way there. I was going to put it in, but I was running out of time. So there is, I'll check it into the GitHub as an example, but there is like a little Ruby, something, something or other, compile my template. It's more of, does it actually run it through the ERB? I forget what it is. And it's not quite as cool as any of the other tools. Like it's sort of like a, so. I haven't been using it. Yeah. You're covered faster. So for the 90% of you that use VIM, really it's pretty easy to put all of this together. VIM, you install pathogen, which basically autoloads other plugins for you. And then install those plugins if you're done. So what this does is it gives you all the syntax highlighting. When you save a file, it'll automatically run puppet lint and then puppet parser validate on it. So all it does at the end picks out like which lines have errors. You have to go figure out what those errors are. But you're done. And you really have to stop thinking about it like, oh, did I run all this stuff? It just happens for you every time you save a file. So do that. So you've got pretty much most of those same options for other editors. Eclipse is probably the most mature next to VIM, if not better. Yeah. Yeah. Yeah. Right. Yeah. Thank you. What do you call it? Emacs is pretty good, but I didn't see all of that stuff. Sublime had some textmate type stuff, but I couldn't actually find a textmate one. So what are you going to do? Oh, OK. There we go again. All right. How many people test their puppet code before it goes to either stage or production on a VM? OK. Like 20. How many people are actually using Vagrant for this? OK, about half the same people. So they look pretty good. Does everybody know what Vagrant is? Very simply, Vagrant is a way to spin up virtual machines using a virtual box. And I believe they've recently added support for EC2, and they're going after VMware support. So basically, from your laptop, you're like, hey, spin me up a Debian host, type it in from the command line. It goes and hits the local virtual box installation or another box that you might have. Spins up the virtual machine that you want, and then you can pretty much run your puppet code from there. You can even kind of work the whole provisioning train into it of, OK, spin up a machine, run this code, let me know if there's any errors. OK, everything's cool. Get rid of it, which is pretty much kind of lost. All right, there we go. How many of you have run into the problem when you write a lot of puppet code? You test it, everything works. You write a little bit more, you test it, everything works. You write a little bit more, you test it, everything works. Then you push the whole thing to production, and you realize you've got errors because you didn't actually test the whole run. It's a very easy thing to do, like, OK, I install Apache. Great, OK, I know Apache is now on the machine. So OK, now I'll install WordPress. WordPress, you didn't add a dependency to Apache because it was already on the machine, so you didn't check. So now when you go to push to the new box, it doesn't have Apache or WordPress. It tries to install WordPress without Apache. Everything breaks. So basically, by testing from a fresh VM, you get the whole run, and then this way you don't have to worry that you have missing dependencies or you didn't order your modules correctly because you're really testing what the whole run will be. We do this fairly frequently at our company. I've just spin up a fresh machine, install a single host group on it, make sure everything was good, especially after we've made major changes in any of our core modules and stuff like that. Puppet environments, who uses puppet environments? Really, very, very little. I'm actually, I'm glad, it's a good talk then for everybody. Puppet environments basically allow you to use the same puppet master with different code for different clients. So in our case, we do it very, very simple. We have production and stage. So we push code to the stage directory, and then all the servers connect to it. Grab the stage code, run it. We see, OK, everything looks good. Then we promote that code to the master directory. Now it runs under the master environment. So any puppet server that uses the environment master now checks that code out, and everything's good. So it's a way of promoting code through your system. Just like how many people use a staging environment for your production code of whatever your company does? See, that's a lot more. You should be doing the same for your puppet code. Standard development practice, development. I usually do development right on my laptop. By the time I feel like it's ready, then it goes to stage, and then it goes to production. There are a couple of caveats. I don't think these have changed yet in puppet three. But puppets and facts tend to leak between the environments, so if you're doing a lot of type and provider testing, I really recommend having separate instances, either separate VMs, or just spin up an actual, separate, completely different puppet master process. Generally, I usually do it this way, I have my backup puppet master and my real puppet master, and then I just put stage on the backup one in that way, and then I kind of push to it if I know I'm going to be doing a lot of testing with providers and stuff that I don't want leaking onto the production code. And it's very easy to switch clients, it's really just a single line, environment stage, environment production. By default, puppet runs on the master under environment production, and by default, the agents connect to environment production. So really, you have to change something to get something different. I usually make it part of the template, so if you have stage in your name, then you always get the stage puppet environment. You can do something similar, you can hard set it, you can kind of set this on the fly, which we'll talk about a little bit later. There are a lot of options here, it's really just kind of what works better for your environment. The other thing to watch out is when you implement environments for the first time, you might need to rearrange things. Probably everybody here is, I bet, pushing directly into Etsy, puppet, modules manifest. You're not. Anybody else have things? All right, all the people already using environments have probably done this, maybe. So by default, you would end up just in puppet, so you're going to have to break your modules into stage and production, and then your regular files are going to be like puppet auth, puppet hera, whatever else you've got in there, so you'll probably need to rearrange your repo a little bit or change how you push. Generally, I don't push these files very often, so I have a separate repo just for modules manifest, and then I push those, and then when I want to push the puppet server, I'll push, actually, out of a different repo. How many of us, everybody, are syncing to their puppet master? What kind of push processes do we have? Subversion? But how do you actually get it on your puppet master? Are sync? OK, on the puppet master itself. OK, so you're doing like an export or? No, not an export. Generally, what I always tell people, if you don't have a push process, go find out what your developers are doing. Assuming they've got more of a clue than you do, steal their process, use the same one. You probably already have the tooling on all the machines. People know how to use it. If you've got questions, you can go ask a bunch of developers how they do it and all that fun stuff. Yeah. We just get, generally, I branch and push branches, but I kind of wanted to keep this a little agnostic for everybody. But basically, figure out a process that works. Make sure it's repeatable. Make sure you can revert to a version if you need to. That's really kind of the most important thing. Oh, I was just, examples, like that's an alias to some nonsense I wrote. Or whatever you might have, or sync fabric have strong examples. How many people are still on puppet two something? OK, so that's about half. How many have plugins think turned on that you know of? That's about a quarter of you. So if you're going to use a lot of the modules, like a puppet standard lib, puppet concat, anything with a provider type or fact in it, you're going to need plugins think turned on. I would just do it by default. It is actually now the default in puppet 3.0. Just go turn it on and you'll be fine most of the time. Test it first. Puppet Master, is it ready for production code? How many people are running their puppet behind Apache or some web server? OK, that's about what I expected, maybe 10, 15 people. So the problem with running up your Puppet Master by itself where you're just etsy and etsy puppet master do you start is it's single threaded and it only uses one core. And your concurrency is one. So if you get more than one server checking in at one time, the other server doesn't get a catalog. One server does, but one doesn't. You start running into weird problems. So what you do is you set up a Apache passenger in front of it. Apache gets a request that says, OK, you're for puppet. It says, do I have a rack process for you? No, I don't. OK, I'll spin one up. Or it says, oh, that one's already in use. I'll spin another one up and give it to you or I'll give you the one that's already there and so on. And it's basically a way to have concurrency on your Puppet Master. Please use passenger 3.0. I just put a thing on Ask Puppet Labs about it, that you should use it. There are official RPMs and devs available. Go install them. They're actually in one of the appendixes in the talk. You can go find them. It's much better at starting a rack processes. It's a little bit faster. But generally, it just doesn't leave. It's better it's spinning up and keeping track of processes. It's less likely to hang, especially if you have a lot of traffic, just go straight to 3.0. Now we need to tune it a little bit. Generally, I recommend if it's a dedicated puppet machine, set your max pull size for two times the number of cores. Set your minimum instances to the number of CPU cores. If all this machine does is puppet, you should just have a process per core there waiting for traffic. The amount of RAM you may have will limit this. Puppet master processes 10 o'clock in at 150 to sometimes as high as 300 meg, depending on what you're doing. So if you have a 16 core box, if you don't have at least eight gigs, you're going to probably run into some problems if you're running that many processes. So basically, I just recommend a lot of the Puppet Labs examples recommend putting your passenger settings in the V-hosts. Generally, if it's a dedicated puppet master, I don't recommend doing it. Just put in your passenger cough, tune it once, and then you can do some settings in the V-hosts. If you're sharing the machine with a puppet dashboard, you may want to kind of work on these a little bit. But overall, just stick it in the passenger cough and keep most of it out of your V-hosts. The big thing is with passenger 3.0, you get the passenger pre-start URL. Basically, you feed it the URL of your puppet server. Because what happens is when you restart Apache, it starts no rack processes by default. So you do not have a puppet master running at this point. So it waits for the first request to come in, and then it starts the rack process. The problem is the rack process takes somewhere between 30 and 60 seconds to start on most machines because it's got to go through, build the upcode cache, blah, blah, blah, connect to databases, whatever else. So it's almost a full minute until your server is ready to serve its first request. So by putting passenger pre-start in, which is again only available in passenger 3.0, you avoid this problem. And also, it'll spin up that many. So even if you set minimum instance, it waits for the first request to come in. So by adding this, you really end up with a lot better puppet master than you would without it. And then in passenger 4.0, there's going to be multiple Ruby instances. So if you want to test, OK, what's my master do on Ruby 187 versus my master on Ruby 193, you can actually do that within passenger without having to mess around with RBM and a lot of other craziness. So Apache tuning, you really don't need to do much. I run my puppet master under pre-fork, I mean under Worker. It really just works. I've done it on Ascentos and Ubuntu. Pre-fork, I would just avoid for this because Apache is really just a reverse proxy to passenger rack processes. So there's no need for pre-fork. It's just going to use more memory that you don't need. You might need to tune in a little bit if you're running a monster puppet server. I have a friend of mine who has 4,000 nodes on his single puppet master with 16 cores. So he has had to do a little bit of work on just how many connections he's coming in. But overall, it's unlikely you actually need to touch that much on your puppet master when it comes to Apache. So there are a couple of other ways to run your puppet master. InginX and passenger, InginX and Unicorn, Apache and Unicorn, Apache and Mongrel, just Mongrel by itself, passenger by itself. I have very little experience with any of these. Generally, it seems that most people on IRC that are running Unicorn have had more problems with it than passenger. Most of the docs from Puppet Labs are about Apache passenger. It seems to be well understood. And generally, people seem to get their problems fixed faster. The other thing is Puppet is a huge transaction. It's a one request, and then Puppet sits there compiling a catalog. There's really not that much chatter on the line, like, oh, give me this page, give me that page, give me an icon. It's really like switching out the app server. I don't think it's going to buy you that much, but I really haven't run any tests. I'd be kind of interested if anyone has. And worst case, if your entire company runs on Unicorn, InginX, and that's what you're comfortable with, then go ahead and run your Puppet master on it, just because you already have domain expertise there. So one thing I like to do is actually isolate the master. So what I do is I don't install on the Etsy Puppet for the master. I install just the agent by itself. Everything goes into Etsy Puppet. I leave that alone. Then I actually create some extra directories. I put it under an unprivileged user, and that way I know it's completely separate. Like, I can actually just copy that file up in R-sync it somewhere. I can back the whole thing up. And it just makes life better. So part of it is how many have run into the cert problems when you first set up your master? Like, you install it as an agent first, then you convert it over to a master. It was using the cert of the host name and the machine instead of whatever cert you wanted. Does this actually happen to anybody? OK, a couple of you. Yeah, don't do that. Don't set it to your host name. Hard set the cert name. And actually, we'll talk a little bit more about that in a minute. The other thing is, what do you call it? I've seen people on the email list in IRC. They've deleted all the certs on their master and then had to research everything. Has anybody here actually done that? Willing to admit to it. One guy. I'm not going to put him out though. He's over there. Because a lot of times, somebody's like, oh, I'm having cert problems. And the usual answer is, oh, delete this directory on the client. People kind of like, oh, OK, you're like, oh, nothing works now. Yes, yes, yes. So the other thing is, your Puppet Master actually always runs as an Unpredict user. Usually as the Puppet user, sometimes as the Puppet Master user, depending on the OS. And it can conflict with the agent, which is obviously running as root. Otherwise, it can't make any changes. So just by moving the whole thing over, having it separated makes everything a little bit easier to manage. Also, I like breaking out the modules. Like I have my installPuppet module. That's installs the agent. I have my installPuppetMaster, which includes Apache, Passenger, all this kind of stuff. They have nothing to do with each other. I do not share the PuppetConf. I don't have to do any complicated concat. Everything's done. So I don't have to worry about my database passwords making it into the PuppetConf on all the clients, which I've seen people do. And then I have to go remove them, change things. It's kind of ugly. And the nice thing is, when your Puppet Master actually uses a file called config.ru to get it to config. So you just tell it, OK, use that file. Otherwise, by default, it uses your EtsyPuppetConf. And you're done. So it's really the only thing you need to do. And then my PuppetMaster conflict looks like this. I just tell it, OK, you live in home. Some user. All these things have changed. That's where your Puppet code is. And you're pretty much done. So it's nice and isolated. How many people have not backed up their certs yet? We've got one. Go back up your certs. Ignore any of the reports in there. There's probably a lot of those. And it makes it pretty easy. The other nice thing is, once you get that figured out, if you want to build a local Puppet Master on your machine with RBM and a Ruby and just get it going, it's very easy to do. You can use all the same configs. And you have instant mini local Puppet Master on your machine. All right. How many people are monitoring their master? Come on. It's like six people. Terrible. So generally, HTTPS, port 8140, make sure you've got at least one rack process. You've got Puppet, so you can say, how many processes are on this machine? OK, make sure they don't have any rack process and make that part of your code. So log watch, make sure machines are actually able to check in or that you're not spinning out a bunch of errors. And you can request a catalog every so often, too. And just make sure that you're not getting a 400 or something on that. What do you call it? What's it? Oh, you mean how to request a catalog? I think it's curl, blah, blah, blah, blah, blah, blah. Yeah. That was actually surprisingly helpful. So how many people are actually monitoring clients? OK. Yeah. So basically, we do it by just checking the last run summary YAML. How many people knew this file existed? OK, about the same people monitoring their client. So it's easy to parse. It's in YAML. It tells you how many things changed, how many things didn't change, and if there were errors or if it couldn't do anything. Generally, load up Ruby, load up the YAML, jam, parse it, grab what you want, and then decide, OK, or critical. I put a simple check in my GitHub, so it'll make it into the GitHub repo of this talk, so if you want to grab it. It's based on Ari's original work. I just kind of simplified it because I didn't care that much about certain things. All right. Your Puppet Master is storing a lot of data you don't need after the first day. Every time a server checks in, it generates a report. Your server keeps that report. It can send it to the dashboard. Generally, I delete them after three hours if you want to keep them around for a couple of days in case your dashboard server goes away. You can if you use them and consume them at some point. Great. Otherwise, after six months with about 100 nodes, you'll be like, why is my server using 10 gigs in the bar directory? So just set up a cron to clean them out. How many people are actually using the dashboard? A couple people. The dashboard is really bad at clearing out nodes and reports. You must set up the prune crons to get rid of them. Otherwise, your MySQL database grows and grows and grows until it picks up all the room on your server. So I'm not sure what problems PuppetDB has yet, but I'm sure that a lot of them are similar. So look and see what they recommend for that. MySQL tuning. If you use the dashboard, this is actually kind of important because you're going to get a lot of stuff in there. The default MySQL conf is always complete crap. Just toss it away. Do at least these settings. And it'll generally work if you've got like 100 nodes. If you have less than that, you can probably get by with default settings as long as you're pruning fairly often. But if you have a system that's actually in use, go with at least this. All right, let's talk about certs. How many people manually generated their master cert? OK. Nice. So you can actually have a master cert. And then your clients obviously have their cert. And then I actually call it application cert, but really it's just kind of another client cert for the most part. But you generate one if you use the dashboard. Or you can generate one for your application to actually talk to your Puppet Master. I like multiple names in mine, especially if I know I'm going to need to move my Puppet Master at some point. I'm running 2.7 right now. I know I'm going to have a 3.1 master someday. So I just stuck a bunch of names in the client cert or in the master cert. So that way, I'm like, OK, I can point it to Puppet New. I don't have to play DNS games. I can just change things around. And it's not like, oh, I don't recognize this cert. It's the, OK, it's a different name for the same cert I've always been using. It makes things much simpler to move. Application cert. So this is actually when our system spins machines up, spins them down. And then we go expire the cert out of the master. And it's a simple core request to the REST API on the master. So you don't need to go in and dig certs out. And you go, OK, that server's gone. By the way, when you clean it up, get rid of the cert. Notice the client. So if you need to list, if you've got machines that don't have their cert yet, you can list those. You can clean up old ones. If you just want to start fresh on a client, you can just whack the whole SSL directory. That's actually usually the easiest way to deal with it. All right. So when you're testing, you don't want to kind of sit there and be restarting the daemon all the time, or maybe you do. But it's not really that useful. And also, when you provision a server, sometimes you're going to do like a bootstrap run or, OK, this server knows nothing about itself. It doesn't know its hostname or anything. So I need to put all that on the command line. I kind of invoke puppet these ways. If I just want to kick the agent off, if I want to tell it a particular server, also I'd plug in sync. If I want to override, like I have a roll fact that tells the server what it is, so I can just do that from the command line. If it's never been run, I'm like, OK, your database, connect to this server. Your cert name is this. Go make all those things happen and kind of keep that going. So it's great, especially for machines you're testing on, like, oh, now your roll is front end. Now your roll is back end. Now your roll is. And you don't have to reconfigure a machine each time. You can just pass these from the command line. Same kind of thing with a puppet apply, but it's more of a, you know, if you're testing code, you can just, OK, I wrote a new module. I don't want to, like, push it into Git and then release at this age. I just want to run it locally on my VM and see what it does. Go ahead, puppet apply. Sometimes it's called masterless puppets. Some people actually run their whole system this way. They push the entire puppet repo to a server, and they just tell it to apply it locally. It never actually checks into a central location. All right, now we're getting into modules. All right, how many people are actually using a puppet standard lib for anything? How many people have actually installed it for another reason other than some other module told you to install it? All right, so a little bit of half and half on that. Really, a puppet standard lib is a talk in its own right, and I believe there have been three or four of them. Go find them on the puppet camp website, and it does so much stuff. I'm just going to do one simple example. It provides this function is an integer, so you don't have to write it. And this is great if you are actually sharing code with lots of other teams where you need to validate some input because they didn't write it, so they don't know how it works. Standard lib provides a ton of this. This is like one of, I think, 40 different validation functions, so just go find it, read it, use it in your modules, and don't write the code yourself. All right, towards a better module. How many people have what I would call some God modules? How many, what do you think a God module is? So my favorite example is a WordPress module where it's like, okay, WordPress, okay, now we're going to install Apache, now we're going to install MySQL, now we're going to install PHP, and it's this monstrosity of installing like 40 different things on the system. Those things should be broken up into individual functionality. Puppet is like Legos, and a couple people have written that talk too, like little bits of functionality build the thing you want. Don't think of your servers as, this must be a web server. Think of it of, okay, this is a server with MyCount. This is a server with HTTP on it. This is a server with our site code. Build it up gradually, layers. Don't write giant modules just because they aren't flexible. You'll have, oh, okay, this isn't a WordPress server anymore, it's a MediaWiki server anymore. Now you have to write the whole thing when instead you could have just included Apache, included PHP, and you're done. So actually, I think a couple people are sparked on this today. Code versus data. The example I like is, okay, you've installed WordPress. It comes with data. The spam is already there. The blog posts are already there. You've got to get rid of all of it and put your own stuff in. WordPress doesn't ship that way. It ships with code, and then an empty database for the most part. And that's really what it buys us here is, if you write code that can take input, then when I give it to somebody, they can put their own code into it, like they care about these things. I don't care, like they want their own versions in there. I want my versions. If I hard code everything, then they got to go in and change it. They got to figure out all the places I referred to it. And then it makes it much simpler to do this, and if it's this role server, do that. So if you start seeing places where you're doing that, or if the name of the server contains a two, then do this. It's probably time to start splitting data out where you can query it. All right, and how do we get there? We get there with Hera. How much time do I have? Oh, I got like 30 more seconds. All right, I think we're gonna end with Hera, and then I'll take questions. So, Hera is basically a hierarchical data structure that you can query to get the data for your code. Most of us probably already think of our systems this way. You're like, okay, I've got databases, and I've got masters and slave databases, and I've got masters and slaves and some different colos, and you've got this hierarchy of how things work. And Hera actually really goes along well with that. And so actually, here's our hierarchy at work. So, fully qualified domain name is atop a hierarchy. If I wanna override something, that's where I'll do it. Like, you're this server, and I need to test this on you. I put it there. Otherwise, it's the, okay, you're a database server in production, you get these values. Or you're a database server, you get these values by default, and maybe I'll override those in stage and override those in prod. Role is actually a custom fact, in my case. So, Hera lets you build a hierarchy off custom facts, or any fact, really. So, if you want a deeper hierarchy, you gotta kind of write the facts, or adapt the existing facts to your system. I see a lot of people will add a location on here, like, okay, I've got stuff on the east coast, they get these settings. So, generally, location, role, environment, or what you see a lot of. So, here's actually, this is my dev machine where I build custom VMs for other things. So, I just set how I want things to run on it. Like, in production, Apache binds to everything, and my laptop, it just binds to localhost. I've got a local app build server, I use less MySQL RAM, because I don't have that much of my laptop. I'm with the latest factor version. I don't want to update Puppet, because that'll screw us up during my run, so I leave that alone. And then I'm doing a lot of Ruby testing, and that's the Ruby test, you know. And then in the case of Factor, I just asked for Hera factor version. By default, it's present, that's what the last one is, and then I just grab it here. So, in my case, because I said it's the latest, that's actually gonna override present. But if somebody else ran it, they would get present, unless they'd actually set something in their Hera file. And I think that's, oh, and these are the usual mistakes. People see Hera Array, and they say, oh, that must be for arrays. Let's put it in there. And they say Hera Hash. So what those do is Hera Array actually collects everything that matches across your entire hierarchy. So if you say, so everything that server matches for you, like, okay, it matches this role, this environment, it's gonna collect all of those. So it's actually great if you have monitoring servers in different zones, and you can collect all those monitoring servers, and then you can allow access to your patchy. However, most of the time, you just want first match, which is Hera. So that's really kind of the major mistake I see people make on IRC. So I think we're gonna stop there. When I get some bandwidth, I'll upload the talk, and you can see the last 30 slides. Yeah.