 All right, my name is David Danzalio. I'm a systems engineer here at Constant Contact. Man, so what kind of departments does Netflix have? They don't have QE or Ops, it sounds awful. I'm just kidding. So this is Seven Habits of Highly Effective Puppet Users. Real quick, how many people here use Puppet, know what Puppet is, you know, can Puppet? Okay, cool. And of you guys, how many people are dev Ops? All right. Okay, well, I'll explain why you're both kind of wrong. Wait a minute. So who am I? My name is David Danzalio. Like I said, I'm a systems engineer here at Constant Contact. I'm also a puppet evangelist, whatever that means. So I get to spread the good word of puppets. My background is sort of an interesting mix of, you know, healthcare, higher education, government, nuclear weapons. And I've worked for, you know, very small and very large companies. And now here I am at Constant Contact. So I've been using Puppet for a long time, sort of since the Dark Ages, as we like to refer to them. And I've been able to see a lot of puppet implementations that run the gauntlet from, you know, like, absolutely nuts to, you know, pretty good. And I've seen a lot more nuts implementations than good ones, so I've been able to see some really interesting things. And so this talk is sort of a collection of observations from my time with Puppet and working with different teams with Puppet. So I've been able to help a lot of teams integrate Puppet into their workflow, and it's really interesting to see, like, a team that's not using configuration management start to use it and start with Puppet. So I've put together this talk as sort of a collection of habits, seven of them, in fact, of particularly effective Puppet users. There's sort of this, like, class of Puppet users that you see, you know, wow, they're really good at it and they're really successful and they get to avoid all kinds of problems. So these are sort of the things that, the patterns that started to emerge when I started working with some of these teams. So, so a habit is an acquired behavior pattern regularly followed until it has become almost involuntary. I read that, I ripped that, like, right off of dictionary.com and I was like, what in the world does that mean? So I figured for, you know, if I'm gonna talk about habits, I should probably break it down a little bit. So the first thing to point out is habits are acquired. They're hard to break and it takes, like, a deliberate and conscious effort to change a bad behavior. You really can't, your Google Foo isn't really gonna help you out with a habit. Habits take a lot of work. Habits are consistent patterns. No one behavior really makes up a habit and said it's a collection of behaviors arranged consistently with a feedback loop. So habits reinforce themselves with good feedback but that also means that bad habits are really hard to get rid of. So, and habits are second nature as to be, you know, nearly involuntary. So there are things that you do without considering the alternatives, like reflexes and this stuff should kind of sound, it should sound familiar because you can sort of extrapolate it as automation for your brain. We do a lot of automation and, you know, Puppet is all about automation. So just think of it as, you know, you're automating your life, your brain. It's automating your wet wear, I guess here, refactoring. So the first habit is to think like a software developer. And this is sort of a theme throughout the presentation is that these effective Puppet users understand that their role has changed and they understand how it has changed. So the first thing you need to know is that newsflash, you're not a system administrator anymore. Puppet is a game changer and just like the assembly line really revolutionized manufacturing, Puppet fundamentally changes the practice of system administration. And this change is so great that you really can't consider yourself a system administrator anymore. You're a software developer. And I know this is a drag for some people out there, a lot of ops folks that I talk to, so, you know, I don't want to be a software developer. Too late. You know, it may not be what you signed up for, but Bob Dylan once said that times are a change in. It's a new world out there for system administrators and like the sooner that you embrace this fact the easier it'll be for you to adapt. So it's not your fault that you suddenly become a software developer, but, you know, tough cookies and now you have to figure out what it means to be a software developer. So system administration and software development are, you know, I think it goes without saying they're different skill sets, right? For years, the idea of software developers having root access to our systems was like what sysadmin nightmares were made out of and they really still are. So we, but we really expected system administrators to step into the role of a software developer, but you didn't really stop to think that maybe these sysadmins could use some training first. I don't know why we thought this would be easy. What do you guys think would happen if you took an airplane pilot and asked them to design a jet engine? I think like without polling I'd be pretty confident in saying that none of you would like to be the test pilot for that prototype. I wouldn't, but this is exactly what we did. We said, you know, here you guys have never done this before. So do it well. And we thought it would be easy because Puppet's a DSL, right? Domain-specific language. You know, Puppet's DSLs are easy to understand. They're powerful. They're quick to get up and running with. But it's also sort of one of Puppet's biggest liabilities. We thought that, you know, the DSL meant that system administrators could write good code without having to know what good code looked like that we had sort of abstracted all that stuff away. But we were wrong. Really wrong. We just made it easier to write bad code. The false sense of security that the DSL gave us really only made it easier to write awful code. And that's exactly what happened. So I think, you know, you can't really expect people to be good at something that they've never done. You know, this is, you know, business school 101. I really think it's a life 101, but I went to business school, so I figured I'd share some of that knowledge with you guys. So in this scene, Scotty tells Bones... Bones tells Scotty, you know, I'm a doctor, not an engineer. And Scotty replies, you're an engineer now. So I think this sort of adequately sums up the situation we're in. But the good news is that software engineering is a pretty mature discipline. People spend a lot of time trying to understand what good software looks like and how to write good software. So we can take advantage of this body of knowledge once we start to realize that puppet is just another language. The second habit, which is pretty closely related, is to treat puppet-like code. Because puppets are DSL, and because it's declarative at times, it's been hard to keep the focus on puppet-as code. It doesn't really look like other languages, so people have a hard time conceptualizing it in the same way. So stop thinking of puppet as just a collection of configuration data. Puppet's a tool to implement infrastructure's code, and I have to keep hammering it into people's heads that infrastructure-as-code is code. Code is in the name, if you had notice. But what does that really mean? What does it mean to treat infrastructure as code? Version control? You might think it's simple, but there are people in the industry in this field that I know and that I respect that still don't have their puppet code under version control. And it's not their fault. It's cultural, but there's really no excuse for it. It's just too valuable. So here's a collection of version control systems out there. I think I did a pretty good job at finding some of the more obscure ones, too. There's an endless supply of them. There's bound to be one that will satisfy even the most committed contrarians out there. So pick one and use it. Documentation. You're not the only person on the planet, and at some point someone else is going to need to read through your code. There should be just enough documentation to allow someone to step in and understand what your code is doing and what it was meant to do. And a lot of people think these version control and documentation, this isn't going to be a very good presentation. He's telling us stuff that we already do, but sometimes you sort of get in the echo chamber and sort of forget that maybe you're better than most people. I don't know, I've seen a lot of puppet code without documentation. So how many times have you tried to edit a script that you haven't touched in a couple of years and then you sort of forgot what it was doing and then you have to spend the next couple of hours reverse engineering it to figure out what it was doing and how to extend it? I mean, I've been there. I'm pretty sure a lot of people here have been. So documentation is not just selfless. It's really good for you, too. At the very least, your puppet module should have a readme explaining how to use your module and what it's doing. Inline documentation is great. Here's an example. This is an NTP module and it has 114 lines at the top of its public class, I have documentation at the top of its public class. This is fantastic. The more I have to scroll to see your real code, the better. Refactoring. Refactoring is a common software development practice where you regularly review your code and make small changes that don't change the overall behavior. This is something that maybe your average sysadmin who's starting to delve into puppet, it hasn't quite occurred to them. But remember that you're a software developer now, so you have to do this stuff. Code review. Get some sort of process in where your pull requests are inspected by a human before they're allowed to be merged into the project. They still need to be fully tested, but this allows for collaboration, increased awareness of the direction of the code. I think code review is actually a really great communication tool, too. Style. Style, I think style is particularly important. Adhering to a consistent style helps ensure that everybody understands what's happening in the code. This is especially important if you're developing in a distributed fashion, but a lot of times style takes a back seat to efficiency or urgency. So I urge you to consider the cost of sifting through all kinds of lint the next time you try and, you know, fix somebody else's, you know, crazy looking code. Puppet lint is a great tool that can analyze your code for compliance with the style guide. Now that we have one style guide and it's kept up to date and it is the official style guide. We can write tools around it. For a while, they were like three different style guides and they all conflicted. Third habit, stop, drop, and design. I think that one of the biggest habits for a highly effective puppet user is that they don't, like, just sit down and start writing code. They actually take some time to design it because software needs to be designed and I think puppet is no exception. This is what, you know, software without design ends up looking like. And then when somebody tries to extend it without having any reference to a design, it ends up looking like this. And then, you know, a few years later when you've left the company and you've left no clues about the design of the software, it looks like this. And if anybody in here spent any significant time in a data center, they'll appreciate that it's accurately labeled. Separation of concerns. This is, you know, a design principle that, you know, dictates separating code into distinct sections, each addressing, you know, a different concern. This results in modular code with well-defined interfaces. And it's been around since the 70s. It's become such a common practice in software design that most people don't even know the name of it nowadays. They just think of it as, you know, abstraction, right? Package file service was one of the first puppet patterns that emerged in the separation of concerns sphere. You'd break up your modules implementation into three classes. You know, one would install a package and another would place a configuration file and one would manage the service. And usually, they'd be orchestrated through a public class that would take your parameters and stuff like that. This pattern works great for small modules, but this pattern actually extends really well to larger and more complex modules as well. Then roles and profiles is a more recent iteration. It implements separation of concerns by introducing new layers of abstraction, a role and a profile, and allows you to construct a little bit more robust and organized code. Craig Dunn really came up with this idea in his blog post designing puppet roles and profiles. I think it's in the further reading section of the slides here. I'm not going to go into depth on it. I'll introduce the concept, but I highly recommend committing his blog post to memory. So a node includes one role and only one role, and a role describes what the server should look like from end to end. So here's an example here. I have two nodes and they all include just one role. A role includes one or more profiles. You can have as many profiles as you want in a role, but your environment can be broken into roles. I have worked with a lot of teams that say, I have too many snowflakes. I can't use roles and profiles because I can't identify roles. That's not true. Edge cases get their own roles. So roles are where you define your business logic. You'll see here I've got three roles. They all inherit the base class. Then I set up different database servers and stuff like that. Profile manages the modules. It implements the technology stack. So here's a MySQL profile. I'm pulling some data in from Hyra and I'm calling a MySQL server class, backing up my MySQL database, and then I've wrapped my Nagios checks in here. So typically in the old days, you pull down a module and the first thing you do is add stuff that was missing. If the MySQL module didn't have Nagios checks in it, you'd add a MySQL Nagios class. But then you'd realize, I can't take advantage of updates and stuff like that. So roles and profiles allows you to add functionality without having to modify the Forge modules and stuff like that. It's sort of like Model View Controller for Puppet. You should focus on identifying the interfaces to your modules. You should limit your entry points and keep your interfaces simple. I have a diagram here. This is an SSH module. This is a really simplistic and sort of a contrived example, but you can very easily see the interfaces and you can tell exactly where the data is coming from. There's very clear data flow through this module. We'll take a look at the code in just a second. Public and private classes. This is sort of a new concept in Puppet design. We sort of knew what public and private classes were, but it was sort of this nebulous concept that we didn't really put a name to. So identify which classes are public and private. Public classes end up looking a lot different from private classes. And public classes are where you define your API, your entry point for your module with your parameters. So public classes should validate your inputs and pass data to your private classes. That's really all they should do. So public classes should be the only way a user can interface with your private classes. And Puppet has, I think a couple months back, it still hasn't even been released, but there's a private function in the standard lib. So now you can explicitly mark your classes as private. I think that's fantastic. So here's the code for the diagram I showed you earlier. You know, I have four parameters. I'm validating the input to manage client and managed server. They're both Booleans. And then if I want to manage my client, I'm validating the client options hash and including the client in the same thing for the server. So this has well-defined interface, parameter validation, good separation of concerns, and implementation of public and private classes. So this allows for also good data separation. You want to understand how data flows through your module from your public to private classes. You should easily be able to pinpoint where the values from your parameters are coming from because data in your Puppet code is bad. Data in your Puppet code effectively makes your Puppet code a singleton. So, you know, embedding data in your modules hinders abstraction and code reuse. And by embedding things like package names and version numbers and URLs or host names, you're virtually guaranteeing that your Puppet code needs to be, it's going to need to be rewritten should your use case change, like even the slightest. Here, I've got an NTP class, and I'm implementing data separation by pulling my data into my params class, setting the defaults to my parameters with the values from the params class, and then passing it back to my private classes. Here's my private class. I've marked it with the private function. And then I've parameterized my private class, which, you know, there's sort of a debate should we parameterize private classes or not. I think it's great to parameterize it if you mark it as private. And then I'm just setting their values to their values in the NTP class. So, I know the values of those. I know exactly where they're coming from. They're coming from the NTP class, and at all times they will be coming from there because I've made it a private class. Stability is a key feature of enterprise software. And working in sort of the open-source world, this is sort of a forgotten design principle. You users want to know that your software is going to behave consistently and predictably, and your public code should definitely have these qualities because it's pretty important stuff. You want to identify a stable feature set and make sure you sort of, like, write it down. You want to track the features that you consider, you know, central to your module's functionality and make sure that they don't get accidentally removed. It sounds ridiculous, but it's actually pretty easy to, like, accidentally remove some core functionality. Semantic versioning helps users understand how your code has changed between versions. It's very easy to implement and helps ensure that there's sort of a method to the madness. You get a major, minor, and patch element to the version number. Increment your major version when you introduce breaking changes, so change to your API. Increment your minor version for new non-breaking functionality. And increment your patch for backwards-compatible bug fixes. Very simple. And that segues nicely into simplicity, which is another often forgotten design principle. Your public code should be easy to understand, easy to read, and you should do as little as possible in your module necessary to implement your desired functionality. Remember the Unix philosophy. Do one thing and do it really, really well. Focus on your module's core functionality. An example of the violation of this is, you know, I hate modules that include monitoring. You know, like, you pull down Forge modules all the time and they've got some Nagios class that's included by default, and I hate it because, you know, like, I guarantee that my implementation of Nagios looks nothing like your implementation of Nagios. And I don't want to have to rewrite your module because you thought it would be a good idea to include monitoring checks in your log rotate module. I just, it's not, it's not part of your core functionality. Leave it out. Hooks. I love hooks. Provide hooks to enable or disable functionality. An example would be, you know, a hook to install a vendor's package or something like that. So here, I wrote just like a quick snippet of a jabberty class. And I've got three hooks here. Manage package, manage repository, manage service. So, like, I can opt to manage any of those elements if I wanted. It can be very helpful to organizations that have their own internal repositories or for someone who wants to use their own package or for somebody who wants to use some non-standard process management system. Write modules if they're going to be open sourced. So this means designing them with somebody else in mind. You know, make them flexible, easy to use, and hard to abuse. Make them extendable. Habit number four, create a culture of testing. You know, Vagrant makes testing inexpensive. It's really revolutionized the way we develop software. If you aren't using Vagrant, you really need to start spinning up a Vagrant box to test your code is just so inexpensive. And it should be at the center of your development process. But for more intense testing regimes, the tooling around a testing puppet has improved dramatically. Just like in the last two years, it went from like non-existent to actually pretty robust. So there's puppet syntax. This is a gem that checks to make sure your puppet code templates and higher data are all valid. This really does nothing more than make sure that your code will parse. So don't fool yourself into thinking it's a very good test. It won't catch runtime errors. There's RSpecPuppet, which is a fantastic tool for writing unit tests for your puppet code and for doing test-driven or behavior-driven development with puppet. Here's a quick RSpecPuppet example. This just checks to make sure that your code compiles. I've got my class and it should compile with all dependencies. Super simple, really easy to do. Obviously, you should have a lot more than just it compiles, but Beaker. Beaker is an integration testing framework written by Puppet Labs. I think they originally wrote it to test puppet enterprise, but it just became so useful that people started using it for open source and then for all kinds of other stuff too. It used in conjunction with Vagrant or some other hypervisor or middleware. It'll spin up a virtual machine, run your actually execute your puppet code on it and make sure that the end state of the machine matches what you've defined. So this should be central to any sort of continuous deployment with Puppet. RSpecPuppet is pretty central to Beaker. It provides a DSL. So here I've got three test examples. I want to ensure that HTTPD is installed. I want to ensure that the service is enabled and running. I want to make sure that I'm listening on port 80. So this checks to make sure that your code to install Apache is actually working. Not that these resources are present in your catalog, that when you apply the code, the end state actually matches what you've defined in your model. So why test puppet? A lot of people ask, why should I test a declarative language? The answer really is because puppet modules increasingly take arguments. They increasingly contain logic and they increasingly interface with other modules. So make your tests count. Focus on the interface, not on the implementation. Here's an example of not a particularly useful test. This basically is just recreating your puppet code in the RSpec DSL. It doesn't really tell us anything meaningful about the SSHD config file. I'm just checking to make sure that insurance is set to present, owner is set to root, group is set to root, and mode is set to 644. What is that? I mean, that doesn't tell me anything about. This is absolutely useless. Here is something that's a little bit more useful. This tests my interface. This ensures that the value of the port parameter is valid and that it's actually in the file that gets placed by my puppet manifest. So I'm checking to see if I set port to 2222. If my SSHD config file actually contains the stanza that does that. This is really powerful. I mean, this is very important to make sure that your code is actually running. Here's an example that tests my logic, right? You can test for things that shouldn't be there. If I set manage package to false, I obviously shouldn't have a package in my manifest, right? This is helpful. Continuous integration and deployment. I'm not going to go in too much depth because we just got a great presentation from Netflix on this. This is probably the strongest indicator as to how effective a puppet user is. So continuous integration and deployment are pretty common agile practices. You want to integrate small amounts of code frequently instead of large amounts of code infrequently. At the very least, you want to do it for Ryan Gosling, right? So continuous integration reduces the amount of code in each integration, which ultimately reduces the risk that an integration will fail. It allows you to develop software faster and cheaper. But the first thing you do is to build a pipeline. Make sure that your deployment pipeline is the only way for a new module to make it into your environment. Automated testing. Turning your pipeline should require one touch. That should be a developer giving some sort of go signal. And it should kick off a fully automated test suite. Your test suite should be robust and rigorous in proportion to the importance of the code, right? Your test should be matrix builds. They should hinge on several axes. It'll be resource intensive, but it's important that your code works across your entire environment. So what test axes should we use? Puppet versions, right? If you have multiple puppet versions in your environment, you should be testing all of them. And you really should be testing the next release of Puppet, right? You want to make sure that you can upgrade your Puppet implementation by making sure that your code is compatible with the next release. Operating system versions. If you have a lot of different operating systems in your environment, you should be testing on each platform. And you should also be testing different point releases of those platforms. Ruby versions. I've seen some really strange behavior with Puppet. It's the same version of Puppet across different Ruby versions. This definitely makes sense. So if you have multiple Ruby versions in your environment, you should be testing your code under each Ruby version. Standard library versions. The standard library is so central to most puppet modules that you really want to be testing multiple versions of it. You should also test modules with the exact dependencies. Don't just use the fixtures file and pull down the latest because that may not be what matches your environment. Automated deployment. Once your code is succeeded, it should be ready to deploy. That sounds pretty simple, but a lot of people don't get that. You cannot for a zero-touch deployment or one-touch deployment here, whatever. But it should be automated. There are plenty of tools out there to help you do that. You know, these are just a few. There's library and puppet, R10K, M-collective, Capistrano. I could have written 20 slides on this alone. Just make sure that your deployment is automated and you can orchestrate it across multiple puppet masteries. Hebit 6 says, make nice with Ruby. I hate to dive into a religious war, but it's really important that you understand that Puppet is written in Ruby. And Puppet really is Ruby, right? I mean, and all the tooling around module development has grown up in the Ruby ecosystem. So as a result, you know, Puppet module development ends up looking a lot like Ruby development. And I recommend that you just get over it as quickly as possible. Cast aside any language dogma and dive into Ruby a bit. It'll make your life much easier and in the end it'll make your approach to Puppet code a lot more robust and informed. So, you know, like what tools are you gonna need to use? You're gonna need to use rake, you know, Ruby's build, you know, Ruby make, right? You're gonna need to use RSpec because RSpec implements the, RSpec Puppet implements the RSpec DSL, right? You're gonna need to use Bundler to handle all of these gem dependencies because you're gonna have quite a few later on. You know, Guard is actually a fantastic file system watcher that can run your tests automatically as you save your code. It's just fantastic. Custom types, providers and facts. You're gonna need to know Ruby to write custom types, providers and facts. I mean, you can write facts in, you know, Shell or whatever, but custom types and providers at the very least. You're not gonna be able to do these without knowing Ruby. The last habit is to get involved. Effective Puppet users share their knowledge with others and they get involved with the community, keep up to date and know where the community is going. That's because Puppet is evolving rapidly and, you know, getting involved with the community is not entirely selfless because, you know, Puppet's open source and very heavily community-driven. So you need to be involved in order to know the direction the community is moving in. And, you know, I've seen a lot of people fall behind on Puppet because they didn't have their ear to the ground. So definitely, you know, you can, you can get involved in a number of ways, but it's important to know when you go, when you try to get involved with the Puppet community is that your experience is valuable. No-to-Puppet implementations are alike. And, you know, Puppet is still really new. So we need contributions from everybody and, you know, from the whole spectrum of experience in order to identify, you know, design principles and best practices to help the language advance and to make it easier for the next guy. Share your modules. This is pretty simple. This is a great way to get involved. You know, put your modules on the forage or on GitHub. You know, let other people contribute to them. Let them do the work for you. That's always, that always sounds really nice for managers. Take advantage of the active, you know, the community that Puppet Labs has spent a lot of time and money to foster. Speak at conferences. You know, I think it's a good idea. There's Puppet Conference in San Francisco. I'll actually be giving this talk there on the 23rd. There'll be a lot more curse words in it though because it won't be at my employers. So, Puppet also hosts, you know, annual Puppet camps around, you know, around the world, really. I spoke at one about a month ago here in Boston. You know, and if you can't make it to any of these, find a user group. But, you know, if you just are a hermit, you can go online and find videos and slide decks from Puppet camps and Puppet comps. And finally, answer questions. There's ask.puppetlabs.com. There's the Puppet IRC channel, Stack Overflow, Google groups. There are plenty of people asking questions but not enough people answering them. So, dive in. So, on the topic of questions, do we have any? I know this is right before lunch, so everybody wants to go eat. I mean, I'm with you. All right. Cool. Thank you. Thank you.