 So welcome to Chef Cookbook Design Patterns, I'm Joshua Tumberman. Thanks for taking my talk in this time slot, I know we had a lot of choices today. So I went ahead with a one-hour talk recommendation that Mike had sent out, but I still have here 29 slides to go through, so let's get started. So who am I? Here are folks, enough said. Folks for IBM, with any business that did all the enterprise and business, Java, and XML and a lot of stuff, managing systems at IBM. Then I managed systems at Sands, where I operated infrastructure with Pobet, Capistrano, and a bunch of other internal tools. And then that brought me to HGK Solutions, who did systems administration consulting for startups. We did a lot of work with Pobet and Capistrano and Rougain Rails startups and that sort of thing. So HGK Solutions became Moscow when we published Chefs. And currently I write lots of cookbooks and I also do our training program and our consulting services and I also do evangelism and talking to conferences and that sort of thing. So who are you? Who's the developer? Like everybody, right? So who's the systems administrator? Who's actually a systems administrator? Who's the developer who does systems administration? And who wants to get rid of the systems administration books? Alright, yeah. Who are business people? If you're an independent consultant, you are a business person. So we're going to have a logo for Chef. That's kind of cool. So you can stop using like, you know, the Chef from the Muppets, the Chef from South Park. So let's have a show of hands. Again, who's using Chef Solo? Your people. Through Ranging Yard while you're using Chef Server. Using the Oxford platform as your Chef Server. We love you. Thank you. And we really love you. So if you're not familiar with Chef, it enables infrastructure as code, which is regularly since it's been created there. But this isn't a talk on how Chef works or you have to agree. I've got HIPAA talks and training classes. Basically, we don't have time for that in 30 minutes. So if you have any of those questions, direct them to any of those resources or just scream on Twitter and say, hey, how can I help you? So Chef Solo provides an MPC framework. That's pretty cool. So how does it do that? So the node attributes are the model. Information about the system or configuring is the data that we're going to do something with and then mold it in some way. The configured node is the view and the recipes are the control. And since Chef recipes are just Ruby, it gives us a lot of power and flexibility. And since Ruby is a programmer, we want to be able to do things properly and design them well because we're going to put those recipes into what we call cookbooks, which are packages of recipes and supporting code. So these cookbooks have applicable design patterns. Now I'm not a computer scientist, so if you are and I'm misusing design patterns, I apologize, and you have that discussion at the hack desk and that is definitely pretty cool. But these cookbooks represent best practices. And so when you apply good design patterns and then you build these into best practices, those cookbooks can be used by other people. But when it comes to best practices, there are opinions. There's opinionated software out there where everybody knows and loves and there's a lot of best practices encoded into the way that people build and manage systems. So it's a good idea to get those opinions out there and to share them with the world so that they can tell you how wrong you are on an internet forum and you can call them a troll. No, you're a troll. No, you're a towel. So we're going to talk about cookbooks. Oh, maybe not those cookbooks. Maybe they often cookbooks. We've got a lot of cookbooks, a lot of branches and cookbooks. Now who's for the cookbooks repository? There's only like four people here that have 300. Who's contributed a patch to the cookbook project? Contributed a license agreement. That's because we chose the imaginary license and we wrote a blog post about it and the important part is that we're not going to re-license your contributions. It's that you get to keep those, we just redistribute them for you. Contributed a license agreement is gay. People through our community side, we really appreciate our community and we want to provide a place for people to be able to share Chef Code and cookbooks and be able to make ratings and have this whole social networking thing going on. This is the newly redesigned cookbook site so you can share it there. You don't need a CLA. You don't even need to license yourself a patch. Though if you do post things on here, make sure that you have the rights to redistribute it because you put non-restributable code in the cookbook and somebody says, hey, off-code, I'm going to upload the MCA and it will say, okay, I'm going to take it down. So make sure that you can distribute the code that you're distributing and cookbooks are writing yourself and we also want to follow up and design that so that people can get lots of really awesome stars. So these cookbooks package up a configuration. They have a whole bunch of stuff in them besides just recipes. So let's talk about this stuff. The first thing here. Who saw this blog post? I'm talking about some more from GitHub. So you wrote this blog post on readme driven development and it talks about how the first thing you should do with a brand new project is write the readme. It helps you get clarity about the code that you're about to write and it helps you basically do some brief documentation about it. So really, write the name because your users of your cookbook will appreciate that. So after you print the written domain, it's time to write some recipes. So let's go through a nice contrived example. This year I went to Fosdom, which is a large conference in Europe. Has anybody heard of it? Do people? Has anybody gone? There's like 5,000 people there. It's the largest open source convention in the world and there's like 300 talks in two days. It's kind of intense. The talks are split up into these different devs and one of them was the GNU developer and I attended a talk on GNU Parallel, which is pretty cool. It replaces XRs and lies which are brought in a bunch of typing a bunch of tasks to Parallel and will just spawn them out. And so with our remaining 20 minutes I'm going to tell you all about GNU Parallel. Actually I'm going to tell you how I wrote a cookbook through the talk. But that's not because I'm awesome. Caps with laser hacks are awesome. It's also not because the talk was boring. I'm actually one of the right people anyway. So it's like just by any kind of GNU software you get a tar ball, you untar it, you go to the directory and you configure make install. Or you do a crew install Parallel. And then you rejoice that you're not. But it's not packaged by any of the major Linux distributions so I want to do that in the recipe. So we've got two resources, a remote file where you get that tar ball and the aft strip bit does the untar and building it. So that's not hard, but it's pretty bad investment. And it's not bad because it installs a source unless you're a unit sysadvent of the beard and it's negative. So really, why is it bad? The remote file resource itself has some problems. You can't download it from a different location. The source is the gnu.org. What if I want to mirror that on my internal server and distribute it from there? Or what if I want to have a different version? That was the version that was curbed that day but I'm sure that since it's active the development of the version has come out since then. And the other thing is the way that the Chef remote file resource works is it will download that file every single time to a temporary location compare the checksum to see if the target in slash TMP matches the checksum to that. So we're going to use some customizable attributes. So design pattern 1 used attributes. So in our cookbooks attributes default on our default we can define some same defaults. So I can say here's the URL where we do parallel will be from. Here's the version to use and Chef 256 checksums don't fit on slides. Also the slash TMP is in a great location because some systems turn on reboot. Not that you ever reboot your servers or develop a workstation or wherever else. But that would cause the file to be downloaded again and we can use an internal Chef configuration value called fileCachePath which is an hard to read, sorry. That's file, underscore, path, underscore, path. We can use that to download to a temporary kind of temporary cache location that Chef uses internal. So design pattern 2 exploits Chef internal values. So now we can rewrite our recipe for remote file using attributes and the internal values. So there's a little more a little more Ruby going on in there. We've got some string expansion and some attributes with hashes and that kind of scares this guy. He'll get over it. The master of tests and problems because, you know, the least of all is that it's compliant with the source. The version needs to be an attribute so we're going to reuse that. We can also... We might want to modify the way the configure options are running. So we might want to have some configure options. So we're going to use more attributes. So we're going to set up a default value for the configure attributes as an empty array and that same default can be easily changed by the user by modifying that attribute within their environment. So then we can have a little bit more cool stuff with the configure options so we're going to join that array and then we'll just pop that right there on the configured man and off we go. And this guy is a little less scared because the Ruby says that we're going to do this and he now knows how to modify that. Of course, he's going to make a package anyway so in an action file we can select a platform based on the node platform value and it shows that it's automatically detected when it runs and so this guy likes to use CentOS and so he's going to use created a package he's hosted it in his internal repository and he's going to have the... he's going to select the package value for that actually when it's on CentOS it's the source installation then the new parallel notebook that we've written has a recipe that will include a recipe based on the attribute so we'll just put that package or source will be a recipe that we'll just get inserted there for the recipe inclusion and then we have our package recipe that just says package, new, parallel and off we go. So yeah, the number of design pattern things I lost track of very quickly. As I mentioned, that's the sort where I can't even say it so I just the main thing is to use platform-specific conditionals so you can make your recipes and cookbooks usable on multiple platforms because everybody wants Bantu who doesn't want a Bantu? Who's using an NGR Cloud? If you're using an NGR Cloud you're not running a Bantu it's gen 2 I believe correct me if I'm wrong but not, fairly not for long but anyway, so we've double that, we've set up this cookbook and now we can run Chef and we can get a remote file and we can set the mode in the file and then we can run the script builds to be new parallel and I wrote all that in about 30 minutes because I really am like yeah, the place runs so here's a giant hover dot and we'll talk about recipes a little bit so we like to split up recipes by functionality generally when you're building Chef cookbooks you'll have a cookbook per type of service in your environment so later this afternoon we'll hear more about service-oriented architecture and how that stuff takes a plane the way you build applications I guess it's a good idea to split things up it makes it more modular and so when we write cookbooks we have a cookbook per thing that we're configured in so we have a Nagios cookbook and we have a MySQL cookbook but with each one of those we have several recipes based on the default thing that that cookbook should do like the default attachment cookbook should install a patch because that's kind of reasonable the default client cookbook MySQL should set up a system to be able to connect to a MySQL server and then of course the MySQL server cookbook would be setting up the actual server that's going to run in database each of these recipes as we've seen should avoid hard coding data besides node attributes Chef has some other rich features for not hard coding data and being able to express information about your infrastructure in arbitrary and cool ways one of those ways is data bags it's a server-side feature where the server stores some arbitrary data about something that you have written a JSON file out there and it will just make that available to be used in recipes and then Chef the key features of the Chef server is that it has a rich search API using solar and you can detect all these node attributes data and some other stuff so combining all these different ways to dynamically build configuration gives us the ability to have code reuse where we can publish cookbooks that dynamically build data and discover things about infrastructure and then we can share that with other people and then they can populate their Chef server or their Chef solo attributes with whatever data is required to build the recipes because you did write a review file that says what attributes you're going to use so all that said design method for the seven recipes so we're going to look at another contrived example where I'm going to switch over to our nodious example where we're going to have a few recipes so we have a default recipe which is what we generally use to install the common components for a particular thing so nodious has common stuff that will be installed on clients or servers then we've got some client setup and the way that we're going to do this is we're going to use the Chef server search to search for who the server is and nodious clients will be able to a lot of server to connect and to monitor and then the server itself will do a search for all the clients so all the nodes that are checked in it's going to monitor all those nodes and as they save their data then they become discoverable with the server so nodious basically we're going to install some packages and create a plugins directory that's got some various nodious plugins now a little danger here a lot of people are using Chef solo some of these features are only available in Chef server and this is one of the differentiating reasons to use the servers to enable the ability to do this kind of search and this central storage of your infrastructure data so here's the part of the plugin directory and the important part here and that red doesn't show up but at the top we've got a search we're going to search for the node that has a particular role and then we're going to add that node's IP address to this monhose value and then we're going to pass monhose into a template that's going to render out a config file that has the appropriate value set up in the nodious client NRPE would then have the monitoring server be able to connect and run NRPE and do monitoring checks then in our server recipe we are going to search second lineup there is going to search for all the nodes that have a hostname and then we're going to put those nodes into a host config file and then any new system that's added to the Chef server will then also automatically get monitored by the logging server so this makes it very easy to be able to implement your somebody else's cookbook that can use the search feature and you can run that in your environment with the logging servers that automatically build configurations but I use Chef Solo you say sorry but the server does give you the persistent node data the arbitrary search of data the search indexes and finally an API so if those are no other reasons for you and you still want to use Chef Solo you can check for Chef Solo or somebody else can check for Chef Solo using an internal Chef configuration value if we have Chef Solo and if so then if it's Solo then we're going to have a node attribute that says here's the node list for some attribute because my thing is going to have a node list otherwise we're going to search for the node list based on the role of my thing and that's arbitrary you can put that into your recipes in any way that makes sense for how it is you're deploying that recipe so hard coding data is an anti-pattern it's one of the key things about sharing customizable cookbooks with other people we encourage people to document the default attributes and populate the reading file that describes how that's put together we use roles heavily within the Chef community and within the way that we write cookbooks so let users override those attribute values based on the presence of that attribute in the role and we can also abstract some of that stuff out to the data bags and play nicely with Chef Solo so next we're going to talk about some templates and files these are the static assets the assets you can drop off by Chef and have them configure big files or drop off car files and that sort of thing they follow a file specificity and the reason for this is primarily to give us platform specific big files so for example my SQL has different big options based on its version and every single platform has slightly different versions of my SQL as packaged by default so you can split it up based on platform or even specific versions that might have changed to configure files so you just drop that in there and with Chef runs it will use the file specificity to pick the appropriate version of that template so that helps you be able to write a cookbook that people could use on other platforms then we've got static versus dynamic resources the cookbook file is static there's no dynamic building of the big files and the templates are ERV and they just drop off with whatever values generated and processed by ERV you of course want the template to be there because it's easily shareable, gives you a better way to do data driven configuration and you can use these various data sources such as data bags node attributes and search there's much other stuff that goes into cookbooks some of this is more advanced topics but know that it gives you all more flexibility and might even use to build your infrastructure so we'll take a look at these briefly first is libraries so libraries can give you recipe helpers library resource and provider helpers and new heavyweight resource and providers so if Chef doesn't have a resource for managing something you want to manage such as a React cluster you can write a React cluster heavyweight resource and provider and do all kinds of ruby code in those to be able to build that particular kind of resource the recipe helpers use the Chef recipe class the methods are available directly in recipes and for example in the library we have a class Chef recipe and then a method definition and this one the edge which is determined basically it was just returning the value of that attribute and then in the recipe we can just use if rate of edge then we're going to use this deploy resource else we could do something else library resource providers help you not repeat yourself and abstract API calls we have a cookbook called AWS that will manage EBS and Elastic IP resources on AWS nodes right now it uses right AWS and at some point soon we can convert it to Fog because Fog is even more flexible in batters and darts but we can use we can use that library use library in our cookbook to create a connection to EC2 and then we can use that within library resource provider in order to operate on EC2 API directly we can also do use library resource providers these are the full Ruby classes like those that are found in Chef itself they also allow some behaviors that are not allowed in the library resource providers such as inheriting and extending existing resource providers so if you wanted to have say a remote file that protrudes from that free bucket there isn't a provider in Chef to do that you could write that into a cookbook and also distribute those as gems such as Chef Deploy as was published by engineer so definitions don't use definitions anymore because they look like resources but they're actually replaced by the resources they contain and they're not as flexible and they don't behave quite the same like so instead we encourage people to use library resource providers this is a DSL for creating resources for libraries inside your cookbooks and the resources contain two things, actions and attributes those attributes can have validation parameters don't use these attributes with the node attributes context is important and naming is really difficult so there's a bunch of validation parameters you can go to the wiki page on this to read more about it I've got a bit of a leak there and I'm going to go to that slide so you don't have to get that so this is an example of resource an example of the resource DSL we have a list of actions this one just added one action and then we've got a bunch of parameters like the host, the user passwords, the database so we can create a database using a Chef resource then the resources are going to be providers and the library provider DSL defines the action methods and the Chef Recipe DSL is extended into the provider so you can use regular Chef resources in the action methods to short press things like being able to build templates so the provider action code, this is pretty good it's going to use an execute resource running command to create a database and this one's even better because it uses pure Ruby to talk to the database directly with the library we connected the database with and so the design pattern 47 is to use more Ruby the provider needs to be indebted so we don't try to create the database over and over and over again so we have a method in providers called load current resource we can use that to determine if the database already exists and then we can use that in a recipe to create a MySQL database it's just going to set up the database for my application now a word about metadata, cookbook metadata we need to find stuff about the cookbook, one of the important things here is that we have a dependency that declares another cookbook that we need to have in order to use this cookbook so in the group parallel we're going to build a resource so we need double essential we'll install compiler tools so we have that installed and we'll use it through the recipe and if we're going to use any other cookbooks libraries, library resource providers templates or anything else from other cookbooks you need to declare a dependency on it in the metadata using shell solo the metadata is not required because you have to ship all the cookbooks anyway because there's no server to distribute them and so I'm going to go through testing cookbooks really quick I'm not talking about BDD and TDD of cookbooks they are Ruby so you can use your favorite test framework to do that but what I'm talking about is uploading a cookbook to the server or sending the tar ball over and running the shell solo to the client or whatever you like just as long as you run shell client or shell solo in the system to test if the recipe is actually different that they're supposed to do so it's important to show your work we're a community, we're open source we want to share these with other people putting the effort into making it usable by others so you can get pushed your favorite GitHub repository or you can build the extra stuff and use Nike to communicate with the hospital community site to share with others and then the cookbook would be available for others to use these are some of the cookbook examples that are on the site they have a shortened URL ckdk.it slash name we'll go to that particular cookbook so it's easy to use in Twitter and that sort of thing and it doesn't look like we have time for questions but I'll be asking you in your practice tonight if you have any questions about Chef and we've got a bunch of resources and space for a class thank you