 The next session will be about the two topics I like most personally. It's Drupal and Automation with Chef. So without any delay, cloudy with a chance of chickens. I give you a set code. Thank you. Thank you. All right. Let's get started here. We had a little technical difficulty. Get that going. Microsoft PowerPoint. Don't count on it. I don't know what I'm saying. He's here. He's here. He's here. He's here. He's here. He's here. He's here. Clearly that was the meatball service. And it had high availability. All right. So welcome to Drupal Chef Bork Bork Bork. So who am I? Well, I'm Seth Cohn everywhere on Drupal.org, Twitter, et cetera. I am an alpha geek. That's what my business cards say. For a day job, I am the technical lead at Bioraft. And I've been doing Drupal for about nine years since 4.6 days. I did tech editing book pro Drupal 7 for Windows Developers, which I'm very pleased is the only book that actually analyzed Drupal 7 in terms of the technical behind the scenes, how a page is actually built. There's a chapter in there that if you work with Drupal 7 and you really want to know what exactly happens in order to generate that page, it actually is documented all the way through so that you actually understand what happens to generate that page. And that's the piece of the book I'm most proud of is we went through with a fine tooth comb and really made sure it was accurate. I'm a former politician. I actually took two years off from Drupal and got elected to the New Hampshire State House and got open source, open data passed into law. So I'm one of those activist tech geeks. And I'm actually really proud of that. I think that's going to help change the world. And Drupal is definitely part of that. And finally, I'm a muppetty man. Or am I a manly muppet? I'm still not sure which. So what is BioRaft? Well, I just describe it as preventing the next zombie apocalypse. My marketing people aren't that happy with it, but they were okay with this slide. So it is built entirely using Drupal as the software as a service model, so we provide everything. We provide lab safety, compliance, and training software. We do things like track hazards, compliance management, training delivery. Basically, if you're in one of the institutions, quite a number of universities and some pharmaceutical companies that use our services, we at all enter into a lab. We track who you are, what kind of lab you're in, what kind of training you've had, et cetera. Make sure that the labs are safe. So we're talking about data and important real-world consequences if that machine goes down, if that data gets lost, et cetera. This is not a hobby site by any means. This has got real-world consequences, which is kind of why I joke about the next zombie apocalypse, because we've all seen those movies, and what is it that usually causes those things? It's some guy who goes ahead, reaches into a rat cage, gets a bit, and says, oh, it'll be okay. I'll go home. And next thing you know, half the world's gone, and zombies are everywhere. So as long as there has not been a zombie apocalypse, I'm doing my job. So what are we covering today? We are going to cover Chef. What is it? Why use it? We're going to cover the basic terms in what I call Drupal-ish. So basically, you currently speak Drupal. Chef uses a different language. My goal is for you to walk out of here understanding what Chef does in terms of how you understand Drupal. We're going to look at a very simple example, a LAMP stack. There are certainly a lot more complicated things that can be done, but in the interest of time, we're going to keep it fairly simple. And then finally, some basic issues and tips. And so we see there the most interesting Swedish man in the world who says, I don't always deploy lots of service, but then I do, I use a Chef. So why Chef? Well, it's right there, time and money. Chef is a way to build stable servers for development and production which save you time and money. And, well, what do we mean by that? Well, without Chef, right now, how do you build and maintain your existing servers? Who does it? Do you use package management? Do you use scripts? Do you build everything from source code, combinations? Who manages those scripts? When? What happens if there's an update? Security? OS level changes. Oh, well, we were using Debian 6 and now Debian 7's out so we're going to have to upgrade everything or Ubuntu released a set of patches that, yeah, they broke something, we have to roll things back. There's all kinds of things that come up. Do you use virtual machines? Do you use physical machines? Do you push stuff out to Amazon? You push something out to Amazon and it works great and then one day the server disappears and, of course, Amazon says, well, not our problem. We don't guarantee 100% availability. It's 99.99 and you just happen to be the one in 10,000 that disappeared today. So these are important questions if you're dealing with any kind of reliability. Personnel changes, that's the last thing, the bus factor. You've got a great person, maybe you're that person who's maintaining servers and if you got hit by a bus tomorrow what would happen to the service? What if you were not available for a week? What if you want to go on vacation? Wouldn't it be nice to know that your servers are going to stay up and you're not sitting there sitting with a pager? Do we all have pagers anymore? Does anybody have any? Is it all phones now at this point? Basically it would be nice to know you have reliability. Well, that's what Chef is about. That's what we're focusing on. So this is not another what is DevOps talk. There's a whole track. This is one answer. This is not the only answer. There's lots of other pieces to this puzzle, but my goal is that you walk out of here basically going, I'm going to go implement Chef because it is a piece of the puzzle that I'm going to use and I hope it fits for you. So what is Chef? It's from a company called Ops Code and it's about turning infrastructure into code because we all know that once you turn it into code you can control it. You can version control it. You can back it up. You can track your changes so that when somebody changes something and it breaks you can figure out exactly who and when that change happened. Most people don't have their servers under version control and so they go, yeah, I think we did something last week but it wasn't me. I think it was the other guy. This solves that because what we're going to do is we're going to take everything that is your infrastructure turn it into code and gain all the advantages that we already know code control is good for. So Ops Code describes it as configuration templates with business logic to describe infrastructure and establish relationships between boxes which is a fancy way of saying that it's not a simple script. There's a lot of complexity to this. Not only to describe it but it actually allows some interaction. So start slow, build up. That's one of the things. Chef has a lot of pieces. Chef has a steep learning curve. You don't need to do it all at once. If you add complexity slowly, Chef lets your servers work together. So you will start rolling things out and as you add new pieces you say I'm going to add logging and you can add logging and your existing servers basically if you've written your scripts right start to get logging as soon as that new server comes up. 15 minutes later they might recognize hey there's a logging server I need to go and I need to start sending my logs to them. You don't have to start with that. You can add the building blocks to it so don't feel like you have to solve everything with Chef to start with. So we know Drupal. Can we learn Chef? I think so and that's the goal of this talk. So we're not covering because there's a lot of things we could talk about. We're not going to cover Chef versus Puppet versus other tools. This is a Chef talk. I've been to Chef talks or somebody says well Puppet's another answer. I have decided to use Chef for a variety of reasons and that's what I'm going to focus on. If somebody wants to play with Puppet please feel free. I will tell you right now I think you're going to come back to Chef after you play with it. Vagrant. Vagrant's a great tool for spinning up virtual machines. I highly recommend it. Again it's beyond the scope of the talk. We could do a whole hour just on Vagrant. Berkshelf. Berkshelf sits on top of Chef. It's a way to speed up development. Once you understand the basics you will appreciate what Berkshelf does. There's some great videos on it. But again I'm trying to teach the basics so we're not, but definitely. And I've got it mentioned at the end. This is something you want to look at because what they did was they said well this is great. This is the way it normally works. Is there a way to do it faster and better and Berkshelf was the result. And there are Chef developers out there who thank God that Berkshelf exists because it speeds up. So instead of waiting 10 minutes you might wait 30 seconds to test your changes. We're not going to cover Chef Solo which is a way to do some of this stuff just locally on your own machine because my goal again is show you the basics. I want you to walk out of here understanding the core concept of Chef and Chef Solo sort of bypasses that. We're not going to cover Spice Weasel. That's too futuristic. Come back in a thousand years. We're not going to cover food critic and a lot of other nifty tools. This is Chef Add-ons. I definitely recommend looking at all of that stuff but again we're going to keep it really simple today. My goal is you walk out of here and say I want to know more about Chef. I'm not satisfied with what I learned. I want to learn more and there are resources. So cooking for geeks. Learning Chef begins by understanding the language. If you don't speak the lingo it's all Greek. Sorry it's all mock Swedish. So we all speak Drupalish. We're going to start with Drupal. So here's the Chef terms we're going to learn. You can see there's a little roadmap there. And we're going to come back to this list and hopefully by the time we have gone through these slides you will go okay I understand what they mean by this as opposed to what we mean. So how is Chef like Drupal? Well I have here a slide that shows the latest fork from Drupal. Paul heard a backdrop etc. This is Drupal which is the Swedish Chef fork of Drupal. My goal here was to basically show that we can look at something and even if you don't speak the language you can understand it in the same terms you're used to understanding. Because that actually is, I found a Firefox add-on that rewrites the site into mock Swedish. And so that's actually what that is. So how can we understand Chef concepts? Well let's compare it to what we understand for Drupal concepts. So Chef uses Ruby. So what? How many people here got started with Drupal and weren't necessarily wizards in PHP? There's a few. You can learn to use Chef without knowing Ruby. It helps. But if you five, ten minutes you will have enough Ruby knowledge to get started. You do not need to know Ruby in order to use Chef. Actually the most useful thing is one of the same things that it's useful to know for Drupal which is JSON formatting. And that's becoming more and more important with things like Drupal 8 because as YAML all of these things are very structured text documents and if you make a mistake in the configuration it doesn't work. So we have the core word. We all know the word the node. Everybody knows what it is. It's an item of content in Drupal, right? Well a node is not a node by any other name because in Chef node means something completely different. A node is a complete website. It's a complete server. A node is a virtual machine with whatever it's running and all of the various pieces. So that's the first thing is we're used to thinking of a node as a discrete item. Well it is a discrete item for Chef. It's a discrete server instance. When they talk about a workstation they mean your laptop. They mean whatever it is you do your work in. It's your local development environment. Chef server is the central control. Think of it as your own private Drupal.org. What goes up to the Chef server is code you consider to be either ready for production or it's your development environment if you're using some versioning and all of your nodes out there talk back to this machine and basically say, hey, do you have any updates for me? And that updater is the Chef client. That runs on each of them. So we're all used to the updater. We have that in Drupal. We know that you can go and run the updater in Drupal.org, there's a security update, et cetera. It's exactly the same concept. Each of your nodes talks back to your Chef server and says, hey, I'm server X. I need to know, is there anything waiting for me? And if there is, it gets the instructions and it basically downloads and does all the updates. So repository, that's again, we're familiar with that, it's code control. Now in the case of Chef, for the most part it's Git, there's a lot of development on GitHub. There is quite a community out there. Now this is my favorite, this is Knife. So we have the equivalent of Knife when it comes to Drupal. It's called Drush. If you're not using Drush, you should be. Drush is awesome. Drush is basically the command line tool that does everything that you'd need to do in Drupal. And Knife is exactly the same thing for Chef. Knife is the way you can do 95% or maybe 99% of the stuff you're ever going to do with Chef. You're going to do it from a command line. You're going to do a Knife command and you're going to say, I need to upload some new information. I need to check on the status of a server. Knife does all of that. So the metaphor in Chef is they use this cooking metaphor. Knife, well, cookbook. Well, what's a cookbook? Well, in Drupal terms it's a feature. So in Drupal we have created a packaging system for not just modules but configuration and all the pieces so that if you go get a feature, you can add it to your website and you know that that complete package comes down and works, sets up all the pieces that are needed. Well, that's what a cookbook is. And there's some overlap because installation profiles and distributions and features are all not quite the same. Cookbooks cover all three of those. A recipe is like a module. So the usual syntax that Chef uses is cookbook, colon, colon recipe. So bacon, colon, colon, crispy would refer to a file in the repository that you've got under cookbooks, slash bacon, slash recipes, slash crispy dot rb, rb being a Ruby file. But if you think of a recipe like a module, you'll be offset. That's really what it is. So what do we say right now? There's a module for that? Well the answer is there's a recipe for that. You can go looking and you will find there's tons of recipes out there for just about every server task you want. And it may not be perfect, but just like with modules, you can customize them as you need to. And you can write your own to do all the things that you need to do. So a resource is an action. Anything you need to do on the server, I need to add a file, I need to add a user, I need to change a permission, I need to do whatever it is you need to do. Any of those things is termed a resource. Basically resource collections are recipe. Recipe collections are cookbook. Now if they'd set ingredient, that might have fit the model, but I think they just said resource, it's resource. So a data bag is configuration storage. The closest we have in Drupal might be variables. The idea is these are things that are stored potentially for use by multiple machines. It has fields you can set, mostly done in JSON. So it's a little bit configuration. You can encrypt them. So for instance, you might store your user list. So you might have a list of people who are the users that you would be allowed on a particular machine. And you would have a list of these users and potentially SSH keys, you might have passwords, you might have all of this stuff stored in one place. And every one of those machines would have access to this depending on how you set it up. So templates are how configurations get filled out and we have the same thing we call it tokens. The idea of Drupal tokens is that we go ahead and say here's the token variables and drop them into a form and you get the page. Well, in this case we're going to take tokens and we're going to use it to fill in the blanks on server config. So you might have three different web servers and the configs are almost identical except for a couple of small changes and you would just use basic templates so that way you'd have one version and just fill in those changes on a server-by-server basis. So config files are like a .info file. That looks a lot like the standard module .info file and if you look at it we have dependencies and we have versioning. We have some basic stuff that goes ahead and describes what the thing is and who's responsible for it and that's basically what it does. In this case is a Drupal recipe and we're going to come back and look at this a little later. Actually, we'll mention it later but you can see right there on that line it says depends and it basically loops through and says this has got dependencies on PHP and Apache 2 and MySQL and open SSL. So Chef will know that in order to run Drupal you need to have these resources installed first so it's smart. It knows that you can't build Drupal unless you have all those pieces. Now you might say well I don't use MySQL we're building Drupal sitting on top of MongoDB. Well great, so you make a recipe that has a MongoDB dependency instead and there's your customization. So libraries are raw code. You will occasionally find that a cookbook has some raw Ruby functions in there occasionally something else. The idea being sometimes you just need to get that level of access. You need to be able to run something directly in basic language as opposed to through Chef's things. Searching, well searching is solar. Everybody's here used Apache solar? I mean basically Drupal decided that that was the way to go. So again nothing you're not familiar with. It's nice when you use some of the same tools. So roles, roles are server identities. So for instance we say these servers over here are web servers and these servers over here are database servers and that's a logging server and you can then identify these things as groups so you can say hey web servers go find out who the logging server is and send your logs over there. And you never need to specify IP addresses or names or anything else and Chef will essentially take care of all of that for you and will make the connections as needed. Again this allows you to build things slowly and also have a lot of redundancy because you can say go send stuff to the logging servers and maybe you have two or three of them and one goes down and the other is there and 15 minutes after that machine goes down the other servers get updated and they go oh well there's only two machines now I'll remove that other one and therefore they just keep running and they don't try to send logs to some non-existent machine because that machine was taken down for maintenance. So this way these servers are never going to give you up they're never going to let you down they're never going to run around and desert you oh I'm sorry it seems this presentation has been Rick rolled Well that sure made us LOL Laugh out loud No lose our lunch I'm even bringing my own hecklers there we go So attributes are properties so now we're getting to some of the JSON formatted stuff this should look pretty familiar and I'm not going to go over all the details here but the basic concept is you have a run list that says what recipes to run so in this case that configuration for a machine says you're going to run Drupal remember we talked that Drupal has dependencies so it knows that if it's going to run Drupal it's going to have these other pieces in place it has, you can see there a Drupal DB password setting which is a random string one of the nice things about Chef is for the most part the recipes are written so that if you don't specify a setting it will generate a random one for you so if you tell it to build a Chef instance of say Drupal and you haven't pre-told it what the password is supposed to be for the database it will make one up and it will start in a config file and you don't have to go figure it out you just have to open the config file up because it's right there now in the case of the next thing down the MySQL settings I actually specified those just to show that you can put them in so in this case when I built that config I actually said these are the root password and Debian passwords I want you to use for this particular config because I put them in there when Chef goes to set up MySQL create a random one I'll use the existing one and then there's a name and then there's environment Chef does have environments I didn't really touch on that here but essentially you can do things like build the same server in different environments you can say build this machine this way if it's my production server but build it this way if it's set as the staging environment or the development environment and that way you have one set of recipes that can potentially behave differently depending on how you're using them so that way maybe your development environment ignores certain errors because it just doesn't have the production pieces it's looking for so that way you don't get all kinds of warning messages and you go oh yeah just ignore that I need to fix my script you can just make your script work the first time that it pays attention to what the environment is so node workstation, Chef server client, repository, knife, cookbook recipe we talked about configs and attributes policies are basically all of those things it's all your settings resources, data bag, templates libraries and roles so if you look at that thing now you now understand the basics and you can see they're really not that much different I mean there's a little bit of difference but there is certain similarities it is a very structured system for building servers so let's take a practical example you are tasked with setting up Chef to build a new staging server for your team how do you start we're going to keep it really simple we're going to say the requirements for the new server are just the lamp stack that's it that's the only thing we care about we don't care about a firewall, we don't care about log we're not setting any of that up somebody says hey I need you to spin up a new lamp stack for me how many people have had problems with that setup alright so bare metal and this is one of our goals of Chef is that essentially you can through having your configuration and code your data backed up so whether that's your MySQL dump or however you're storing your data some kind of backup and having your code like Drupal all in repositories you could essentially bring up a brand new machine from bare metal and have it be exactly the same every time so how will it be provisioned well it could be a physical box it could be a virtual box maybe Amazon web server you've got a variety of OS choices you'll pick one in this case let's say we're going to build it on Ubuntu and we're going to say that it's a virtual box doesn't really matter next step is to get Chef server now you can set up Chef server on your own it's open sourced ops code provides a free trial of their hosted solution called enterprise chef to get your feet wet that's a great way to get started I recommend it you don't need to set anything up you just go you say hey give me a free account they say sure give us your email and you're ready to go and then we'll walk you through installing Chef on your workstation and essentially that one line which is it just goes and gets a script and runs it that's the magic in some way shape or form that script or what that script does runs on every single machine that's part of what you're doing that script will install all of the Chef tools all of the Chef clients and servers everything you need to make your machines talk to each other so once you run that it basically installs all the pieces including Ruby the whole works it basically looks at your system and says what do I need to make this functional and it does it so they've got the magic down to one line if you basically memorize that line you can sit down at any machine you need to and boom you've got a Chef workstation which is great when your laptop decides to act up and you need to grab a friend's laptop so now you have to decide on which cookbooks you're going to install and so you can go to among other places the community site for ops code where they have essentially their modules list if you think about it that way features list they have hundreds if not thousands of recipes cookbooks for all kinds of services and machines and programming languages et cetera so it's just like going to drupal.org you basically say oh I need this and I heard really good things about that I want to try that out so in this case we're going to say okay knife I know what cookbooks I want so I type to command knife cookbooks site install Apache 2 MySQL PHP and knife says oh okay so you're telling me go to the main repository site and go grab the Apache 2 cookbook the MySQL cookbook and the PHP cookbook and by the way any dependencies that it has so there's a couple of small dependencies that some of those want and it will turn for a little bit it will go download stuff and you will end up with some Git repository files each of those and you can at that point commit them into your local code repository and you're all set but we need to send that to the Chef server that's our own private server in this case it's hosted but they don't put anything into it you don't have to worry about oh they have a version of this or a version of that they give you the blank slate they just say it's empty you put into it what you want so you don't have to worry about oh they have an obsolete version there's nothing in it the only way that things get in is because you put them in so we say knife cookbook upload all and it goes boom boom boom and it sends it all to the Chef server and voila just like in the matrix Chef knows kung fu so now we tell Chef to control our new server that's the virtual machine we said we were going to set up so the only thing that's on that machine is we basically format our virtual machine we said install in buntu because that's what we're going to use so whatever you're used to using maybe you can use windows whatever it is we basically just do a knife command from our workstation and we say knife bootstrap the IP of the server and then we give it an SSH user and we tell it to sudo so it's got some root access so it can install packages and we say okay here's what I want you to run go install mysql apache 2 and php that one command ssh is over to your new virtual box it says your chef on here we didn't put chef on it so it goes and it runs that nice script it goes and it grabs all the chef stuff puts that on and then says oh I've been told to put mysql apache 2 and php onto it it looks at those cookbooks and it says oh there's some dependencies there and it goes and it basically will bootstrap the entire thing and when it finishes and depending on the speed of your machines and the internet it might take 30 seconds it might take 10 minutes depends on how complicated a recipe you do basically when it finishes you have a lamp server you're done you don't need to do anything else you might not want to tweak some configs etc because that was a very simple recipe we didn't customize it at all but basically with what did we do three commands go grab some stuff go push some stuff up to the server go bootstrap a now if we wanted to do a second machine you don't have to do those first two steps they're done you just go and put another IP address in there and you're done there's another machine and it's identical to the first machine except maybe it has a different IP address you can customize it again but the goal here is replicability and so it's really simple to do it so alright but we didn't put Drupal on let's bring it back to Drupal so let's use a Drupal cookbook that we looked at earlier so it has the dependencies for MySQL, PHP, and Apache 2 so the only line we would have had to have done is cookbook site install Drupal because that recipe knew that we needed all the other things so that has built into it the fact that you need a lamp stack so again don't even have to go provision anything somebody says hey go spin me up another Drupal server there's your command and then you run your bootstrap command only in this place you go recipe Drupal and you wait for it to churn and you go okay new machine here's the IP address go ahead starting up servers has never been easier especially when they're virtual machines because you can basically go ahead and spin up as many as you need as quick as possible so let's talk a little bit about configuration pieces things to put in data bags these are things you don't want to put in a recipe think of recipes as sort of public I mean you can still have private recipes but if you get an Apache recipe you might say I need to set up certain Apache users or MySQL users or whatever the case may be so things like SSL certificates, passwords users these are things that are private data you put those into a data bag they get stored encrypted if you want them to be on your Chef server and so you do have a little bit of an issue there because you still have to control who has access to your Chef server so there are still some security issues you don't necessarily want everybody who has access to your Chef server for instance to be able to add a new user to the system so that's one caveat you want to make sure that you're paying attention to so use Ubuntu if you can I wish somebody had told me this sooner I started using Chef and I've been a Debian fan for years I've used Ubuntu I like it but I was like oh well Debian is really solid long cycle and I usually know I can trust the packages are absolutely solid and the problem is it's the lack of eyeballs you end up with a lot of scripts a lot of recipes that only work on things that people have tested it on well if you want maximum compatibility use Ubuntu use the long term the LTS images I highly recommend it I think in the long run just in terms of comfort to know that there's lots of other people out there using it you're going to be the happiest but to be clear Chef will run on anything it will control Windows boxes if you've got Windows boxes you want to be able to reliably bring up the same Windows box every single time building it from a static image every flavor of Unix OSX if you're running for some reason Mac servers you obviously have got for some reason a lot of money to spend but that's great you know if that's what you want I'm not sure why you wouldn't be using Microsoft at that point if you had a lot of money this is a classic Drupalism don't hack core same lesson applies if you're downloading a cookbook yes there are times you might change it but if you change it you're making it harder on yourself because when that cookbook gets updated you're going to have to figure out the changes build on top build custom things that sit on another layer so take the standard cookbook and then build your own custom little recipes that sit on top of that and run those recipes don't go hacking the standard cookbooks you will be glad you didn't do that for the same reason you're glad you don't have core because basically you're killing kittens and contribute your patches back it's really nice I have one I've got to give back where I discovered that for some reason I was using a user creation cookbook and it allowed me to set a password which was great so I could set up a default password for the user so I'd spin up a server and they could log in because they knew what their default password was and everybody had a different default password the problem was is that Chef Client runs regularly usually you put it into a cron job and you just tell it to run every so often let's say every 15 minutes so that way it keeps checking well unfortunately what would happen was every 15 minutes it would check and if they had logged in and changed their password it would go change their password back it would basically force their password back to the default and I said I don't want that behavior I want to set a default password and make it so that when they change it you leave it alone and I found a way to do it and I just haven't had a chance to submit that patch back but that's not my to-do list is to add that back in it's a great example where how many other people might find that useful quite a lot so one cookbook to roll them all it's really tempting to build separate cookbooks for all your pieces we're all used to compartmentalizing and trying to make sure that you keep your project separate I did that and then I was kind of sorry I did because the way servers tend to work is there's a lot of interdependency and you end up with this really nasty web of dependencies and what I found was easier was for instance firewalls I had a custom firewall cookbook and what I realized was that I don't have one firewall I have a lot of firewalls I have a firewall setting depending on whether I'm running a particular service so really what I should have is I should have a in with what sets that service up the firewall instructions and that way I spread it apart so the only time those firewall instructions go into a box is if I'm putting the important pieces like the service that I'm firewalling off because of that your best bet is to start from I'm going to build a custom cookbook and put all your recipes into that and don't break it apart at least to begin with it's really nice to be able to say hey I'm going to contribute stuff back and I've got this one cookbook I want to do build yourself a cookbook that's got everything and then at some point go these three or four things are really worth sharing and put that out but don't split it up at first I think you'll be sorry because it gets way more complicated it's not like building something that doesn't have dependencies when you start having five or six servers that have interdependencies and slight different changes it's really nice to know that there's one place to go look that you're not sitting there going which cookbook did I put that in so that brings me to the next point which is record your changes and make sure your coworkers record their changes because there's nothing worse than having a worker who says oh yeah there was a problem with such and such and I went in and I tweaked a config file and you're waiting and they don't say anything you go what config file did you tweak oh I don't remember well you can figure it out it'll be okay well if you really are careful about documenting your changes you will put them back into your recipes because the goal of Chef is every change that happens should end up in a recipe in fact your production machines you should never have to even log into ideally because every single change that happens should be under version control should be in a recipe should be something that you can document and say when it happened and when it was when it took effect and if you log into a particular box and you tweak a config setting it might be perfectly fine and then three weeks later you change something and Chef client runs and oh wait that configuration changed and you may not even remember what tweak you made and now you're back to square one so document your changes Chef works if you work it if you break the habit of one off fixing your servers you will be much better off so questions are there any questions I had a brief test of the server and I was wondering how open source it is because when I tried to download it wasn't so easy I didn't find a github repository I know you can test and use the online enterprise version but I couldn't find a github repository I had to fill out a form and then I got a terrible to download it just made me wonder it's actually very open source and I'm not sure if you can understand the intentions they have done for hosting and they freely admit that that's their business model is if you are big enough that you need some of their extra services because you're running that many servers they want to sell you that service I believe that you can just download it from github if we had the web here I could show you but basically if you go to ops code they do offer download the Chef server in my business case we actually have to keep certain sensitive data and so we just decided that having an internally run Chef server made the most sense because we're having to maintain all the other information anyway in this way it's completely self-contained I never have to go make queries outside of the local organization so it is open it is completely open source there's a couple of pieces that are fairly advanced that no they're not in the open source one but I was wondering how accessible it really is and have you any experience in the community in submitting patches yes in fact actually I've submitted quite a number of bug reports and all the ones that I've submitted including questions when I had an issue got answered right away they basically use they use github I mean it really does most I would say 95% of the stuff you're going to find is being hosted on github and I've had things where I went ahead and getting this error message and I had a developer come back and go oh yeah you're using too old a version of Ruby and it turns out he was absolutely right that machine I actually had too old a version of Ruby I was getting an error because of that which is pretty good support because I mean he could have just shrugged and said I don't know I don't have that error he actually had the answer which was look on my local machine and he was right so in terms of availability of the community I think it's awesome I would say that you're going to find almost every major server piece that any of us here have to build there's probably if not a chef recipe already done at least something that will point you in the right direction I did have to build some custom recipes we're doing some stuff with shibboleth for instance for being able to do custom users sign on things and it turns out shibboleth stuff that existed didn't work for me but I mean my custom script is two files three files I mean it really is not much it's and that's the thing is that the models are out there that even if there's nothing that's exactly what you need you can find something that gives you the pointers so you can build what you need to do it's obviously you need to understand how to build a server to begin with you need to know how to build whatever piece it is you're going to build anyway to make your script work properly you know being able to say bring up an Apache two servers great but if you don't know how to configure Apache you're not going to know where the tweaks are you need to make to set them up same thing with MySQL tuning you know it's not going to do that stuff for you so this is not going to replace your sys admins this is just going to make it so that they can spend time on the important things not oh yeah boss I need to bring up a lamp server I'll have that tomorrow because we just saw how easy it is you know now if somebody says I need to bring up a lamp server your answer is okay in five minutes later what's what's the workflow for handling configuration changes so for example in PHP you might want to on one machine up the max file size upload file size you don't want that to propagate to all the other all the other PHP running machines right so every single node has a config setting and basically it's a YAML JSON style file there is a hierarchy of override so essentially you have the default settings then you might have settings that are set by PHP for instance then you might have settings that are based on things like roles so you save all of my web servers have this set but then ultimately you have sort of the top level and you can on a particular machine do an override so it will be probably as simple as on that particular machine config so let's say it you know something where PHP needs to have twice as much memory because you got something and you know it just needs to have that so you'll go into that one machine you'll basically build out your JSON array you'll have something on the order of say PHP mem size you know let's say 3 gig whatever whatever it is and that machine then very next time the chef client runs will actually go in modify the PHP settings so PHP I and I in most cases change it to 3 restart the services so 15 minutes later however you've set that chron to it immediately will take effect you might then say oh yeah that's too much bring it down to 2 you could tweak it as often as you wanted you could go ahead and then I'm going to set one machine this way one machine this way one machine this way run them for a while and then go look at your logs and then say now I want them all to be this delete delete all of those settings and go drop it into say maybe a PHP config and suddenly they all get that so there's a hierarchy yeah so then if you install a new machine with PHP it will take the default setting PHP thing and not the specific machine exactly okay thanks two things first of all Amazon Web Services do you know about ops works yes they have a partnership at this point and they have actually as the way I understand it I don't work for ops work so I'm not privy to it but I do understand that it looks like the partnership they have is going to make it easier and easier because it looks like basically Amazon has decided that in the battle between the various competing platforms chef puppet etc they've decided that chef is the way to go so it's going to get much easier to do chef things and that's a good example I just was part of a webinar I just watched it that went into a knife plugin to work with VMware for instance and that's the nice thing is there's tons of plugins that people are writing for all of these different you know virtual platforms etc that make we saw what we were doing there with a bootstrap for me to go ahead and not just bootstrap but actually just say go create a new instance of a virtual machine on my VM where server would be again one line it's that simple I wouldn't even have to log on to the server I could actually say go use this OS image go drop that into the machine and then bootstrap it'll start from basically no machine at all and bring it up yeah that's also my experiences but with Amazon you want to just bring up servers whenever the load is high and everything has to go automatically I just want to hear if you had experiences or could recommend it yeah I'm not using Amazon I've basically been using VMware but my experiences are pretty similar from what I hear I mean I know Aquia is using Amazon with good success they had they gave a talk at Portland that I know is online that talks about how they do it they run quite a big test suite and it's all spun up using Amazon web services and I remember they said that they actually have a failure rate that if a machine doesn't come up in five to ten minutes they basically just assume it's never going to come up and just do it again I mean there actually is a measurable failure rate that the other things just don't work correctly so which is kind of that's why I was joking earlier because it's absolutely true everybody I've heard it says Amazon's great when it works something totally different but don't hack core or don't hack contrary modules contrary recipes I tend to find some things for instance PHP PHP cookbooks all assume that I'm running Apache so they have their dependencies do I have any way of overwriting say you don't want to use Apache you want to use nginx yes and no the problem is that you're right the default cookbooks assume Apache because 90% 95% whatever the number is of people who are running PHP are running Apache I would first go look at some of the things like nginx recipes and see if somebody's already got a PHP thing that you know you can say you're running nginx for instance I will tell you that I actually had to build machines that ran PHP 5.2 and so what I did was I actually copied some of the recipes I in this case said alright I cannot generate the image I need from any of the existing recipes but I can build stuff from source I took the PHP 5.3 source recipe because there was a recipe to say build PHP from source and install it even though I was working with Debian and Ubuntu servers I mean normally you don't want to do that but in this case that was the easiest way to do it it would actually go download the tarball of 5.2 build it from scratch all under the control of chef which means every time I spun up a server it would go grab a tarball and this is a great example of a gotcha I started getting these things where I'd go to spin up a server and nothing would happen and I'd sit there and go what's going on what's the error and I'd check into the logs and it had gotten stuck and it turns out that the archive I was grabbing it from at PHP.net was down and it wasn't answering and so therefore I couldn't go grab this tarball and the script would die my solution for that was I grabbed a tarball and stored it locally I removed the external dependency because I always wanted the same tarball but it was really easy to just go and grab it from some place else until it wasn't there so it's a good example if you really want to test your scripts just in general assume the rest of the internet is gone make sure your stuff still works now DrupalCon is a good test for that right because the rest of the internet is gone it's just DrupalCon right now Hi my question is partially linked to the story which you just told I am completely new to chef and I would like to know if something goes wrong how friendly chef when reporting that something went like in your sample when you try to execute in your example like recipient my SQL for example something went wrong and second question if I apply my custom change to my SQL initialization file for example something that went wrong in custom code so in both cases there's that amazing logging by default without changing it when you run Chef Client you can actually see all of the interactions you get to basically peek behind the scenes between the communication that that server has that the node the client has with the server so you can see it says hello I'm this particular machine and the server says hello machine I'm validated that yes you are the right server I have something for you you need this this and this I don't have that cached can you send that to me yes and so you can watch as it streams the things and your client basically caches stuff so it's holding on to it and the interesting thing is the next time it runs if it says I don't need that anymore it flushes its cache it doesn't just hang on to everything they try to be really efficient but you can watch every single step then that that client is going through the various scripts including everything's good here I don't need to do anything move on to the next step so you can actually just look at that log so in the case of for instance that I could basically say oh it's trying to go get this file oh why isn't it getting that file and then I go and I sit there in my local browser and I go and I get server down so very good debugging abilities because you really can track that stuff when it comes to custom code that's actually a really good point if you make a mistake in JSON formatting for instance I tend to sit there and edit config files on the fly like I'll sit there and just you know I set my editor to nano so I'm just sitting there editing in a terminal window the problem with that is is if I miss a comma and as a result my JSON is invalid I lose the entire thing it goes to save it because it's actually doing a live edit basically even though I'm on my workstation I'm saying go edit this config it actually goes to the server gets the config grabs it locally I'm editing a local file and when I hit save it's going to push it back because I'm actually changing a config file and if I break it it says no no no you can't do that it's a broken file I'm not even going to push it it just loses it so I was doing a lot of nano stuff I started using other external editors like TextMate2 which is really good because you can do things like check stuff for JSON formatting so it's only as good as your other tools it's not the only tool in your toolbox thank you hi thank you for the great presentation I wanted to ask about the build and deployment and how could it be integrated to Chef do you have some recommendations well as I said start small and grow it ideally you should be able to build a setup where you have your development environment your staging environment your production environment and have them as close as possible within you know reasonable limitations and it's really not hard once you have one built to clone and extend so in terms of deployment my deployment strategy is basically I've got scripts that will go and do whatever is needed like for instance for a Drupal install it goes and it goes check out of my code and it goes and it grabs a database backup most of the time the most current but I actually have stuff so I can go grab backups for instance and it basically will reset everything so that way when the scripts run I can know that my servers in the state I want it to be and if I needed to spin up another server for testing purposes ideally it's going to be exactly where I want it to be so you would roll your own recipes backed by your own scripts well you are going to need to do that I mean obviously it depends on what you're doing and Chef is not going to solve all of your problems it's just going to give you a structure to build better tools I've got rollout scripts for instance that use water that actually log on to servers and click through some buttons and my goal is to replace that stuff with Drush and then my goal is to make Drush be called by a Chef script so the goal is get it out of being and it is code right now it's a recorded script and it's actually a Ruby script but it's doing it through a browser and as a result I've got a lot of inconsistencies because browsers what version of Firefox you're running today so the goal is make it less brittle bring it into in this case I want to be doing more with Drush because I want to make sure that Drush is well aware of you know I can send it commands like go get me yesterday's backup and put that on to a machine so I'm going to have some Drush scripts but then Chef it might just be run this Drush command so it doesn't mean that Chef isn't necessarily doing anything fancy it's just running a custom script thanks how concerned do I need to be with the operating system that I'm going to apply the changes to so if I've got say one server running Debian another server running Fedora I actually need to be that consistent you will find lots of recipes that work fine on anything so for instance let's take the Apache recipe you might have an Apache recipe that runs on Fedora and Debian now those are completely different packaging systems if you spin up two servers one of which is Debian and one of which is Fedora nothing about them is the same but Chef says I've been told to put Apache on and it knows how to use app and it knows how to use yum and it actually can because the recipe is smart enough to have pieces to deal with both of them will go and get what you've told it to so for instance if you tell it go get this version of Apache and it's available for both of them let's just assume that it will go and it will install the software it's coming completely from different sources but you may not care about that you might actually need to have for support purposes two completely different machines if you're running the same version of Apache it can build both of them with one recipe so in the actual recipe would you specify what the package name is so obviously on Fedora it would be HTTPD and one to Apache too yes in fact that's one of the things you can build into your recipes is platform dependent code you can say in the case of the node is this platform go do this in the case of that platform go do that sometimes there are agnostic so for instance there's the package command package install Apache 2 that might be the only thing that chef needs to know to install Apache 2 for instance but it then knows the package command which is itself basically telling it how to do something it's a resource it's go install something it then goes and it goes to the underlying code and it knows oh I've been told to install it but I'm on a yum system or an app system or whatever and handles it appropriately or let's say it's a windows box you might have a script that goes ahead and says go grab go grab a windows installer if that's in the recipe you could have a thing that installed Apache onto a windows box and it might be the same exact recipe that somebody had just built to make sure it worked on every platform alright it looks like all the questions I want to give some thanks I want to thank Bioraft great company to work for giving me the opportunity to be here today work for wonderful people including Nathan CEO and Ben the CTO and I want to give a shout out to Michelle Lauer who's project manager who helped me put this presentation together and everybody else at Bioraft it's a great team to work with Jim Henson rest in peace for doing the Swedish Chef all of the Opscode people Drupal developers everywhere you know we all stand on the shoulders of giants some like Dries really are giants for more information Opscode.com that should be your first stop there's so much stuff there I mean you could spend hours and hours and they do have links to all kinds of video presentations they do a conference which is a lot like DrupalCon so there's tons of videos that you can go watch tools I think you should check out to continue to build on Vagrant absolutely awesome you should have Vagrant running on your local machine so you can run virtual machines Berkshelf which as I mentioned is a great way to even speed up developing with Chef and finally BHA there's a bunch of BHA sessions there was one that happened just before hopefully a bunch of who were in there I'm a huge proponent of BHA I'd love to see the community really get behind it I did start groups.drupal.org slash BHA and I'd love to see everybody participate in that I think that is going to revolutionize how we think about the reliability of Drupal which is the reason I use Chef to begin with that's why I see it's related so thank you all and please go fill out the survey because I'd love to come back to another DrupalCon this was great thank you the internet is up the internet is up one ping if by land