 Okay, so this is intro chef before we get started how many people would identify themselves more with a developer type of role Okay, and more of like an operations or admin Few okay great Okay, so this is intro the chef a little about me like Colin said I work here on the developer productivity team That I'd be in Watson. It's my Twitter handle emails get hub. This will be on like the last slide, too So you have this problem. We have lots of machines. Maybe you have One machine or five machines or maybe you're at like facebook scale. You have thousands of machines You also have various software components. Maybe you're working with apache My sequel postgres both at the same time You also have to worry about your own software. It's not just machine Resources or machine services You have multiple operating systems. You're really good at dbn, you know, you've gone to $1,500 training for this You've You run your blog on dbn You're very familiar with dbn then your boss comes to you and tells you we now support solaris So learn everything you can about solaris real quick because we're putting it in production in like two months Uh, you also may have had may have no documentation on any of this or most often it's out of date So you go to upgrade your Jenkins at your first day you come in You install You grab the newest war you pull it down you announce that you're going to be upgrading Jenkins You throw in the war You upgrade the plugins and then you come back tell everyone okay. It's ready to go You come back someone pings you later on and says, you know, none of my builds are pulling from our repository So You go back in you figure it out a couple hours later You figure out you've upgraded a plugin that you were using some hacked up copy of Because you know one feature broke one time. So you've you've hardcoded this So then you have to revert everything go back test again. Make sure everything works So you might think uh, okay. I'll just keep my documentation up to date Um Here I have shown just one document for installing and configuring Apache as you see there's multiple links levels of links links within links linkception Also, this is going to change from different versions. It's never going to be the same and that's probably any different across platforms If you're a developer and you and you get your new laptop on your first day And you try to set up your developer workstation We all know it never works out within the first three hours You might spend two days just trying to get your workstation set up Then you get this workstation set up and you get a new laptop Now you have to do that all over again Or you have a laptop at home that you want to keep in sync because you like things. So here's instructions For just a typical stack a lamp stack So I don't know if you've ever used node before but just getting node on a machine and then installing all of your modules can be a chore So you might think this seems kind of like a job for a bash script or even better a pearl script uh the problems with these are As the problems sometimes the documentation might not get up to date And two years from now you may have written this pearl and most often you'll come back and have no idea what it does So another thing is these aren't very configurable per system unless you're filling in a whole bunch of if and if else uh case statements all of these things So I might think maybe a combination of documentation and scripts would work, but Indeed want to be a baller So the answer is chef So chef, uh, this is kind of a little bit of ripoff from chef's site Because I couldn't figure out a way to explain it chef allows you to Design your infrastructure as you would a software application. So simply a framework for defining your infrastructure as code What does that mean though? If you've ever spent any time configuring services on machines You know that every service comes with its own configuration file. It comes with its own microservice the run under services uh, and every Every service has its own guide and it's different as I said across all platforms So you want a unified way to combine all of your configuration together such that You can put in a if or whens or anything that you can Pull across Sorry, you can pull your your code down and have it work across all operating systems Or just the ones that you decide to work with So why chef? Maybe you've heard of ansible. Maybe you've heard of salt. Maybe you've heard of puppet I personally like chef because one is written in ruby You might say puppets written ruby also But puppet forces you into a kind of a very specific dsl ansible can do the same With chef you can write raw ruby out And use it directly in your code Also some models can only be push Ansible does this so you set up your ansible thing you fire it off And if ssh goes down and you're doing this across a whole bunch of machines, you're kind of so well if it doesn't work Chef takes a pool model whereby you have a chef server that stores all of your code And nodes that come online Pull from the server normally you'll have a a cron service setup such that this happens about every 30 minutes This scales a lot better than push as I said ssh can be very unreliable And also there's a very extensive community if there's a service that you're looking to configure the cookbook probably already exists And I'll show you where later on So your typical architecture Is going to look like this you'll have your chef server as I said, which is kind of the master model This also contains all of your cookbooks, which I'll explain in a sec And then over here you have your nodes. These can be virtual machines They can be cloud instances physical servers docker images, whatever you're feeling And then most of the magic happens on your workstation Where you write cookbooks and you interact with the chef server through knife All this code is also uh normally stored in a repository probably in uh git You can also use subversion or Whatever product you're feeling So I keep on mentioning a cookbook and what is a cookbook exactly The cookbook is a basic building block Of your infrastructure a cookbook can represent a service So you could have your Apache cookbook or you might sequel cookbook Or your postgres cookbook A lot of times you'll take these cookbooks and you'll combine together in a wrapper type of cookbook In which case you'll define An application that you want to install say you want to install Conboard or wordpress So wordpress contains a few things that contains nginx. It plays the application itself. It contains my sequel Also, you might want to for run it in there So you're going to be combining your application or my sequel your nginx cookbook and a wrapper cookbook And it kind of configures all of these things together You also may have different environments for your application You have a development environment where the settings of the machines are slightly lower than your Production instance So you want a way to deploy to all of them But a way to override defaults for maybe memory or hard drive space Or maybe you have a load balancer in production, but you don't have it in your dev Okay, so I'm going to roll right into a demo Just to show you how to create a cookbook So on my machine. I have a few things You can choose to Host your own chef server, which is very easy. It actually uses chef to install chef Or you can go with the hosted solution Which I would say if you're just messing around with this, you should go with the hosted solution for now Just to play around And you don't have to worry about managing the server. So that's done here go to manage.chef.io Make yourself an account. I think it's blocking something Okay, so you get a nice web interface like this You can do most of the work through the web interface or like I said before you can use knife to interact with it Uh, a lot of the cooking terms are on purpose because it's called chef So to get started with this like I said you sign up for this account you go in here To the administrative panel you generate a knife config You generate a user And then you download a key shove all that under a dot chef under your home directory And you're ready to go So the first cookbook i'm going to do is i'm going to piece it out. So chef works off of uh resources And these resources represent certain things like a package or a template or a service So the first thing that I did is I created this example my Apache recipe So this is a recipe that I created from scratch Using this command berks cookbook And then the name of the cookbook that I wanted to be This will scaffold out a lot of files and you'll end up with a directory similar to this here A lot of these you can ignore for right now, but I want you to focus on attributes recipes and templates So the first thing I'm going to do is I'm going to crack up with the recipe The recipe is going to define most of my logic For the steps that are going to be performed So the first thing that I do is I need Apache itself Because I'm targeting sentos I specify the package as htdpd because that's what I know what it is on sentos So this right here is a resource package htdpd The next thing I do is I want to override defaults to come with sentos So in here, I'm specifying where in my configuration I want this template to be Uh And that's going to come from the templates directory right here. It's an erb kind of like rails So the template where I want it is going to be etsy htdpd cough htdp cough owner and group permissions And then down here I actually want to start the service after it's done whenever I'm configuring the service and I want to Make changes to my conference file I also want the service to be restarted to pick up those new changes So that's what this does right here I want to notify The service to restart every time something new happens here so to give custom Attributes I go to the attributes file. You can think of attributes as almost global variables. They're going to they're going to be Accessible throughout your whole cookbook and then there's ways to reference them from wrapper cookbooks downstream So I say my Apache name of my cookbook Where in my template I want to actually Uh change this And then the value that I want assigned So I'm changing two things in the template the timeout and the log level And like I said under under here is actually going to be that file that I referenced before. This is the htdp cough erb Taking a look at it Like I said, I wanted to just change log level and timeout So right here is where I'm doing that Uh, if you look at it, it's very rails like so you're doing the equals And then name of the attribute that I want filled in here And that'll get filled in Timeout looks just the same Here, this is the metadata file where I'll reference other cookbooks Since I'm not referencing other cookbooks. This is going to be largely blank However, I want to be cognizant of the version, which is a semver version To test cookbooks on my local machine. I'm going to use a mixture of what's called Uh test kitchen what's come shift with the chef decay Which you can find and download here Uh, this will be your complete package for interacting with chef contains All the tools necessary to test create cookbooks and interact with your chef server So test kitchen is jammed that comes shipped with this And I can test to make sure this works using kitchen If I do a kitchen list, I could see what operating systems and suites that I have set up Currently, I'm just testing for Ubuntu and sentos We know this is going to fail on Ubuntu because the way that I structure the recipe just because this is an intro Uh, I didn't put any defaults in for what happens if I want to do this on Ubuntu I can figure this with a kitchen file. So I'm using vagrant as the driver for this The type of provision I'm using chef solo The platforms that I'm targeting And I'm referencing the run list The run list has the ordering for which recipes are going to run when So I just referenced that I'm running this local cookbook here So do a kitchen converge Sentos if I don't have this it's going to pull down the machine from the vagrant hub. Normally it's going to be a chef provided box And then it's going to run something called chef solo So as I said before the normal model is you're going to have your chef server And you're going to have nodes that are pulling from that If you don't like that model you can use something called chef solo In which case it's going to run all this locally But you need to have particular steps for how do I get chef on the machine? How do I set up paths and I need to provide Uh, some sane defaults for how chef solo is going to run So this will run for a bit. Uh, and then we'll see at the very end how it works Uh, there's a lot of information to throw at you at once. So real quick Does anybody have any questions specifically about this recipe at the moment None So like I said, the first thing it's going to do is it knows Uh, several different options to grab chef and it grabs this uh from packages provided by chef It'll try wget. It'll try curl. It'll try python Some of these things aren't listed out by default on the machine So it'll grab that and Start to converge while that's running We can go over here because Uh, obviously a lot of people have probably done this before So we can go to supermarket supermarket supermarket has cookbooks created either by ops code Or by individual contributors themselves most of these are going to be hosted on github And you're going to be able to uh, search The ops code approved ones or the chef approved ones through the center face So we're looking for a patchy Go in here search patchy And we get multiple options There's not always going to be one because people have different use cases for a patchy. So let's use this one And we can see a little bit of information in the read me about how to reference this as I said before you could pull into the cookbooks So you're going to be referencing this as a patchy two Uh, we see a version that we can log to here. We could also check out the source which is normally on github Right here. So this will provide a bunch of sane defaults for Uh, uh, patchy And then I'll also go ahead and probably throw on some other things like a patchy modules Or you can even create whole Websites On here it says that Yeah, web app So you can see an example right here So these are things that we're familiar with they decided to call their resource web app You can create custom resources and providers, but that's more of an advanced topic. So I'm not covering it today Uh, but he's just referencing the template And this is an attribute right here So we go back over here. We see that everything is finished. So we can actually log into this machine to check Um If this is installed or you could write tests Similar to our spec to run these at the very end Let's catch and log in We could check that the service is actually running Ed, but this looks locked. So there might have been a problem actually with the configuration file We could take a look at that here I'm wondering if this just did not use my file So 120 was actually what I said it. I'm not sure why it's not searching um That got put in right here We could take a look at log level if search was working So this is a basic example of how to create your cope grid from scratch Um, as I said before you can pull in other cookbooks So I'm going to demonstrate how to do that Seth Varga was supposed to be here And I'm using one of his cookbooks actually to demonstrate So we're going to go to super market. Um, how many people have played with Jenkins before? Okay, so Jenkins is uh for Building normally your source code Uh, you could take a look at it right here It's Jenkins. You can download packages So Jenkins comes with its own set of steps for installing itself and it's different across platforms So I want to figure out is there a way that I can install Jenkins without doing all this myself So I look here. I see Jenkins. I see this is actually sponsored by chef. So to say By sponsored, I mean it lives under the repos basically Take a look at the source So I figure out that this has a couple of dependencies itself. It depends on the java cookbook. It depends on the engine x cookbook depends on the Run a cookbook And like I said, you can find out dependencies of it right here But you don't really have to worry about that because chef comes with its built-in dependency manager similar to ruby gems And it's called berkshelf So to use this cookbook environment in my environment I'm going to take a look at some of the defaults that it comes with If you notice it doesn't have a default recipe Default recipes are normally run every time but you can specify specific recipes right here So he has a java recipe Where all he does is install the java dk using the package He has a couple of wends in here for when it's debian use this package when it's rel use this package And that's just a simple case statement And then I want to install a master node so see that's where A lot of the magic's going to happen right here So using this in my environment like like before I did a berksh cookbook Create my jankins test But I do a tree looks very similar Because I'm defining a dependency because I'm pulling I have to define my dependencies here. So I'm going to be pulling in jankins directory directly Um, like I said, there's sometimes there's more copies of the jankins cookbook We could figure out exactly what it's called By looking in the metadata and we see this is called jankins So I put that right here I'm going to be deploying this on your ubuntu machine. So I want to include the apt cookbook 2 this cookbook has a little bit More configuration that we don't need to go to but it's important because it updates my repositories before It's tries to install any packages So to use this cookbook, I just say include recipe. How do I know what the recipe was called? It's right here So I include the recipe apt if I don't specify the specific recipe It's going to assume the default recipe. So I just say apt it assumes default It's going to run the default recipe which is going to really load my cache And then I say I want job on this machine jankins has a pedency on java and then I want to set up a master So one of the cool things is jankins ships with its own attributes, which I can override in my cookbook So in here, you have a hard coded value in jankins of nil So normally this is probably just going to go and pull the latest version I know very specifically that my code is not interacting with jankins Zero one two three So in my attributes I say that I want to lock this to one five nine eight So as before I say the name of the cookbook that I'm targeting I'm saying the attribute within that cookbook that I'm targeting and I'm locking it to a specific value So I could run this kitchen test, but I know that it's going to work and this takes a little bit So I'm going to do go ahead Configurations do you need to create a new cookbook that depends on the original one? Yeah, so you can pull in all of the original recipes and values and everything just by calling it in yours here So I'm saying I want the jankins cookbook And I'm overriding parts of the cookbook like here I'm overriding the version attribute that's listed in this cookbook If this is the engine x cookbook, I'd probably have default Engine x and then maybe another time out and setting it to something else. Does that make sense? Okay Yeah, so I'll mention that probably at the end. Yeah Okay Uh before I set up a instance in digital ocean Uh, so I have that instance right here. It's a new bantu instance So to upload this cookbook to my chef server, I'm going to use burq shelf so that all of my dependencies come with it So I do a burqs upload If I do a dot it should pick up the Directory that I'm currently in Or not Okay, so this is already being uploaded before And I see this is the same sembler version that I have listed in my metadata here Okay So now I use knife to interact with my chef server And I'm going to bootstrap a new node The ip of that is here I list the user I list the password And then I say in my chef server. What do I want this to be called? So it's going to go out It's going to go ahead and install. It's a chef bootstrap it Then I should be able to see that over here in chef manage Under nodes Oh, yeah, the JavaScript So I see Jenkins right here So now I need to edit this run list. So I do a knife node edit Name And then I could specify in my run list here using json what I want it to be I can also do that through the web interface Edit the run list here I could search for what I just uploaded drag that over Save the run list Run chef client and let's go ahead and run chef client So this is going to take a little bit Uh, any more questions at this point go ahead Yeah Yeah, so digital ocean is just a cloud provider Right here where you can spin up virtual machines or Docker instance, I think you can do docker instances somewhere But this is just the cloud provider that I decided to choose So I have a virtual machine running in here. Yeah, I'm gonna have to log in and everything Do you need me to log in or does that make sense with digital ocean? Okay? So burkshelf as I said before is a dependency manager there's a lot of Quarrel I guess over which dependency manager to use Chef has largely kind of backed burkshelf But this was kind of based on the purpose that it was there And They didn't feel like doing the work. So they latched onto burkshelf burkshelf You could find information about here where you can learn all about Locking versions of cookbooks Running burks installed to grab all your dependencies And it runs similar like ruby gems So chef themselves host an end point Here supervarket chef.io Which contains all these cookbooks. So when you say jankins It goes here retrieves the cookbook knows that's a dependency It'll pull it down on your machine or pull it down onto the server Um, there's also a specific uh So once your stuff is up in your chef server Some people threw a lot of their cookbooks under a central repo Which they called chef repo and then it was chef repo cookbooks all of my cookbooks So you could keep track of it that way danger Oh, I thought you were raising your hand like it's on this So run finished fingers crossed It's going to do that because jankins runs in 8080 Right, so now I have jankins up here So I know absolutely zero about setting up jankins But I can include the recipe And someone's done all the work to set it up Now this is going to be custom out of the box So I'm gonna have to dig in to figure out what the other type of recipes I want to add to further configure this Um, but it's up and running. It's in digital ocean So just back to slides So a lot of information that I just threw out Where to go from here? I wanted you to walk away from here having a basic understanding of some of the principles But I don't expect you to go back and throw everything in and get started Chef themselves publish great video tutorials at learn.chef.io Also, you could take a look at Seth Vargo's book and Misha Taylor's at Intro to Chef. It's a book published by Riley When you're ready to step up your game, you've ridd through these books You've maybe watched some conference videos You could take a look at customizing shaffer and by John Cowey right here Okay So, uh, here's my information again. You can email me at See me Luciano at your groon.io. You can take a look at my github and here's my twitter Any further questions? Go ahead Yeah So a lot of times like say you just do not like the Apache recipe You can create your own Apache recipe. You can upload it to your site You can run your own Burke shelf file to grab dependence or a Burke shelf server to grab dependencies A lot of times this is what you're going to be doing Um You'll find a cookbook maybe that you like you'll include it in your own wrapper cookbook You'll override a crap ton of variables and you'll upload it to your chef's server recipe so One type of thing that you have a problem with is say you didn't lock The cookbook to a specific version and they go and they completely change the cookbook And you have dependencies strongly on that cookbook because you said pull the latest which is probably going to do every time and It breaks all of your stuff. Um, this is a common problem Which is why Another reason why Burke shelf was included so you could lock your version specifically if you look I see the current version is 222 So if I'm really comfortable 222, it's working for me normally I'm going to want to lock My wrapper cookbook to a specific upstream cookbook and that way I know it's not going to change and if it does change It's going to be locked and it's not going to pull the latest. I do that right here So it works kind of like a gem file. You can specify exactly this cookbook. You can say Um at least this cookbook and greater or we can say it ranges like I want Greater than this is cookbook, but less than this cookbook. Does that make sense? Okay, did I answer your question? I'm not sure if I fully answered it or not Okay Right Okay, yeah, so Deploying applications with chef while it can be done is not 100% a common process So there's been a major rewrite on a lot of these application cookbooks for deploying these things um Let's have a look at Cookbook application Here's an example of a cookbook. Um, that's going to get all of your Java dependencies and it forces you into kind of a Certain workflow Which we'll see down here and a lot of this mimics a type of capistrano if you're familiar with capistrano and ruby type of workflow Um, but here he's mentioning. This is a java web app. Okay. I want this database and then further down He lists where my Specific application is because it's java comes normally in a war file. So it's easy to just throw on there Point your war at it and you're ready to go A lot of people that are writing custom cookbooks For their application are going to be doing a lot of customization On resources and providers itself. So you're almost Creating your own resource like java web app You're giving it these parameters and then you write something called a provider So you can think of resource as the declaration of what I want it to be And the provider is another file with the steps To actually get to that point Chefworks is something called like a declarative where you say how I want it to be And then chef knows how to get to that point But with your custom application, you're going to have to write a provider to tell it how to get to that point It's an advanced topic. It can be done with just recipes. You can use the bash resource itself to just do as you normally would do in a bash script Or you can call on other scripts that you already have to set up your application Or you could just use chef to set up the infrastructure and use whatever existing deployment technologies you have One thing that I've done personally is I've used chef to set up the infrastructure And then I have my app in a docker container And so I just have chef pull that docker container down and it had Docker has all the dependencies from my application in it and it runs those so Creating your own custom cookbooks is something that you will see a lot of in the customizing chef book Where he talks about writing your own providers your own resources and goes into a lot more of that depth But it's quite an advanced topic and chef themselves don't necessarily while the marketing people might say Yeah, you chef to deploy your applications not a whole lot of people are using it to deploy Their own custom applications without a whole bunch of work going into creating A custom resource a custom provider and whatnot. Does that make sense? Go ahead So you're saying how do I convince my employer that this is important to let me use it? Okay, so um This could probably be a talk all of itself But one of the things that you might be able to do is Maybe just go ahead and benchmark how much time you're saving by putting all of this into automation I then say oh now I can work on more advanced problems. Maybe we wanted to introduce that new monitoring system Uh, dego, but I'm so busy Logging into these machines tweaking a little line And then logging back out and logging into the next machine And then tweaking this little line, but oh I messed it up So now I need to go back into that machine and then tweak the line again Also, you're going to need to spend a lot more money maybe on training or a lot more time Just figuring out different operating systems. I'm not sure if you only run one operating system But I I've considered uh I would consider myself very uh productive with sentos But then I need to learn solaris. So now I need to go learn solaris That's going to be a huge time suck whereas I can use these automation solutions to Uh leverage the work that they've already done with solaris to get that on there I have shown the hosted kind of sass provider But like I said most people are going to be running their chef server within the organization behind a firewall You can also set up your own berkshelf uh dependency manager for this and uh Or librarian. I'm not really sure how librarian works The best thing to do is just to kind of show them the time saving show them the real value of these things Any other questions? Right. Thank you for your time