 Good morning everybody. Sorry to have kept you waiting. So my talk is about using Chef Recipes to provision the server and then use Fabric for deployment. I am from Hyderabad and I work for a startup called Agilik. I manage the servers for the Agilik website and the clients. And I am from a Python background. I write Django. So I would like to move Is it okay? The volume? Thanks, thanks. So, how many of you are from PHP background? Like how many of you are Swiss admins basically? Great, great. Nice. Front-end analysis? No, right? Okay. PHP background? And Python? So you do everything? Nice, nice. I don't know much about PHP or Python. Okay, so let's get started. So we'll be talking about, the talk is not for advanced people. So any of you advanced user can teach me something. We'll be talking few basics for better comprehension. We'll see first a non-Chef approach to understand the pain of not using Chef. We'll look into a few of the Chef concepts in details. We'll work with Chef Solo in detail. We'll have a brief overview of Fabric. Time does not permit to give me a demo for provisioning by Chef. But if time permits, I'll try a Fabric demo. 80% of the time will be used on Chef and 20% on Fabric. So few basics. Provisioning, like it's basically when you prepare installing packages on the server, creating users, Wikipedia defines as the act of preparing the system for use of a service by a consumer. Deployment is when you move your source code from your host to the live server so that it can be used by a vehicle for your driver. As access to your cloud server, you update your system, you upgrade your system, you set up your time zone, you add users, you provide them pseudo access, pseudo privileges, you change your, configure your cell, and all that stuff. And from the Python background, so it has to be, the presentation had to have something of Python. So probably you create a virtual environment over here. You install lightweight, some server, Unicorn, for instance, you install reverse proxy tool, you install nginx, and you do all such stuff. You configure your server, you create a database, you write a Python script, or a shell script to start off, update your start off of your server. And the problem is you have to do it every time when you have a new server. And that's going to be a lot of pain. So that's why I need for Chef. Chef is a library which helps you manage your infrastructure as a code. The big players, Google, Yahoo, and everyone else, like they initially, I mean, all the session admins over there had their own reiterative process of the source that, which they have come up with as of now, they brute force and at each and every step incrementally they went on improvising it. So Chef brings all those philosophies over here and it provides you tools that are used, I mean, that uses the best practices to manage your server. And the basic paradigm with Chef is that you use, you see your infrastructure as a code, your infrastructure is no more system, you can see it as a code, you can maintain it with, say, Git or Subversion, whatever you want. Chef helps you to manage your configuration. Chef helps you to manage your configuration as good important resources, meaning you have to do it once, you don't have to do repetitive things one time and again, you just have to run few commands and things will be done for you. So why Chefs? Scalability. I know everyone who makes an app or breaks a web service has this wish that the first day itself it goes 100 million users, 20,000 per second. So scalability is the desirable property of a system which tells it how easily, how ready your system is for scaling, for handling more amount of users, for handling more amount of work. So you need to scale and the problem is you won't be able to scale unless and until you design your system to scale. And even if your system is designed to scale, you'll have a lot of pains. Chef tries to take most of the pain to itself. Again, the second factor is manageability. Say you have a good cool app, publicity hits, you add load balancers, something like Nginx with Facebook does, you add more servers, still things are slower, you add multiple databases, probably think of re-architecting the app, optimizing database queries, caching, re-caching, using memcache, database partitioning, horizontal scaling, vertical scaling, buying new servers, it's all, I mean, more pain. You need to invest in more sys admins. You need to provide them resources. It all hurts actually. So use Chef. I mean, use Chef meaning you can use some of these provisioning tools. Chef is one such example. You can use Puppet or Salt for Python. So why use Chef? The basic features are like, it's feature rich. You don't need to reinvent the wheel. The way I mentioned like, all the big players have come up with their own secret sauce. So you don't need to create one when it's already done by Chef for you. Chef is feature rich. Also, when you, I mean, one philosophy that is followed is like, you need to keep things dry, DIY, do not repeat yourself whenever. I mean, that's why we have frameworks. We don't need to, we should not write the, say, login function each time. The framework does it for you. So basically, you do not repeat yourself. So why repeat yourself when you have new servers? So Chef does it for you. You save time, you save money. That's why Chef. So let's look at the basic architecture of Chef. A Chef has a central server. Central server is sort of an API. It returns you a case and response from the server. And it has some databases indexed in the CouchVV database. Client on the, on my right hand side. Client is your workstation, that's its admin's workstation, where you have your books, where you have your metadata. And you use this data to, and you have different roles over here. I'll, I mean, I'll describe each of the terms in detail later in the slides. So Chef Client is where your, is your desktop actually. And from here, you put the data and the resources into the server. And nodes are your basically database servers. Node is basically your, say, mail server. Node is your AWS EC2 instance for your, say, database server or for any other cloud solution that your, that your app is using, this node. And there's a command line tool called NINE, which helps the client to talk to the server, to the main server. And there is a Chef Client, which helps you, helps the node to make API request via authentication and authorization by sharing of their public and private keys. And, and the server sends the API response to it. And the Chef Client install package is depending upon what the response has been sent from the server. Now, let's look at the server in detail. Server has a CouchDB database on top of the MC. So it's the basic database and designs scalable database to meet the heavy requirements. All the database, which is inside the CouchDB database is indexed by solar. Solar is nothing but an Apache wrapper, I mean, a wrapper written over Apache solar. It has a wrap and queue bufferer. It is over there. So that in case of heavy traffic, this buffer comes into play and takes most of the load. These Chef Clients are nothing but the database and the servers, which I mentioned, your nodes, and it provides a web UI or a NINE. So in case you want, if you're comfortable with the command line tool, so the NINE, NINE is the command line tool, which helps you manage the Chef server from your sys-admin's desk. And in case if you don't want, they have a very good web UI, which helps you do stuff, do the same stuff that NINE does. Okay, let's talk about the Chef Client. The Client, yes. For example, users change the nodes while not doing the nodes. Is there any... No, user don't change the nodes. Nodes are your, say, web server, a node in your web server. And you can actually, but you don't want to touch nodes. What you do is you assign different roles in your code books and you use NINE or the web UI to send those two books to the server and then that server talks actually to the nodes. The client over there on the nodes install, depending on the data which has been sent from the server, install different packages. No, actually, the methodology you use is like you don't need to touch a node. What you do is just change the code books, assign different roles to the code books on your system itself, send that to the server and depending upon what you have sent to the server, the Chef Client on the node will talk to the server via a school API. And depending upon the way you have quoted your books, it will install the different packages on the nodes. So ideally what you do, as of now, in case you are not using a tool like Chef for profit, so you basically go to the nodes and depending if it has web server, you will be installing the web stuff, say, Unicorn or Apache or whatever. And if it is a data server, you will be probably installing whatever it is. So Chef Solo is a client application. I will be talking about it in detail in later slides. Chef Solo is where everything is on your database and you have only few of the servers. So in that case, Chef Solo comes into play. Chef Client, as I already mentioned, it talks to the database server, to the server basically. And from the server, it pulls data via a RESTful API and then it installs packages. Chef is an interactive server. I mean, from those of you with Python background, you have Ipython. It's an interactive server for Python. Similarly, Chef is an interactive server for Python. Sorry, for Chef. So the different flavors of Chef. Obscored provides a service called Hosted Chef. So in case if you don't want to manage your servers, so Oxford does it for you and they basically charge you for that. Private Chef when you are able to manage your servers on your own. And Chef Solo, the target audience for this is basically the low end users. Say you have only two or three database servers and I mean, if you're not running something like Facebook, you probably don't need Chef Server. So you need Chef Solo where you have only three or four systems, the cloud systems basically, and you manage them from your desktop itself. So now let's get into Chef, the core components of Chef. So these are the cookbooks. Probably I should give an example and then deal with the theory that will be helpful. So this is basically an example of a cookbook. A cookbook has attributes. Attributes are nothing but the key value pairs which are used, which mention what packages you want to install, this version of the software you want to install. It has a metadata. Metadata is basically a cross platform abstraction. Say if you're using Fedora, you'll be used to use Yum for installing the packages. If you use Ubuntu, you need to use App for it. So all these cross platform abstraction, cross OS abstraction is done in the metadata. It also tells which packages to be installed. Recycling is basically a collection of attributes and templates populate the data which the node files has the constant. Say, yes, I use a name, I use a password or this is what will populate it through the templates. Yes, you can write. Yes. As already mentioned, resources are the basic unit working for Chef. Say you want to install Python, you want to install X, Y, Z. You mentioned it in your resources. Recipes are your collection of resources. Metadata is cross platform abstraction. And then you have node specific attributes where you mentioned the IP address of your server. You mentioned the roles. And then database packs are the, when you upload your cookbooks to the server, all these are indexed as we have seen in the, they think are saved in the CouchDV database that we have seen and they are indexed. All these collection of data is called data bags. Say now you have a server, database server and a mail server. The database server basically has a role of storing your database and the mail server is you will be used for sending mail. So database and a mail are two different roles. So in the cookbooks you can assign. This particular cookbook should be used for database and this role Y should be applied to alpha database. So cookbook is collection of recipes and it contains something more. It has definition, attributes, library, template files, metadata. And environment is basically, it allows you to set policies which dictate the version of a given cookbook which should be used on your testing server or your deployment or staging server. This is Chef Summarized. You have your central Chef server which is basically a RESTful API. You have your nodes. Nodes are your different servers. Here you are, you are the client, you are the sysadmin. Chef server has a data bag. A data bag is nothing but all the decent data indexed and which is available for querying by the help of ChefSolo. Environment as we already mentioned, it's either your testing server, it's all your production server or your staging server. Attributes are the key value pairs where you mention which package to install. So, only one device. So the Chef client on the node will, you will upload these data to your Chef server and that Chef server will tell its server to install its packages and that is done through Chef client. So, you don't need to package the libraries which you want to install. Yeah, you can keep it under gates or subordinates or whatever you want. No, you can, you have to use gates. I mean, the cookbooks already should be initialized. Failure handle, that's a deep question to ask. I mean, that's what Chef does under the hood for you. Say, okay, I'll take an example. Say, you have Apache installed on one of your web servers. So, Chef knows that Apache is already installed on that server and it won't install Chef. It won't install Apache. Yeah, it will give you the, it will give you the direct-on film and all those stuff. So, when then this run list is the list of things which you, the list of the recipes which you want to run. That's most about it. So, since, yes. Yes, yes, yes. You, that Web UI. So, you can query, I mean, all this data and logs are stored on the Chef server, which you can query and get the information regarding your notes. So, I guess mostly you have lower end assessments, meaning you won't be running thousand, you won't be configuring thousand servers at a time. So, for me at least, Chef Solo is what I find most handy and I thought about talking off over here. So, this is your system. This is my system. I have to boost all the config files and the data is stored over here. This is my cloud server. There's a client called, there's a thing called Chef Solo, which helps to put all these files present over here to the server and using our sync or whatever. And then, nice over here, the nice client over here, installs all these packages. So, your data logs, the notes and the side books, side books are basically restaurant books, which you want to write. And then, this file, that does all the stuff. So, Chef server does that thing for you, in case we have a very huge infrastructure. Does Chef Solo make sense? No, Chef Solo makes sense only for the low-end users. Yeah. Yeah. So, something like Facebook won't be using Chef Solo. Whereas, you and me, we have a small website, we'll be using Chef Solo. So, is it here? Yeah. So, this is the anatomy of our solo.mv file. You mentioned your file cache path, which generally shows the cache. You have your cookbook path. Just one thing to mention, cookbooks are placed inside the cache directory. So, you have cookbooks under this, where Chef Solo and your cache path is where Chef Solo. JSON attributes contain the in a JSON format. They contain what packages to be installed. And the recipe you are probably, you have your recipe. It's a good practice to deploy them to say, Amazon S3 or probably, so from there, instead of maintaining all these things over here, since you already know what packages you have to install, you package them and put it on an S3 server and pull it from there again and again. An example of a node.json file, node.json file over here is, it tells me the name server. And it, for instance, it will, this run list contains only engineers and then install engineers. Even with Chef is really, I mean, sometimes it becomes tedious. So, we have so many wrappers written on them. So, these are some good wrappers written on Chef Solo to make like easier for us. I prefer, you can have your own choices. I prefer my Solo. And let's talk about it. I mean, these are very easy stuff. So, the first four steps over here are one time stuff. So, you install Chef, you install line Solo, you configure your defaults, configuring defaults meaning like you will generate your private key, public key and put them in the respective files so that you talk to your server and the step, the process of authentication authorization is done. Then the fifth and the sixth steps are say, when you have a new server, what you do is you simply nice prepare user name at particular IP address. This will put all the files from your, from your machine to the server and when you nice cook, so it will install all the packages. That's about it. Now, let's come on to Fabric. Fabric is a Python library. It supports Python 2.5 and above. It basically helps you streamline your SSH process. It provides you functionalities where you can get input from the terminal in the client. You can start a process, you can stop a process and can do a lot of stuff. Generally, the basic of deployment is like you copy code or your media files, images file and video files or whatever to the database. You need to run the database migrations if you have changed something in the database and then probably start it's WSAIs for Python or Ruby or something else. You'll have to start some different demons. So, this is the basic of deployment and we need to do it every time. So, Fabric helps you automate these stuff. One small example of a Fabric file is, I'll do a git to say, you generally give your source code in a GitHub or somewhere in a repo, maintained probably by GitHub subversion. I wanted to write a function which would pull my database, pull my source code from the repo to my server. So, what I do is with CD, root path, root path is a constant where your project root is and you run this file simply, git pull, origin master. And when you do Fab, space, git pull, it does the things for you. So, you don't need to go to the server, you don't need to log into the server and do these things on the server. You can do it things from locally. I'll show you a small demo. Things work well. I have a virtual machine running which has an IP of this thing. So, I put it, the user name, the user name is who would do the password. The host name is, you have to provide the host name and then you go to your root path. You have a function called git pull, you CD to your root path and then you pull from the repo. You run the sync DB. I mean, this is a Django specific thing. Not sure if everybody is getting it. What sync DB does is basically the Django is an ORM of the relational mapper. And what that relational mapper does is you write your Python script as class and class and objects and it generates database volume. So, sync DB uses that file, modules.py to create a database. And the sync DB command helps you create those databases from the module site. So, run sync DB to create your databases and Unicorn is a simple lightweight Python server. And you use Unicorn to restart the server. Git pull over here. It's a virtual machine running on my desktop. I have my basic Python application, I mean, Django application. It's a Pulse application. Those are familiar. I mean, it's the hello work of Django. So, a very basic application. I'll try and run. I mean, things go well. For no reason, I don't understand why the virtual machine is not running. But all you need to do is have deploy and the code will be deployed to the server and everything will be done. The database migration and the server restart, everything else will be done. And then this automates, like going to the server and doing all those stuff yourself, you can use the fabric file to deploy stuff. So, that's most about it. Probably, I'm looking for your questions. So, you have a, you're talking about nodes, right? Yeah, yeah, yeah. So, there are a lot of... When you write a code, you upload the code to your server. Now, to make it to your node is you apply it to the longest of that node. Now, either you can set up the prongs of that and not the shed crime, that particular command, it takes 30 minutes from now. You want it to do two nodes. Or if you have command only for one node, then how do you get it to like SSS server to run a particular command on a particular node that is a little bit of a problem. That is really a problem. But if you do it for some reason, it will be very useful for you. Can you push it from the server? You don't need to push it from the server. You don't need to, but in case it's a small thing, you don't need to do it for a credit card with the previous code and change the code. Can you accept it or not? Because this is a random game. I haven't tried that, sir. I basically rely on prongs. Yes, how do you do it? Basically, you have to know which node it is, find it, and most of it. Do you know what it is? What is it like? Huh? It's super cool, sir. Because it's super cool. So, ideally, not so bad. Bigger is not so bad, not so bad, okay? Yes, it's super cool, sir. So, yes, I'm done. In case you have end-outs, you can contact me anytime. So, that's most of all. Anything else?