 Thank you Garrett. Good morning. So third or fourth day of Duplicon, I hope the coffee starts to kick in and I try to be as entertaining as possible at that time of day. So let's and get the less important things out of the way. In that case, that's me. My name is Jochen Liedrich. I'm from Germany and as one starts with rounds like this, hello, my name is Jochen and I'm a CIS admin and this is where you say hello Jochen. I found Linux in about 1993 during my studies of computer science and instantly fell in love with it, started doing system administration, started my own first business already during my studies and in consequence nearly failing my graduation. Then I worked as a SUSE certified Linux trainer for some time, became an IT manager at WebDE, which is one of the biggest web portals in Germany. Then one-on-one acquired WebDE so I changed to one-on-one and became head of IT core services there, leading at three teams with 10 persons each doing system administration, providing basic IT services like backup systems for 50,000 servers, software deployment, all kinds of automation, so that's when I started to specialize in topics like I'm presenting today and in 2009 I decided to found another business and started early 2010 with my company Fry Steel IT and the product that I presented first at the Drupal Developer Days in Munich in 2010 which is called Drupal Concept. At Fry Steel IT we have the motto ops for devs so we do IT stuff that helps developers be more productive, be more efficient and our main product is named Drupal Concept which is a platform as a service or infrastructure as a service that's completely optimized for Drupal. We do the management of the whole infrastructure, the infrastructure is completely automated and so our customers which mostly are development shops and agencies can concentrate and focus on developing websites, building websites, consulting their clients but they don't have to get up early in the night because some hard disk has failed and the lessons we learned building Drupal Concept and the complete infrastructure are mostly about automating IT systems. We are running now more than 150 servers with a two-person team and when I started thinking about what to present here at Drupal it didn't take long to get to the topic okay how can I use this automation knowledge in a development setup. When we're talking about development setups there are a few challenges. Having a local development setup makes you free to work wherever you are if you have a development setup on your laptop you can work at the office as well as on the train on the plane and so you can use your development environment wherever you are. That's handy but it requires a lot of effort, installing and maintaining so sometimes you lose a lot of time by installing a new development setup at the beginning of the project, maintaining it, doing software upgrades and all those things. Another problem is well it worked on my machine syndrome when launching a website on a production environment and finding out that something doesn't work as it's expected to do because there was a mismatch in software versions another version of a PHP extension for example and other things that have to be as equal as possible between the development environment, your staging environment and your production environment. There are software versions, there are dependencies between software components, there are updates that require you to update your software on every environment and this takes a lot of time if you're doing it manually. I'm happy to say that there are now tools that help you to minimize the time you spend maintaining your development environment that help you to make sure that your development environment is set up similarly than the production environment and that help you to reproduce a setup every time you need it. So you won't have to fear that the development setup you use with your new project will differ in any way from the one you are used to from the previous project. There are mainly two tools I want to introduce you to today. The first is named Waygrant. Who have you already heard of Waygrant? Hands up. Okay, so I won't waste much time explaining what Waygrant is. Waygrant is a Ruby gem, a package that uses virtual box to build VMs from a central configuration. It makes setup of VMs very, very easy. You just need a few commands to get a new VM up and running. The first step you do is to install Waygrant which you usually do via the gem command. With gem install Waygrant you will download the Waygrant package, download all dependencies and install them. The commands that follow are the usual commands of a Waygrant cycle. First you need something that's called a base box which is an image of a pre-installed operating system. In this case I use Waygrant box add to download and install a Linux VM image that's based on Ubuntu 1204 and especially its 64-bit version. The Waygrant box add command will now download the image from the URL that's on the command line, store it on a central location on your computer and from then on you can reuse it to set up as many VMs you'd like to. If you want to start up a new VM it's easy. You just create a directory where all the necessary configuration files will be put. Then you do a Waygrant init precise64 referencing the base box you've downloaded before and with Waygrant up you'll have a running VM after about a minute or so. Finally to use your box you can SSH into the box just by issuing the command Waygrant SSH so you don't have to deal with local IP addresses or things like that. Waygrant gives you an efficient command line interface to do everything you need. Before I'll show you how that works just a few words about the boxes. Those Waygrant boxes are pre-configured with mostly a minimal setup and you can download them from files.waygrantup.com where there's a community repository of predefined about your box images you can use. Of course you will also be able to build your own boxes. These commands help you with that. You can use Waygrant box and different options and Waygrant package to build your own Waygrant images and there's another tool that will make that process even more easy. That's called VV. It's created by Patrick Dubois which everyone in the DevOps world should already have heard of. All you need to spin up a new Waygrant VM is the Waygrant file which is a small file containing configuration commands. Basically it's Ruby but it's a domain specific language so you don't need to be able to code in Ruby to configure Waygrant. You'll see that in a minute. That Waygrant file can be replicated to all the VMs you need to spin up. By using a common VM configuration Waygrant file you can make sure your VMs are set up equally. That's how such a Waygrant file looks like. It's just a block Waygrant config do to end and at that example I have three basic configuration lines. It's what base image is the box based upon. Should it run in invisible mode which just spins up the VM in the background and you can SSH into it or should it use the GUI mode which VirtualBox uses if you are using VirtualBox directly. And a very practical thing is Waygrant will set up port forwardings automatically. So if you have a VM running a web server it'll map that web server's port 80 to your host systems to your laptops system in that case. If you access localhost colon 8080 you will reach the web server running inside the VM. So you don't have again to deal with the IP address of the VM or doing your port forwardings yourself. Waygrant will take care of that. So let's see how that works. Here we go. So I just do a Waygrant init precisely 64. It tells me it has created a Waygrant file with a default configuration that I can use. Let's see how that looks. So most of these lines are comments. Here is the base box I've used in the init command. Here I can activate the GUI mode if I prefer that. Here I can define how the networking should be done with this VM. Per default the VM has a local private IP address but you can connect the network of the VM to your host computer's network or even your local network so the VM will be reachable from your office network for example. And here's the port forwarding. And there are many different additional config options I won't get into now. So all I have to do is to do a Waygrant up to start the VM. Studying it for the first time will take a bit of time because it needs to import the base box. All subsequent startups will be more quickly. It's almost done. Here we go. And with Waygrant SSH I can log in into the VM and I'm ready to go. You could argue that that's not very impressive because you can do that with VM imaging tools provided by VMware or other virtual environments as well. So I'll spice it up a bit by introducing the second tool which is far more important to be able to automate your development setup completely. Its name is Chef and it's built by Opscode. Chef will enable you to automate all system administration tasks and by defining how your setup should look like you'll be able to run machines that are configured completely equal. It's also Ruby based has a domain specific language to describe your administration tasks and it works in a way that you define which packages you want to have installed, which configuration files with which contents you need and Chef makes sure that your box looks like you described it. With every Chef run on the box it'll check are the packages installed, that should be installed, are all the configuration files configured as they should be and so on and it makes all the necessary changes. So the first Chef run probably will be the most long and all subsequent Chef runs probably will be very short because everything is already in the state it should be in. Chef replaces manual system administration tasks by describing how your system should look like which in the DevOps world is called infrastructure as code. To that goal Chef uses different parts mostly cookbooks which describe different software components you want to install. For example there is a Apache cookbook that contains all the details how Apache should be installed, how it should be configured and gives you the opportunity to change specific configuration options. Cookbooks consist of different components for example recipes. Recipes are a kind of high-level script that say okay for example for Apache first install Apache then start the Apache server then write a configuration file from a template that is predefined in the cookbook and then every time the configuration file is changed do a restart of the service. All the system components you can administer with Chef are called resources. I come to it on the next slide and all your servers all your boxes are nodes in ChefSpeak. Every node has attributes for example an attribute could be which port should Apache respond on and to make defining how a web server node for example should look like as easy as possible there are roles where you can say okay the Apache role means that I need the Apache cookbook I need the MySQL cookbook I need the PHP cookbook and pre-define certain attributes so if you once you have those roles defined you just say okay that's a development machine and in the role development machine everything is defined what a development machine means. Chef can maintain all kinds of system components it can install packages it can start or stop services it can create files or directories it can create configuration files from templates it can run single commands or complete scripts it can install cron jobs it can clone git repositories and many more things for example that could be part of the Apache recipe which means take the package Apache 2 and install it what it means to install a package depends on the operating system you use on reted based linuxes it'll be it'll use rpm to install the Apache package on Debian based linuxes it'll use the app system that's all behind the chef abstraction layer you don't need to define the actual commands to install the package you just say install the package and chef takes care of that then you can say start the Apache service again our service is started or stopped depends on the operating system you use chef of course knows on which operating system it's running currently so it can take care of that without you knowing exactly what OS you are using at the moment this could for example change over time and then we have a template definition which writes the Apache 2.conf file you define which source template in the cookbook should be used and a few options that define the permissions and ownerships of the file and the last line instructs chef to restart or reload the service if the configuration file has changed you can find out more about chef on the different community websites mainly the ops code wiki on the ops code community website you'll find a huge library of cookbooks people have provided as open source for example the Apache cookbooks there's a mysql cookbook an engine x cookbook and many many more and on github there's also a huge ecosystem of chef cookbooks for example ops code themselves provide their own cookbooks which they maintain under the third URL here okay the real magic happens when we combine those two Ray Grant on the one side enables you to easily spin up a VM based on a predefined image and chef then will enable you to configure that VM in exactly that way you want it to be there's not much you'll need mostly you'll need the chef cookbooks you can download them manually for example from the ops code community website all cookbooks are provided as star balls you download them then you unzip them put them in a directory and use them to provision a vagrant VM there's a small tool that makes that task very easy it's called chef librarian it's another ruby gem that is simply installed with gem install librarian you do a librarian chef init which creates a basic chef file where you describe what cookbooks exactly you'll need and with librarian chef install it'll download those cookbooks you need and put them into your VM directory let's do that so at the moment I'm in the VM okay let's get out of here at the moment all that is in my directory here is the vagrant file so I do a librarian chef init now it's created a chef file that looks like that it basically defines where to get the cookbooks you order it to install and then you just define the cookbooks for example here's the Apache cookbook I don't need that for a Drupal development VM then we'll need the PHP cookbook and the MySQL cookbook those cookbooks contain one or more recipes for example the MySQL cookbook contains a recipe to install the MySQL client and another recipe to install the MySQL server so you can decide which components exactly you want to install that's all and then I'll do a librarian chef install it'll wrap those three cookbooks put them inside a directory so it's very very easy to get those cookbooks and update them that's how that looks and if we take a look into the cookbook directory you'll see that we have the Apache cookbook the MySQL cookbook the PHP cookbook and since those cookbooks also define which other cookbooks they are dependent upon it already downloaded the build essential cookbook the OpenSSL cookbook and the XML cookbook which are needed by one of the first three now we are good to go here's the next step we'll have to define what of those cookbooks we'll want to use the best way to do that is to define a role which you can reuse and reuse and reuse that's Ruby again but as I said it's a domain specific language so it's basically syntax but with keywords that are special to the chef you define a name of the role and a description and then the main thing is the run list which defines which recipes should be run in a provisioning cycle we need the Apache 2 recipe to do the basic Apache installation then there are three additional recipes that install special Apache modules I'll have the recipe to install PHP I'll add the MySQL extension to that and since I want to run MySQL locally in the VM I also use the MySQL client and the MySQL server cookbook a recipe pardon with that definition you can spin up VMs that look the same every time you do a fresh install and all you need to do to connect chef and vagrant is to add a few lines to the vagrant file that's totally awesome I just add a VM provision block that uses chef I'll define where to find the cookbooks I'll define where to find the role files and I say which role that VM should be and that's all I need to do after that I can just do a vagrant provision and vagrant will run chef I'll show you that in a minute with that you have a VM that has all the software packages you need that has all the configuration files you need and that's a great way to spin up VMs that look the same every time you run them or in a team setup that look the same on every development workstation gone is the problem that your development environments looks totally different from your colleagues gone is the problem that introducing a new team member to setting up workstations takes a lot of time and errors so with those tools you can now do some really awesome stuff and if you're done for example if your project is finished you just do a vagrant destroy which will completely erase your VM completely delete the VM image you've built and for a change that won't be a problem every time you need another version of that VM you just do a vagrant init vagrant up and you're at the same point as you were previously that's basically it just a few tips to speed up the process process of provisioning and installing VMs keep your building blocks local use a HTTP cache for example to cache packages so every time the provisioning installs for example the MySQL package it won't have to download that from the internet store your base boxes at the central place so all your team members can use the same box image and that don't have to download the the base box onto their workstation with SSD based laptops disk space is valuable again store your base boxes on a central location and do do the same for the cookbooks so don't download the cookbooks every time you install a new development workstation but put them in a central repository put them in a git repository for example so you can see what what's changed with the cookbooks over time I'll start my demonstration at the end of my talk because it'll take a bit of time so I'll already come to my conclusion by combining ray granted chef you have the opportunity to set up dedicated development servers that don't have to be shared between developers they are configured consistently and they are completely disposable you can get rid of them every time you want because you can spin them up every time you want and you can be sure that they they look the same all the time and it takes a lot of time out of getting new team members into the project because all you have to get them is access to your vagrant files your chef files and your cookbook repository and you're done okay let's see how that works I already have my cookbooks I already have my chef file and I already have my vagrant VM running in the background so all I have to do is ray grant provision that's now running inside the VM so I just do a ray grant SSH really running no it isn't so the most easy thing and just restart the VM ah okay yeah there's a step missing anyone noticed what I forgot to do yeah I didn't connect the ray grant configuration with the chef configuration everything's in place now I just have to connect those two in the ray grant file so let's go down to the chef configuration there's a puppet installation as well for those of you who already use puppet so here we go I'll just need those modify them a bit since the cookbooks and roles directories are in the same directory I just need that and then I define that VM as a Drupal VM I just have to add the roles directory get the Drupal role let's take a look at that again here's the role that defines which cookbooks and which recipes I'd like to use I could add attributes here as well so if I run my Apache web server on port 81 all the time I can define that here or if I have special configurations that regard the PHP extensions for example if I want to define how much memory the APC cache is supposed to use I can add that here as well without defining them chef will always use the default settings defined in the cookbook and now I'll do a ray grant up which will start the VM again should take less time now and it'll also see that I have a provisioning block in place and after booting the VM it'll immediately immediately start the the chef run which should be the stuff they are included yep here we go it uses the chef directories as share folders and now the chef process has started and and it's now processing the package Apache 2 action which means use apt in that case to download and install the Apache 2 package since all those downloads take a lot of time on the Wi-Fi here I decided to put that thing on to the end of my talk and it'll go on installing packages creating configuration files starting up services and if the chef run has finished successfully I'll have a basic setup running Apache PHP and my sequel of course to get a development VM you will also add configurations you may want to create document root directories maybe even download you may even want to download additional dupal packages installed rush and do things like that but that's only variations of that thing that error is quite common with the new Ubuntu version and I'm quite used to that now all you need to do is update the package repositories vagrant takes care of that it'll use your ssh key by default and also do a port forwarding from the VMs port 22 to your local port 22 22 and connect then to that so that's what happens if you do an ssh you could do a ssh minus p 22 22 vm ip as well but that's a lot more effort than just do a vagrant ssh okay so we now need to start another provisioning run and since the Apache package for example is already installed it skips that step the php module will continue to do all everything that's defined inside the cookbooks we'll take a look at that in a minute okay so how did I get here let's do a step back here's the the drupal roll file again and here's the step I missed at the first time to integrate chef with vagrant in the vagrant file okay and that's basically it from then on you'll be able to create your own VMs without much effort all the effort goes into define your basic environment one time share that definition with all your colleagues and project members and then just use that handful of commands vagrant up vagrant provision to get your VMs running that's all there is to do and so you can focus again on building awesome drupal websites thank you the question was how do I handle the subtle differences between those nodes between those VMs you do that using the chef attributes every chef node so every chef every server you deploy with chef has its own attributes and you can define those attributes either on a node by node basis so you can say okay on this node apache runs on port 80 on another node apache runs on port 81 if all your VMs are equal in that way so if you say my apache always runs on port 80 for example which is the default so let's say apache runs on port 81 you put that attribute into the role so by assigning that role to your VM you not only assigning it the list of recipes to execute but also those special attributes that most of the time land somewhere in your configuration files that's what node attributes are for there are many ways to do that and that depends on your development cycle the question was how now if I spin up that VM how do we do I work with that do I have to SSH into the VM and edit PHP files there or how do I do that that depends on how you are used to doing your development the most straightforward way of course is doing an SSH into the VM doing all your changes checking them we are the web server and seeing if if the result is what you want that's a bit tedious I think so you could as well use the shared folder option of virtual box where you can define a folder inside the VM to be accessible on your host system on your laptop for example and if you so you can use for example the the tools that are installed on your laptop to edit files to change files and that way you may only have to log into the machine to restart the web server do things like that yeah that's for example how vagrant made the cookbooks I have installed locally on my laptop to the VM it just created a virtual folder so the chef running inside the VM could access the cookbooks I had installed on my laptop here so the question is if I understood you correctly how do I get a quick start with an existing development environment and make that able to replicate between VMs most of the time you have to put in the effort of defining all these parameters for one time we are planning for example to provide our customers with special cookbooks that are based on the cookbooks we are using to deploy our production system so these cookbooks are simplified versions of what we are using on our infrastructure and our customers will be able to use these cookbooks to spin up a development environment that's in many parts is completely equal to what we are running in production so if you take the effort of defining that for one time you'll have a lot of productivity afterwards yeah blueprint is a kind of re-engineering your existing system looking which packages are already installed but that's already if I think it through where the problem starts so it can't just use the complete package list and create a recipe for that for example if I install Apache the Apache package all the dependencies will be automatically be taken care of and I don't want to have a huge list of dependencies explicitly listed in my roll file I think it could be a quick start you will have to um clean out the the result of that and I don't actually know which is more effort starting from scratch or starting from such a blueprint just come come forward to the mic so we can hear you let's take a look at it we are almost done by the way the question was well if I if I'm done with developing how do I get the results I've achieved on my development vm up to the production environment the answer the short answer is that depends it again depends on how you do your deployment for example you could use Drush locked in locally to the vm to do a sync db to upload the files on your vm to your production server in our case our complete deployment is git based so you will access the git repository used on the development machine and just do a git push to the production environment there are all kinds of ways to to achieve that result and it really depends on what environment you use in production who's your provider if you are running it yourself so if there's some kind of p a s s ours for example and um uh basically you have all the the tools at your disposal uh you can rush install drush on your vm here um you can use ftp outside the vm if your vm is connected to your local network you can use just ftp to to upload your files to a production system there are all kinds of ways ah okay how how do I make sure my development environment looks like the production environment um the best way to do that is of course use the same cookbooks uh that you are using here locally on your machine uh using the chef server and client system which is a bit more complex um to run uh the your infrastructure on the same code base so we are back to the infrastructure as code um for example uh the same cookbook i used to spin up apache here is used in our production environment to uh run apache there by making sure we are using the same cookbooks and the same cookbook versions on both the development and the production environments we can be completely sure that they are identical no we don't um you can use just the chefs we are running a dedicated chef server which has a huge database of all our nodes all the attributes those nodes have and so on and on every server uh we are running our infrastructure on bare metal servers not on some kind of VMs um yeah yeah we just do a chef client run which connects to the chef server gets all the cookbooks gets all the um attributes and all the things we've defined over time and then it'll run the same installation process you can see here yeah basically if you're using that uh toolchain here for example you would just add another cookbook or another recipes to the role file and if if your convention is to use that role file all the time by updating the role file and running vagrant provision on the different VMs um it'll install your new package do your new configuration files or things like that you just share those basic coded definitions like the role file and uh chef and vagrant will take care of that so let's just uh the chef run has finished by the way just check back with that this here's the central part the role file which defines which recipes to run maybe also which attributes to use when configuring those services so by maintaining that file and sharing it between your developers um you can make sure that every development workstation every development VM looks the same what's happening here there we go that's what I would suggest yeah just create a central repository on a file server order on a central git repository and maintain your vagrant files and especially those uh role and recipe files um on the git repository and tell your colleagues to do a git pull from time to time and do a vagrant provision from time to time a provisioning run by the way is also done every time you vagrant up so when you're you're restarting a VM it'll also do a provisioning run so if you make sure that cookbooks and roles are up to date all the time you can be sure that your VMs are too uh no it doesn't it's completely based on virtual box uh you you'll basically do the same as we are doing in production you just um run chef directly on your uh VMware provisioned VMs you just skip the first half of my presentation and go directly to the chef configuration um since VMware already provides you with VMs based on a central image most of the time you'll just do the configuration management then Chris any more questions that's an interesting thought and I I think it comes down to DevOps culture um I can't speak for a corporate IT department but I can talk about opportunities here um using a tool like chef that uh uh that focuses system management down to maintaining text files uh at least enables you to participate in in this system management so for for example you could really provide your IT department with a such a role file for example telling them okay look uh we need to download and install a patchy we need mysql um and um we'll need php uh php should be configured that apc has 64 megabytes of uh of RAM at the least things like that uh which you as a developer can define better than the IT department that doesn't know about your application um they as well can then um define other things like uh buffer sizes in mysql uh or some some really uh uh down to the metal things that's the IT department is concerned with not you as a developer as much and uh you can throw your know-how together in those files um so you don't the IT department that doesn't need to give you SSH access to do your configuration which will they will never do um and it also enables you to um give your recommendations for the environment in a more um IT like way than just emailing them so I think tools like chef at least enabled you to uh do dev ops together instead of you doing your dev stuff and they doing their ops stuff mm-hmm they they are solutions for the same problem so you can just replace the chef configuration in your vagrant file oh that wasn't it uh I'm inside the VM if you look closely you'll find that before the chef configuration block there is a puppet configuration block so you can just use that block and define a puppet um you define your puppet configuration for example the address of your central uh a puppet server and use that to spin up your VMs yeah yeah yeah the default configuration as well is with puppets is to run a local puppet binary that's not dependent on any outside servers uh uh as was my chef configuration as well you didn't see me configuring some kind of chef server or something like that but you can do that if you have a puppet server or a chef infrastructure in place you just point vagrant to that and it'll take care of it basically you're not thinking in terms of commands you have to execute and files you have to edit uh you uh that boil everything boils down to maintaining those text files like the roll file um and you can use all the tools you have at your disposal to share those files create central git repository send them via email um work on them in a distributed team everything that's possible uh with a text file you can use to your advantage anything else that prevents us from going to lunch okay moses uh we can do that afterwards thank you