 Yeah, so my name is Kay Williams and thank you all for coming out late in the afternoon like it is. Today we'll be talking about the Deploy Project and then specifically how you can automate cloud testing and deployment using Deploy. So a little bit about me. I have been in software development for many years, about 25 years. I spent a long time at Microsoft actually, worked on Office, worked on Windows, and during that time actually a couple of my jobs were related to system and application deployment. And then I've been working on Linux for the last 10 years about. So about the Deploy Project, so Deploy is a software deployment platform. It deals with both system and application deployment. It started in 2004, which was shortly after I left Microsoft. And it started off as not specific to deployment. We were, myself and a couple of other folks, were working on building a personal web server. And what we needed, and this is how I got started with Linux. Linux happened to be an operating system where the price was right and there were a great number of tools that we needed to pull together this personal web server that had a base operating system, applications, the applications that we wanted, some custom configuration. And then it would be able to deploy that and maintain that and user systems. So a couple of years after we got into it, Microsoft announced that they were building a home server product and we did a quick calculation on the laws of tonnage and we decided to go a little different direction. And so then we focused on the deployment aspect of it and that's what is the Deploy Project today. So it is a community project. It has contributors that are current and past, just individual contributors from Microsoft, Amazon and a number of companies that you'd recognize. And the product supports currently the CentOS operating system. And Red Hat Enterprise Linux. So Deploy is a deployment automation platform. It supports software packaging and creating package repositories. It deals with configuration management. It will automatically create and maintain testing environments. And it manages deployment and ongoing maintenance to production systems. And it does all of this in an integrated process that's seamless and continuous. So one of the first questions I usually get asked is, how does Deploy compare to some of the other system deployment solutions out there? The main reason is that it's a complete solution. So it deals with installation and updates, desktops and servers. As I said before, applications, configuration and operating systems. It can be used for physical, virtual and cloud deployment. It can even create CD-ROM installation disks, both mini network boot disks or full complete operating system installation disks, and it deals with automated testing. The other thing, well, okay. So another reason is that it's standard. So it uses standard deployment formats, both for the way that it creates the systems. They end up looking like basic operating systems so that they're easy to deploy using existing infrastructure. And then it also uses standard packaging formats for its application packaging. It is a completely open project. It's open source, vendor neutral and nonprofit. So how does it work? It has a fairly simple command line interface. You start off by creating a definition. And a definition is a file in XML format. It lists everything that's needed to build, test and maintain the type of system you want to build. And to create this definition, typically you'll start with templates of which a number of them are included with the product. And we'll see some of those later when I do a demo. Then once you have your definition created, you run deploy from the command line and you provide the template that you wish to use. Okay, and then deploy, get started to do its thing. This is just an overview and we'll go into more detail on these steps in the following slides. So the first thing it does is it creates a system repository. And this includes packages as well as installation images. Again, more details later. But it's everything needed to install a complete system. The next thing it does is it creates system installation and updates. And it does that by creating new machines, by installing, again, I'll save that for the next slide, or for a future slide. The next thing it does is it will deploy production systems. And then it will, on an ongoing basis, you can continue running deploy to maintain systems. Okay, so the first step is it creates a system repository. This is essentially a mini operating system distribution. It looks exactly like a version of CentOS or red hat looks, except typically it's much smaller because it just has the packages you need for the system that you're building. So it contains RPM packages and installation images. The next thing it does is it will do test install and test updates. And to do this, it creates separate machines for you. And those machines, they can be virtual machines running on your own, like on a laptop or a desktop of your own. Or you can do tests and installs to cloud-based services. And we'll show both of those when we do the demo. To specify the deployment environment, you use deployment templates. And those are environment-specific. So if you're deploying to a Libert machine, sometimes you need certain things on the client machine. Or if you're deploying to some other environment, you might have different requirements for those clients. And all that gets contained in the deployment template. And the deployment templates specify the exactly how you would install or activate or delete systems for the different environments you might use. And then once we've completed, once deploy has completed the system install, then it runs test scripts. And we run some different scripts in different cases. So in the case of the test install, since we're starting off with the package repository, everything, like even the configuration you specify, we've turned into RPM packages. And so when we do the test installer, the test update, we're just working with RPM packages, which makes the testing much simpler. We'll see that more in the demo. But so in the case of test install, after the install's completed, we can do a simple scan of the install.log file to see if any packages failed during installation, or if any scripts failed. And if so, then we can stop the process. We also will do a test removal of packages at that same point, so we can see if anything is going to fail during package removal. And then in the update case, for test install machines, we create the machines, and then by default, we just delete them afterwards because the purpose was just to perform the install. For update machines, they're long lived, so you do the install once, and then as time goes by, and new packages come out, we're checking to make sure that the packages apply over existing ones properly. So those are the things, those are the tests we do by default, and then users can provide scripts to test their applications to ensure that they're running properly. Okay, so then during the deploy to production systems part, the one thing that we do every time is we'll make a copy of the system repository and put it on a web server. By default, that's just the web route on your local machine. And then from that web server location, again, you can download the install image and install it any way you want. But then in addition to that, you can use deploy to manage production system deployment, and that's done using the same deploy templates that you would use for testing. So the last step is the ongoing maintenance, and so over time, probably in a cron job, you'd run deploy to keep updating the system, and each time it runs, it checks to see if there are any changes. So if there are any updates from the operating system repository, and if so, it brings those in, recreates the pack, it's system repository, retests everything on the test systems, and if that all goes through fine, then it will apply it to the production system. The other thing I'll mention here is that because everything's in a package, or just an RPM package, is the updating is done using YUM, and that can either be done by scheduling YUM updates on the client system, or you can use deploy, but in that case, the default way we do the updates is just SSH to the client machine and run YUM on that machine. We include by default a plug-in for YUM that's called the sync plug-in, and that plug-in sets up the system repository as a master repository, and so when we run YUM, we just do a sync. So any changes in the master repository are brought down to the clients, and any changes, any packages that were removed from the master repository will get removed from the client as well. So now we're going to jump out of slides and head down into command lines and text editors. We're going to create a Drupal server, and in that we'll show configuration templates. Initially, we're going to deploy it to virtual machines using Libvert, just running on the local desktop. We'll run deploy to create the machine and those test machines as well, and then we'll have a look at the results, and then when we're done, just checking my time, we'll try the same steps by switching and using a deployment template that points to the Windows Azure environment. And by the way, I used Windows Azure just because they gave me some free space up there. If anyone else runs a hosting service and would like to give me free space. Okay, so here we've got a Drupal server definition. Should I make that bigger? Yeah? All right. Is that okay? All right. So let's have a look at the definition. So this is an XML file. Let me get to the top. And inside of it, the first section we're setting up some macros, and the macros just define some content that get reused across templates as a way of sharing information. So it says the name of the server or the system we're creating is a Drupal server, the operating system, the version, and architecture. And then we're setting up some macros that get used by the Drupal configuration template. Next, we've got some macros that are used for deployment, and one of those is the one that says we're going to use a Libvert deployment. And then here, this macro defines a script that tests the Drupal system. And this one's fairly simple. It just brings down the web page and makes sure it's got the site name in it that I gave as the site for my Drupal web server. Next, so this is the meat of the definition. And so we just set up the metadata, the name, et cetera. The next thing we do is we define the repositories we're going to use. And to do that, we're just including another file that's got a list of repositories defined in it, so base URLs, GPG keys, et cetera. And then we're telling it from that file we want the repositories named base and updates. The next section, we specify what packages we want to use. And we're defining this. Packages can be brought in in a couple of ways, one by specifying it here in this section, but the configuration templates can specify additional packages. So for that reason, we're just specifying a single group, which is the core group, which will end up bringing in about 225 packages. The next section is where the config RPM section is where we define the system configuration. And here, again, I'm just including another file that defines the configuration. Actually, this template is a meta template that points to some additional templates. We'll go take a look at it. So here in this template, we are pointing to some additional templates. We'll look at one of those as well. But one sets up the web server. Another sets up the Drupal specific configuration. Another is setting up the database that we'll use for the server. And then we also have a couple of templates that are setting up the backup, the database backup on the system. So let's look at one of those. Okay, so this is a template that sets up the web server configuration. So it specifies some requires, and the requires can be in the form of other packages that are required or binaries that run on the system. So for the web server, we're bringing in HTTPG IP tables for the firewall and a couple of other things. The next thing we have is a file. And so this specifies a file that will get installed on the client system. And we are including the contents of that file here inside of the HTML, but you can also have the files residing separately on the disk and have us pick those up. Then for the web server configuration, we also are running a script. So after this package gets installed and the files installed on the system, then this script will run. And the script is just starting the web server and setting it up to start at system start each time and then setting up the firewall. So that's a basic configuration script or configuration templates and scripts. The last sections in this file are the system deployment sections. And so there's one for test update, one for test install, which I'm not going to show you now because it takes a while to install a system and it's not that interesting, and then publish. However, if you are interested in seeing an install, us deploy doing an install, we do have a booth upstairs so you can swing by the booth. Okay, so here in these sections, we're including a deployment template and it's just going to grab the contents of the macro that I set up earlier. So it'll use libvert and then a test update template or the appropriate template for the section we're in. And then we're also including that test, the custom test, that we had to check to make sure the website had the right information on it. And let's take a look at these templates. Again, they're meta templates. So let's look at. And so the libvert template will pull, pulls in a kickstart and then pulls in a number of other templates, some of which are common so we use them across all the deployment templates and those are the things like our basic test to make sure that the install log doesn't have any errors in it. Some of the templates like these, the ones that are in the libvert folder are specific to deploying in a libvert environment. So let's take a look at the install.xml template for libvert. And here we just have, it's a bash script, a shell script and it's running libvert and providing it the path to the system repository. So libvert will just run and perform the install. In the case of a different deployment template, so let's look at the windows azure one. Windows azure has a command line utility that can be used for creating systems, yes? I'm sorry, so your question was what is the template for testing? So it, maybe that will make more sense when we actually run it and you can see. But basically the template for testing it, it just runs through a system install and then inside of that, inside of where you specify test install, you can include additional scripts that do any additional testing that you want to do. So if you've got a custom application and to make sure that it's installed properly, you want to run test A, B and C, you list those scripts inside of the definition. Okay, so anyway, so for an azure install, they have a utility that's similar to vert install that will create new azure systems. Okay, so now let's try running deploy. And the deploy application is highly cached, so when you run it, it's tuned to check for changes and if you run it two times, one right after another, the second time it'll notice there were no changes and so it doesn't do anything. So that you can see something more interesting, I'm going to force it to do everything from scratch. So the first thing it's doing is bringing down the repository metadata for all the package repositories it uses. Then it's taking all of those configuration templates and creating appropriate RPM packages. Once it gets through with that, it reads the kickstart that was provided in the deployment template and writes that out. Then it downloads the install images from the operating system repository and it will customize those to be specific to your server. The stuff that Word by just now, where all the packages that we downloaded from the base operating system repositories, they're very fast because that's all cached as well. Now it's... Yes, it does. Yeah, so what you could do, let me kind of keep talking through what's happening here, but I'll come back to that. Okay, so then now what it's doing is it's going on to the test update section and so we're going to switch over to a different application so we can see what's... That's small, I know, sorry. So the lower one is the test update server and we've created it before, so that's why you didn't see it run the complete install. Usually when we create those test update servers, we shut them down afterwards, figuring that you don't want to keep them running it all the time. So when we do the test update, we first activate the system and that's why you saw it coming to life and then in a minute after it's finished running the... So right now it's running the update script so it's just running yum on that system and then it'll run the other test scripts, some of which we've defined by default and some of which you might define. So one was the Drupal test script, it ran that and then when it's done it runs power off and then that machine will go to sleep and since everything went fine, now it's moving on and running the deployment scripts on the production system. Okay, so I can come back and answer your question about what if you're using a different configuration management system. So let's say for example you're using Puppet for doing configuration management. What you could do here, so instead of defining all of the Drupal server information in our XML format, you could just have one configuration RPM, one config template that sets up, what did I say, Puppet, and specifies the, is it a channel, I forget how they call it there, what Puppet channel to use, what's that? Yeah, so you could just have, that could be all the configuration you specify here and then when the machine gets installed Puppet gets set up automatically and it picks up from there. Yes. I think the biggest, so that works great in the, oh I'm getting the, oh I'm sorry, I'm supposed to repeat the question. Okay, so the question was how is using deploy different than using something like vagrant where it spins up the virtual machine and then you use Puppet after that. I think the main difference, well I mean it's a different approach for one where it creates the system and then does the install of that. But in a, if all you're doing is managing cloud machines or virtual machines, then that environment, or even I guess if you're doing physical machines that environment works fine. The main thing that deploy would give you over that is that it gives you tighter control over the content of the machines right from the start. Things ever gonna get on that machine that wasn't part of what you defined initially. It's a little more efficient that way too, possibly. Because if you're using, but it depends on how tightly Puppet is managing the machine if you end up removing a bunch of packages when it takes a long time to remove the packages off of an existing image. Or you can create the image yourself beforehand if you want, this would help you create the images that you wanted to use as your starting images. And in fact that's another way you could use this is to create images that then you start from that stack of images and deploy out. Okay, so let's go back now and so we can, oh well I'll just bring up the, so here's the site that I created. It's a very simple Drupal site and it's got my Drupal site on it. I'd show you the test update site but we've shut it down so it's not active at the moment. Let's go back and change this so that it uses the, rather than Libert, we're gonna try it to the Windows Azure environment. Make a mistake, so it's telling me. Somebody saw that right when I did it. Okay, so in those templates for the Windows Azure environment there's a different repository that I need to include and pull packages down from. So Deploy is noticing that change and grabbing the new repository. There's some other packages that need to be installed on the client machine so it's setting that up. It's downloading those packages, updating the repository and now it's getting ready to do the test update. Since the update server was also shut down it's running an activate script to bring it back to life and I'll switch over and I just, sorry this is gonna be small and hard to see until I fix it up. So this is the list of machines I have running in the Azure cloud and the top one was stopped. It's still stopped. Sometimes it takes it a while. Okay, it started it. You can see ready roll now. We don't have a nice little visual that shows the lines going on it but essentially it runs the same process and then now that machine up there is on cloudapp.net And we'll take a look at our update machine. Anyway, we won't watch the rest of that. You can sort of figure what's happening but so that's a quick overview of deploy and how it works and how you can use it for managing cloud deployment. So we do have a booth here. It's up in the exhibition hall on the right-hand side or left-hand side when you go in the door and you're welcome to stop by there and I can go into things more in depth. You can download the product or view documentation and there are also mailing lists available from the deployproject.org site. And that is a wrap-up of what I have to show for today. Any questions? Go ahead. It is. It can be extended and we'd love to do it and the architecture is pluggable so really to be able to support Ubuntu, for example, we'd need to be able to create packages in a dev format and we'd need to be smart about the right update package, an updater program. So it can be done but it's not as easy as writing one of these templates. Other questions? All right then. Thank you very much for coming.