 Welcome, everyone. So before we get started, let's get to do something. I've got a seven-year-old and a nine-year-old at home, and they have no idea what I do. So I just got to get this going to make sure that I can help them out. All right, thank you very much. So we're going to talk today about rolling your own images with Packer using OpenStack. A little bit about Spencer and myself. Spencer is a cloud engineer for Selenia. He came to us from Boeing. He helps our customers with both OpenStack as well as DevOps process. Similar to some of the stuff we're doing here today. We've done what we're talking about today. We've done with several of our customers in the field today. I'm the VP of Operations for Selenia. Been working with OpenStack since about 2010. I worked with some of the largest public and private clouds around the world. So just a quick word from our sponsors. These are the nice folks of Selenia that brought us here today. Our focus is open infrastructure. We're purpose-built around cloud computing. Our team comes out of the OpenStack community. OpenStack experts have been contributing to the project basically since it started. We've got a proven delivery model, some differentiating IP. Goldstone, if you had an opportunity, stopped by our booth earlier this week to help people operate clouds at scale. And our team comes out of the cloud computing space. We understand how enterprises take on this new type of infrastructure. A couple of customers, we're working with Deutsche Borussia in Germany, financial services space. We're headquartered in San Francisco. We've got office in Seoul and the US soon to be Japan as well. So what are we going to talk about today? Image building is a pain, right? We all have to do it, maintain these images, get them through the environment. So this is what Packer's about, and we'll give you a little bit of history on where it comes from. We'll get an introduction to templates where we talk about how they're built and how they're constructed. Some of the building blocks of a template, a builder, helps you understand or build for a specific cloud. Packer's designed to work with OpenStack, as well as Amazon and others. Provisioners, what are you going to do to your image as you're constructing it? What are you going to add to it? What configuration changes you need and so on? Post processors, where does it live? We put it into glance in this case. Spencer is going to test the demo gods today for us and do an actual live demo of this running through. He's going to show you how to install it, get OpenStack ready, get all the information you need out of your OpenStack installation to actually do it. Go through some building basics. We'll do testing, because if you're not testing, you're not working, right? Everything has to be tested. Code isn't done until it's tested. And go through and actually build a base image. So when we talk about image building stinks, we talk with our customers about baking and frying. No software project is done without a good metaphor. So we talk about baking and frying. It's a very simple example. When you're frying, you're adding ingredients one at a time and you're cooking the dish real time. When you're baking, you assemble everything, you put it all together, you put it in the oven, and it comes out all at once. So there's no sort of complicated process. The advantage you have over baking and frying, when you're frying is really about bringing a VM online to complete the metaphor and installing stuff as the machine comes online. When you do that, you have to test as it comes online. So you get the advantage of not having a test. When you test once, when you bake, you add your functionality to the VM, you do your testing, you bake it, seal it up, deliver it, and you're able to make sure you use it over and over. The other advantage you have is the virtual machines will come online significantly faster. This is pretty critical when you're looking at an autoscaling environment. You want a VM to come online and start taking workload very quickly. If you're not frying things into that VM as it comes online, that VM can come up and really take on that load much more quickly. Typically, when you're looking at building an image as a collection of bash scripts, you hope they run. Maybe you test them. Maybe you don't. You put it out there and see what happens. Base images tend to be different. They drift over time. The idea behind Packer gives you that model of infrastructure as code. We all talk about it. This is a great way to codify your image building process. Take a Packer template, keep it in Git, version it when it changes. You know when it changed. Git is good about telling you who changed it and how they changed it. And you're able to get that history of the virtual machine. Typically, if you're just versioning a Kukau image, for example, that are much harder to version control, with the Packer code being in Git, you're really able to get that going. And then you're wondering where your configuration management scripts live. Are they on a dev box? Are they on my laptop? Where does that live? And you want to make sure your virtual machines are spinning up the same way every time. And once you've baked in those configuration settings, you're really able to have that consistency in the environment. The other problem when you sort of do this without these tools and Spencer will show you all this is a great log output. So you can really see what has gone on here. You can use it. Some of our customers are using that as an audit process. What actually happened to this VM when it came online? Did the image get built correctly? Or you can really use it to get through the whole process. So what is Packer? Probably a lot of people have used Vagrant before. Packer is developed by the same group at HashiCorp to bring this to market. It automates the image creation process with a wide variety of clouds, not just OpenStack, but it'll support VMware environments. It'll support Amazon, Compute Angel, including also do Docker images. So you can actually have version and control Docker images the same way. The templates are easy to write and version control. And Spencer will show you that. You'll see the templates. They're really straightforward. And you can actually create several images at the same time. So if you are supporting a hybrid environment, you need to output images for two cloud infrastructures. You're able to do that with Packer as well. So you have that consistency. And we all talk about immutable infrastructure. How do you get there? This is it. We have that straight through path and everything follows the right course. You know what your infrastructure looks like all the time. So just a quick intro to templates. They're JSON formatted, very straightforward. There's three major sections to them. And Spencer will show you how those work. There's builders, provisioners, and post processors. And you really just sort of, it's almost like a fill in the blank. And you can use a lot of different tools. The example today that Spencer is going to show us is all about Ansible. But you can use just about anything you need in that location and sort of fill that out. And the post processor tells you where it's going to live. So with that, Spencer's going to start testing the demo gods for us and run through. Yeah, so first I'm going to talk a little bit about the different building blocks that Spencer sent in. The first is builders. And that's going to be something that's required in your templates every time. You've got to have some cloud to build against. And it can be multiple clouds at the same time. So it can be a Docker image. It can be an OpenStack VM, Amazon, Google Compute Engine, VMware, if you're doing that stuff. And if you'll take a look at the code block on the right, this is going to be, this is what I actually used to talk to our OpenStack cloud and our test cloud. So there's some things you have to know about your environment going in. And I'll show you guys that in a second. Under that, you'll also see a Docker type. So you can base it off a Ubuntu image, export it as a tar, and then take it wherever you want. The second thing is going to be provisioner. So this is going to be our do stuff portion of our image bake. Packer is going to log into our instance via SSH. And it's going to provision it the way that we tell it to. Provisioners run in serial one after the other. So you chain them together to do all your stuff in one go. Like I said, when we do our second example, you'll get a better idea of this. We're going to do several things. And so there's lots of options. File uploads, shell scripts, Ansible playbooks that we're going to use today. We're going to use all three of those. You'll see them. But you can also do Chef, Puppet, Salt, whatever you've got going on. And then this code block. So just to give you guys a snapshot of what provisioners look like inside of a template. So the top one is just uploading a file to a known path. So I've got a tar archive. Give it a source locally. Give it a destination. It plops it out there for me. And then we can act upon that. And then secondly, we can do a shell. I'm using the inline shell here. You can also, if you've got scripts already written, you can simply use those. But here I'm just doing an app get update and installing Ansible. And then finally, a post processor. So this is after your image bake is complete, what do you do with it? You're not going to use it for OpenStack or Amazon, actually. There's nothing out there for post processors to use. But if you're doing things like VMware baking, you can register an image in vSphere. If you're using the virtual box builder, you can create vacant files, which is pretty interesting. And then the example I've got here is using if I'm doing a Docker image, I can tag it with a version number of the repository and then push it to the hub or a private registry that we're hosting internally. So installing Packers is super simple. It's a zip file. You unzip it, throw it in your path, and you're ready to roll. That's the Linux way to do it. If you're on a Mac, it's just a brew install. And there's a couple of installation links there, too, if you guys are interested in that. So this is the way that we gather the info we need for OpenStack, right? So the first thing is make sure you've got SSH allowed in your security group. You're going to hit that the first time, or at least I always do, right? A new cloud baking an image that's going to fail the first time, I haven't enabled SSH. And then Packers is going to use the environment variables from your Keystone source file. So source that stuff. It's going to know your Keystone endpoint. It's going to know what else is it going to get out of there. Your admin credentials, all that good stuff. Then you're going to need some unique IDs. So the example I'll show you today is basing off of a Ubuntu image that I just downloaded from Ubuntu's cloud website. And I'm going to find that image ID with a glance image list. Then we're going to get a desired network with Nova Netlist. And then in my environment, an instance will come online, connect to the private network, and then I'll give it a floating IP, right? So I can connect to it. And I'm going to find that with a floating IP list. So this is going to be, I'm going to show you two examples, or try to show you two examples. We'll see. The first one's going to be super simple, right? So I'll walk you guys through this. I'll watch it. And then we'll go through, we'll talk about testing, and then we'll look at it. So this is just a builder, right? So you have to have a builder. It's all you have to have. But this isn't going to do anything interesting, right? It's going to launch an instance. It's going to give it an image name, and it's going to take a snapshot. And then it's going to tear it down. That's pretty much it. So I'll kick this off. Like I said, we'll see it. We'll get familiar at least with the output. And then I'll talk about testing. So the first thing I'm going to do is source my Keystone files, and then I'm going to... So I'll let that run in the background while we talk about testing a little bit. This one's really quick. Like I said, there's no provisioners. It's going to be pretty simple. So the first thing, when it comes to testing your templates, is using Packers built-in. So there's a Packer command called Validate. It works pretty well. I haven't had any issues with it, right? It's going to let you know if there's things wrong with your JSON, something's formatted the wrong way, if it expects anything to fail. So if it returns successfully, you get the output we see in the code block there. And that's from the example one that I just built. Otherwise, the check JSON is a JSON.lint.com tool. I haven't used that that much for Packer, but it's out there. I've used it for other things. It's handy. And then your provisioners, if you're going to test those guys, it depends on what provisioner you're using, right? If you're using Bash scripts, that's different than testing Chef Cookbooks, Ansible Playbooks. Testing Bash is hard. But Ansible Playbooks have a dry run built in, the dash, dash check flag. It'll do a dry run and test that for you. And then finally, trial and error is helpful, right? To give it a shot, build an image, see if it works, see if it performs the way you expect it to, throw away the instance after you're done. Sorry, I keep going the wrong way there. So we can see we're finishing up here, Glance is taking our snapshot. But you can kind of see here what's happening. So OpenStack first is going to verify the flavor that I've passed in. And that flavor is going to be a small, right? It's going to find that flavor, make sure it exists, and create a temporary key pair for us. And so Packer will create a key pair and tear it down after it's done. So you have to mess with keys and all that good stuff. And then it's going to launch my server. So once my server is up and running, the floating IP we talked about gets attached. And then when you get to about here, you see we connected SSH, and then we stopped. We didn't give a provisioner, so there was nothing for it to do. Once it's up, SSH didn't, it's got nothing to do. So we can see that as far as output, it created this image, Selenia example one, and then gave it a UUID. So if we head out to our OpenStack installation, I can see that it's been created, maybe. So there it is on the top. It's a one on top. I've baked several images today in preparation. But it's going to be that guy right there. So it's there in existence, ready for use. And so what I'm going to do now, actually I'm going to go back a slide, but I'm going to kick off the second example because it's much longer than the first one. We're actually going to do some provisioning and I'll walk you through what we're going to do. So I'm going to throw a time command in there just so we know how long it took. So we'll let that run in the background while we walk through the things we're going to do. So hypothetical example here. Say we're creating an enterprise image for our business. We need things baked in, certain things we want all of our images to have and inherit. So the first part's going to look extremely similar to the last part, the exception of the image name. So I'm using a, Packer has some built-ins around, I can timestamp my images, I can use some variables, things like that. So it's another kind of little more advanced feature to explore, but I give a date, so we know what one is created. We know we're using the right image. And then I'm inheriting this source, which is the Ubuntu image that I talked about earlier. And then the networks, the floating IPs, all that stuff's the same. So then we get into our provisioners, which is a little new. So we do the file type and I'm going to take several different files and I'm going to put those into temp ansible. And so those files are going to be an NTP configuration. Message of the day, we're going to add that into our image and then something else I keep forgetting, DNS servers. So I'm going to point my image at Google's DNS server just as an example. And then we're going to roll into a shell script. So this is an inline shell, Packer will package all this stuff up, SCP it over and then just run it as a shell script. So I'm going to sleep for five seconds, Packer sometimes, as your images come online, or your instances come online, it gets a little bogged down, so wait a second basically. And then we're going to do an app get update and install ansible. So ansible will require several different things, it's kind of too long of an install really, but install a C compiler, we install the Python DevTools and then we'll do a pip install of ansible. Once ansible is installed, we'll run the ansible local provisioner. And so the way this works is is this playbook file that I'm passing in gets SCPed onto the instance and ansible will run that on against the local host and then it will clean it up after it's done. So here is the playbook that we're going to use. So if you've never seen ansible or YAML files, it's extremely straightforward, it's kind of self-documenting, which is a good thing for your enterprise. So you'll see that it's going to run against local hosts, like I said, it's going to use a remote user Ubuntu and then it's going to have root access. So some of the things we're doing require root, so it's going to use it. And then there's several tasks. So like I said, we're going to build an enterprise image and we're going to put our message of the day out there with a Selenia banner saying, hey, this is our image, unauthorized access is prohibited. And then we're going to create a temporary, a standard username cloud user and make sure he's present on that system. So I'm using a state equals present there, just kind of interesting, right? So ansible will look, make sure that user exists. If it doesn't, it will create it, if it does, it won't mess with them. And then we're going to inject an SSH key for that cloud user, right? So I've trimmed the SSH key out because it's extremely long, but it's my public key, that way we can SSH in after it's done. And then next we're going to add a Java app repository. So this is a pretty common use case for a lot of different companies, right? So if I'm hosting app repositories internally, I'm hosting yum repos internally, I can use those, I can connect to those. This is a public example, it's just the Oracle Java installer, the app repository for that, but you get the idea. So next we're going to do some hackery to accept Java's licensing. It gives you a terminal pop up, you have to accept it, you have to agree to it. We're going to use debconf to work around that. And then finally we're going to install the packages that we want on our system, right? So it's kind of a, this one's kind of weird looking I guess compared to the other ones, but you'll see that it's going to update the cache since we've added a new app repository, we need to do an ad get update. And then it's going to iterate through those three items and install those packages for us, right? So NTP, UFW uncomplicated firewall, and then the Java 7 installer. So once that's done, we're going to lay down the files that we want to use. So I talked about the NTP com file. We'll give it a source of temp ansible. If you remember from earlier, we put it out there the first thing, and then we're going to put it in the scntp.conf directory, right? After that, we're going to update our resolve com with our custom DNS servers. Like I said, these are just the Google servers, but if you've got your own, your internal, your data center, you can use those. And then finally, we're going to turn on our firewall, make sure it's enabled, make sure logging's turned on, and then we're going to allow port SSH port, and that's the only port that we're going to allow through our firewall. So that said, let's see if it's done. It's creating the image now. So we can walk through kind of what's happening here, and then we'll launch it and kind of take a look. So you can see that there's lots of logs. You can see exactly what's happening as it's running your provisioning scripts. So there's that connection to SSH we talked about earlier. Sorry, there. And then you can see that it goes through. It's uploading those files that I mentioned into that temp ansible path, and then it's going to provision with the shell script that we gave it, right? So the first thing in that shell script is simply an LS of that directory so that you can see kind of what got put out in temp ansible. So there's a base file, there's a message of the day, there's an ntpconf, and then we're going to roll into an app get update, right? So this is going to be a ton of output. And then, so that's the end of the update, sorry. And then here's us installing GCC and Python DevTools and all that good stuff. So like I said, this is extremely reverse output here. It goes on for quite some time. Once it's done with that, kind of look at pip. So here's installing pip, and then here is where pip's installing ansible, right? So you start downloading ansible there, all that stuff's getting installed via pip, and then once that's done, we finally roll into our provisioning with ansible. So now we start doing that. Our playbook gets put in the temp packer provisioner ansible local, and then it runs that playbook against the local host. So ansible's output is really quite nice, right? It sees that, you can see that here's what changed on the local host, tells you if it didn't change, it'll just say okay, kind of like the gathering facts looks here. But you can see all those things we just talked about, right? Message of the day, our user's been created, our app repository's installed, Java's installed, here's installing NTP, UFW, and then laying down our configuration files. Once that's done, the server's stopped. We've created our image with the name of Selenia example two image and a date stamp, and then we've got our image ID there as well. So I'm gonna launch this guy, we'll launch an instance, we'll SSH in, we'll take a look at it, make sure it looks the way we expect, but in the meantime, so that's gonna talk a little more about baking and frying while this launches, it takes a second, so. So yeah, I think the nice thing that you can do with this processing, maybe you have multiple groups that are responsible for different components, security being one, maybe somebody's maintaining different sets of patches, maybe you have a custom authentication module that you need installed on your virtual machine, on your VMs to have them to authenticate to the right place, right? So you can use this code concept, this code construct for people to not just build an image once, but actually iterate on an image, right? Take an image up, if this is sort of the initial step, some of the stuff we're doing here is really straightforward, but you can sort of see how the complexity can grow over time, and then people can take this image that Spencer's now spinning up and add more stuff to it and turn it over again, and people can continue to do that. For most of our customers, there's one group that basically maintains sort of the base OS configuration, and then there's a different group that handles, this is the Java application setup that we use, or this is the Ruby platform that we use, whatever those different configurations are in your environment, you can really make sure that people are using the same set of code every time as they iterate through and use the different components or add to it for the components they need to run their particular applications or functions within your environment. Perfect, done. So we just went into Active, we can kind of check this guy out and make sure it's coming online. It'll still take a second, you know, networking's got to get set up and all that good stuff, but once it's done, we'll log in. So I mean, the nice thing we've also seen with a lot of our customers in a lot of times the people that are maintaining images in an organization aren't necessarily have a ton of software development experience. The really nice thing about this process is it's very software centric, very code centric, right? It's a good, it's a good really straightforward way to get started for those groups. You're using Git, you can actually have Jenkins trigger or build, for example, you know, your images can be stored in Nexus or Artifactory, whatever that is, right? You can get all those things going and it's a great framework to get people started and prove out how infrastructure as code can actually change your environments and the way you guys operate. So we're up and running now. We'll test this first by SSHing as that cloud user we created, make sure he exists in our system. Except this type. Once VPNs are found, right? Come on, okay. So you can see that a couple of things that we added are already, you know, visible in our image. So if I do a who am I, I can see that I'm this cloud user. He exists on the system we've logged in. So SSH key is there as well as our message of the day, right? This is property of Selenia or whatever. So I'm gonna exit and I'm actually gonna log in as the Ubuntu user because some of the stuff I wanna show you requires root. So let's take a look at the things we added. You can take a look, make sure Java got installed. Seems to be working as expected. So there it is, it's installed, it's the version we wanted, it's 1.7. Then we can take a look at our NTP servers, make sure they're set up. NTPQ-P is the print flag. We can just see that our three servers that we added are there, you know, our NTPs keeping in sync. And then finally we will do a pseudo UFW status. You can see that our firewall is active. Port 22 is allowed, it's the only thing allowed. That's what we wanted. So that seems to work as expected and we can review. Sure, so for every place we work with, they always have some type of image-building process and they wanna be able to modernize it. The first thing we hear is, well, it's so hard, how can we change the way we're doing it, right? But the reality is, you sort of see the constructs here, they're fairly straightforward and you can see how they can scale out pretty easily. Packer is a great tool for this problem. It really gives, getting people working in the same code base, same code framework is a great way to make sure that this is successful. And like I said, that example where you really have the ability to show people what infrastructure as code means, right? For a lot of times in some of the large IT departments we work with, they think they wanna understand it and this is a really great sort of first way to do it. And then the ability to build multiple images at a time is extremely powerful because that will really keep you on track and making sure you're using that straight-through process. If you have to go back and forth too many times, you're not sure where you're gonna get out at the end, so keeping that running for your environment is pretty powerful. So with that, we can throw it out to some questions. We start up here. We've got a Q and A mic, apparently, too, so if you guys wanna, wait, we can catch it on the video at least. Sure. Thank you. We have seen you can configure your script to build the image, but I didn't see how to test the image in an automatically way, not just manually. Do we have any framework or tools to do that? Once the image is created, how do I go about testing that automatically, right? Packer doesn't provide you a way to do that, but you can certainly do that using Jenkins to say, a new image has been created, it's booted, let's run these set of tests that we've written. So it's kind of on you, but you can definitely do that. Thank you. Sure. Another question behind you? How do you manage your catalog of images once they've been built? Because it's sort of like we build an image and then the next day there's a security release for the kernel, and that image is no good anymore. So a lot of our customers do that differently. Packer specifically doesn't have the facility to catalog images. You can use Glance and sort of track them that way, and if you're using your name correctly and the output there, you can use that as a way to sort of manage that. There's also different service catalogs that you can use inside an enterprise to show companies, to show people this is the latest image. The idea behind some of this, if there's a latest patch that comes out today, does it need to go in the environment right now? Well, if your environment's fully dynamic, you can take that new image, publish it, and then everyone can quickly iterate and redeploy their applications. The idea behind this is to have this full CI process where you can redeploy applications very quickly and having sort of this framework gets you that and through whatever the standard catalogs are today. Have you guys had any problems with broken images on OpenStack? Like, I've been using Packer with Amazon, which works pretty much flawlessly. And then on OpenStack, our cloud continually end up with broken images, and I'm not sure if it's a Packer issue. Maybe it's a configuration issue on our platform. All right. I haven't had a ton of the images being broken. I've had some of the builders being broken. Specific versions of OpenStack and specific versions of Packer, they'll be show-stopping bugs. You've got to kind of resolve that one way or the other, right, to upgrade OpenStack or fall back to an older version of Packer where this worked, something like that. One of the things we've also seen a little bit is, you know, this is, you know, for one of our customers actually booting an ISO to get this rolling, and that ISO was mounted across a wide area network, right, so those types of things, you know, you might want to just check your environment for it, right, make sure Packer's moving around a lot of data. With Amazon, it's not, right, because it's spinning it all up in Amazon. You know, maybe you're pushing some packages out to it, but if you're booting from an ISO and doing a kickstart, there's a lot of data moving around the network, so make sure all these systems are close enough to each other and from a network proximity perspective to have the fast access that you need. A couple of things. First of all, thank you very much for coming out and doing this. It's nice to see some real-world examples when the demo gods shine on you, so I wanted to actually provide a little feedback to the question on revisioning Packer instances. I simply use version number strings within the Packer image itself, and then I create an app job that captures a Packer log afterwards and puts that into Git, so I can track everything that came out of Packer against the version number, so every time there's a change in the version number, you can do a Git pull on the log and look at what has actually changed, and that kind of goes back to your point early on for security audit purposes. You can see what's been in the Packer image. My questions for you are, I hate JSON. It's a machine serialized language. It's not meant to be a human, hand-hammered configuration language, and it really annoys me. It's not your problem, I know. But have you found any good tools within Packer? I find Packer just chucks an error that's useless for the most part, and have you found any good tools for debugging JSON syntax for Packer specifically because you have the key pairs that have to match and match with the JSON template for Packer itself as opposed to just, it may actually validate with JSON lint successfully, but it may be invalid. Yeah, I've hit that a few times where it looks fine on JSON lint. It doesn't look fine in Packer validate or it craps out when I go to pay. Usually then it's just Russian roulette whether you finally figure out. Some of it's definitely doing things to Packer away, certain things it expects. Okay, great, thank you very much. Thanks. Any other questions? Yeah. One more question. Does Packer support Windows Azure platform or Windows OS image? I think there's an Azure builder. Yeah, I'm pretty sure there is too. I haven't used it or how good it is, but yeah, I'm pretty sure there is one. Anyone else? Anything else, anyone else? Great, thanks everyone for coming. Enjoy the rest of the week.