 You ready? All right. Okay. Let's get started. So, we're going to talk about image automation, building images, automatically doing functional testing, pushing them, all that good stuff. I'm Ryan Yard, I'm Jordan Calicoat, and we're from the Rackspace private cloud team. Rackspace private cloud, fanatical support, install, operate, open stacks, clusters, anywhere. That's what we want to do. We're awesome. Yes. You should totally go with us. Yes. So, again, automating base image builds for open stack private clouds, that's the topic for this discussion. Again, functionally testing, homogeneous, patched images, ideally. How many of you guys are building your own images? Everyone. Awesome. Are you guys starting with the cloud images and then building your own, or are you kind of like, we're going to talk about the tools that are available to use to automate image building? You guys are doing it by hand. Show of hands. Anybody building images by hand? All right. Shame on you. We're going to show you how to automate. All right. So, we looked at a couple options again. Not an exhaustive list, but we looked at Oz, Veewee, Box Grinder. We had a couple criteria that we were trying to evaluate them on. One, can you run it in a virtual machine? That was a good one. And then, can I do Windows? I don't know how many people care about that, but that would be interesting if you can not only build Windows boxes and Linux boxes. And then, what kind of distros can you run? Is it only RHEL-based, or can you do RHEL, can you do Ubuntu, can you do all that good stuff? So, those are some of the criteria that we were looking at. The VM thing is cool because we want to use open stack to build images for open stack, right? So, that's kind of a good criteria. Right? Build images as a service. Yeah. All right. So, let's just do a quick, this is kind of like talking points, you know, necessarily need to write that down. This is a definition for Oz. It's kind of pithy. Some of these are actually using a kickstart, a little juicy kickstart in the back. And then, the template kind of defines some parameters that you spit into that kickstart. So, let's look at Oz. So, Oz kind of failed in our criteria here because you can't really run it in a virtual machine, at least in our experience. Maybe there's some tomfoolery, boondoggery, that you can do to get that to happen. But also, is XML, Boo? Yeah. Compiles to, like, Icicle, I believe. And just kind of puts bash inline in your XML. And it's not good. So, we're going to show a failed build using Oz. There you go. Oh, let me, let me, let me nuke this guy. Here's my mouse. So, I already built a failed build, and I will build a failed build again. So, this is kind of what it looks like. I mean, I don't know how much value you see in this. But the idea being that each one of them has some different output, has different characteristics, it's doing different things. In this case, it's going to try and use the Fedora source tree again. We tried to build, like, one consistent image across all of them. Some of them worked. Some of them didn't. So, there we go. So, we got an Icicle, but what we actually ended up getting was just a blank image and it didn't really do anything. Oz is interesting in that it actually is going to go and use Quaimoo image, create an image, fire up KVM and libvert and start actually doing the install into the disk that it creates. That's really great if you have hardware, it doesn't work in a virtual machine. So, let's look at the next one, Veewee. This is actually the one that we picked. This one runs inside of Ruby. He's got all this Veewee skills. So, it uses a Ruby definition to basically define how much RAM you want, stuff like that, a base image size where it's going to pull the ISO that it's actually going to install with. It then does a, like, a VNC thing where it actually types in and sends the kick start in or precede if you're building one of them too. Let's look at that and actually demonstrate a Veewee. That uses a post install file that's basically shell script or Python or whatever you want to. It will do the OS install and then inject that and run afterwards. It's not really composable. You have to copy that post install for each image that you want to do. So, and each time you run it, it's going to want to run down, download an ISO for every new build. In this case, we're just going to pull down the CentOS minimal ISO and use that and then it kind of hangs out for a minute, decides what it's going to do and then it'll fire up a VNC session and start typing in the VNC, which is just crazy pants. It's kind of cool because what that allows you to do, at least on the Windows side, is that you can then use that VNC session to actually automate some things. So there you go. It typed a tab and then threw in the kick start thing and then starts running an install from there. Interesting. You could. Yeah. For Windows, it has the unattended install XML that just is like standard and you can have it talk to a WSUS or inject a key or have you want to do that. So we're going to let that kind of crank in the background. This one works inside of a VM, so we like that, but a bit pokey on a laptop. So, all right, box grinder. This one I like, but it's really funky. So whereas Oz and VeeVee will create a disk image and then fire up a VM and start doing the install in that VM, box grinder runs inside of a Charoop jail. So that's bizarre. So if you want to build RHEL, you have to have the RHEL version of box grinder. Right. When somebody ported a box grinder to Ubuntu, so I guess if you wanted to have kind of build servers, you could have an Ubuntu box grinder, a Fedora box grinder or what have you. I think there's some cool aspects of box grinder that I like. It has a notion of inheritance. So if you kind of look at the top here, what I'm doing is I've defined an appliance called the juice and then underneath that I have got another appliance that is just the various juice image itself. So it takes the base image and then you can then start layering applications on top of it. I think that's pretty cool because then you can kind of keep a really standard stock build and then inherit that build. You can actually have multiple inheritances. So you could have multiple appliance definitions inside of appliance. So you can have like your golden image and then your Hadoop image and whatever, but you can compose based on the definitions. So that's kind of cool. That's a really cool feature. So we're going to kick that off as well. Let's see how our view builds doing. I didn't like that. Look at that. It's going to go ahead and SSH and really beautiful. So right now that's actually doing the OS install. And if you were to, you know, VNC and you'd see that actually going through the kickstart. So why would you care about automating stuff and having these images and all these things? You talk about some post-boot things. There was a talk yesterday and I covered a lot of really great material. How many of you guys saw that talk yesterday? The one on KVM, Kwaymu, and customizing images. There's a similar. A few people. Yeah, he didn't really talk about the tooling. And that's kind of where we're sort of focusing. But he talked about some of the cloud and config drive. So we just want to kind of cover some of that from our perspective. Config drive, here you can pass in a file that, say, sets up some network interfaces, maybe pass in your known hosts. And this is what you would do in terms of a metadata service. You can stuff all that in there and then we'll pick it all up. Have you guys used config drive? Have you familiar with that? So basically it gives you a partition that contains all of the stuff that you would get from the metadata service. But it's mounted at a specific path. So it's basically the request path that you would do to the metadata server on a local drive. It can be like a CD, it could be like whatever you can define it, how it mounts that thing. Cloud and I think it's a bit more interesting in the fact that you can actually kind of do some fancy stuff with it. Here, this would be an example of installing Chef using Cloud and it and passing in your keys and assigning roles and doing all that stuff. This is pretty handy in my world. More automation, the better, okay, all right. Problems we encountered. So while we're kind of evaluating things and trying to actually get the stuff going and building images, we started thinking about snapshots, right? So you guys are all familiar with the Udev thing where in Centos and Rell it starts like enumerating your interfaces and doing crappy shit like that. So there's actually some, so you know you can set that out and you can go and nuke that trigger, that Udev trigger. There's also an interesting feature inside RC.System and if you go and look around in there, there's a really cool .file that you can drop into slash called .reconfigure. And if you put that in your template, it's like a run once file. And it'll actually nuke like users and all kinds of stuff and make kind of a clean, pristine template that you can use. And the other thing that you're gonna care about is you have a base image that has a disc of two gigs and you're building on a flavor with ten or whatever, you want that to automatically resize that partition. And if you're using the AMIs, they are just one big block device. So it will automatically when QMU copies that over, it will just need to grow the file system. It won't have to care about growing the partition. But if you're using a standard partitioning scheme, you have to care about that when you're booting on a flavor of a different size disc. Right, so root resize imaging and using cloud in it to do that or like in it ramfs to go and resize all your root partitions. These are things that we were kind of encountered that are sort of tricky and you have to worry about to a certain degree. And it's not consistent across all the different distributions. And there's in Fedora, there's a package proposed called Cloud Utils that has that grow part script from Ubuntu. And you can actually run that in like RC local, you have to reboot. I think there's in the new, I think Ubuntu is moving away from the net ramfs. And they actually have some kind of hook that does some crazy like fake pixie boot stuff so that it can resize that root partition at boot time and not have to reboot. So what if you took this idea of like automating images and maybe using VUE or one of these tools and you hooked it into a CI system. And maybe you like added some functional testing and then you were able to like have maybe Jenkins run jobs to push things to a CDN. Could you create like a cloud marketplace? That'd be kind of a cool idea. Maybe you can not only an external publicly facing one, but maybe you have an internal one for your internal customers that are using your private cloud or what have you. So kind of wanted to run with that idea. So Jenkins, right? You would have to maybe check like for get commit and maybe have a little hook in there that sees, I just committed a VB definition. I want to go run that. And then you have tests that you want to run. So maybe you write some stuff in those to do that. And when I do some reporting, make sure it actually finished. And then saying finished, you'd want to have a job that pushed it out to glance or what have you. And you don't want to manually QC that, right? Like if you have an application that has certain requirements, you want that to be built into the image and you want that to be tested. And you want to know about failures. You don't want to be doing that by hand. Yeah, so here we've got our build system. And this thing has a bunch of Jenkins jobs to find in here that do exactly what we just talked about. So we can kind of drill down a little bit for a minute if we look at this. And so you see we've got inside this pre-build script. We're looking at these various jobs that we're going to kick off from here. And kind of just a chain of different jobs and inside of like say this one here, you have another one that sort of references back kind of image tester here. This one would have, say for instance, some functional tests in there. The tempest has got some pretty interesting tests that come sort of in the tempest project. And I think it's really interesting to use those tests and to kind of build off of them. So we looked at building those tests to do exactly what we've been talking about, resizing the root partition, growing file systems, attaching volumes, various things that you'd want to do when you have an image, maybe taking snapshots, all those sort of things. So that's some of the things that we're doing in our build system. And then ideally you'd go ahead and push that out if you pushed it out to maybe glance backed by Swift or Cloud Files, the ARAC space, with the CDN. And you could distribute this stuff very easily. So that's our talk. Do you guys have any questions? That was really quick. So we wanted to kind of be interactive a little bit and see what you guys are doing, what problems you're having, and maybe if we've seen them or we can share knowledge here. Absolutely. So the question was, yeah, if we've had to integrate with AD or Active Directory inside the image, we haven't ran any tests for that. But I would imagine that that would be a fairly simple test to run. And we were thinking that where you would actually do that integration would probably be in the post build. So running Cloud in it to actually maybe have a role that goes and joins you to Active Directory or connects you to your LDAP system. So with Cloud in it, you can basically write your own modules for it that can be defined to pull data from an external source. So basically you can write your own module that will pull data from an external source or if it's local. And you can do whatever you want with that. So it's just kind of like a RC local type thing that can be consistently defined across your images. Right, yeah, I mean, you'd have to mock something out. We haven't done that. Any other questions? We're gonna be done so early. Come on guys. Is there still food in there? What's up? Yeah, so how would we deal with security updates in patching? Basically that's what the hook's for to listen to get. So basically we don't have an automated thing for that yet. We'd like to automate that as well. But basically you would have to go update the definition. It would see that. It would build the new image and then push that out. So you'd have to come up with some kind of versioning scheme or something to let your users know what was going on with that. Any other questions? Yeah, so we have a .file that we touch or something like that. So basically do we have regression testing that would ensure that your images are being built against the next version of the OS correctly? And we do not really. Yeah, at this point. We were looking at just those ideas of growing the file system, resizing it, all those kind of ideas to start. But yeah, ideally. Our test suite just cares about the things that we care about. So if those come up and it works, we're assuming that the OS is good too. And maybe we should do better. But yeah. Yeah, exactly. Yeah. I'd love to talk about that. And if you have ideas about that. Because that's kind of a hard thing to do, right? Yeah. Sir. Yeah, I mean, just that sort of artificial criteria that we set of can we run it inside of a virtual machine? That was the one. That was the one. It will work across the board for a lot of things. We didn't test a lot of building Windows images inside of OS. And Veewe successfully built Windows images. So that was kind of another sort of criteria that. One of our guys, Joe Brew, actually has a repository that you guys might have seen. And he uses OS for that. So I mean, you can definitely use OS if that's what works for you. Veewe just kind of fit our criteria a little better. Well, you can. So a lot of times what's happening. Do we use LVM in our images? So the little kickstart files that go behind all the definitions, those use LVM. But the problem with those is that in some cases that that kickstart file is the definition language is actually coded that that thing is static so that it doesn't change. Even if you go in there and start trying to flip around volume groups and renaming them, then the definition that sits over the top of it ends up breaking. So that's box grinder, right? Box grinder, OS. Well, because with Veewe, you're pretty much building your own pre-seed or kickstart. So you can use LVM or not. You do have to set that up initially and decide on that and basically generate your kickstart or your pre-seed. And then that will be used in your images. And that's kind of what I'm saying. It's not as composable. You can't reuse a lot of stuff. Any other questions? Yeah. So we don't have an official Windows image yet. But yes, we are testing with that. And it seems to work. Yes, sir. In the back. So how far do we customize images? Do we just do a base OS or do we have some pre-built images for different software stacks? I was going to say, I mean, I think it depends on the time that you need to spend lighting up an instance. So if you need a web server and you need it right away, you'd obviously want to install as much stuff as you can and add time into that image. But ideally, you would do all your customization in post build using Chef or one of those to keep it consistent and up to date. That's a nice thing about box grinder being composable is that you can have your base OS image and then just inherit from that and add whatever you want to do onto that. So out of the three that we showed, that one is probably the easiest to do that kind of thing. With BEWE, you're basically going to have your own post install that does all that stuff and that you have to write. Thank you, everybody. What's up? Oh, oh. What's up, man? Yeah, it depends. I mean, for our base images that we're making publicly available, we do not. But for specific customer use cases, we'll go do that. In Glance, is there a way to let a user move their snapshot across multiple AZs? You can have a isPublic flag in Glance that will let it be seen by everyone. I believe, correct me if I'm wrong, but I believe that works across availability zones. So I think if you just set it to public, but then everyone can see it. So that might be a concern. Great, anything else? All right, well, that was it. Thank you, everybody. Thanks.