 So if people want to get a little closer, that'd be cool. But this is going to be very interactive. So actually, please do get closer, because I'm going to be walking through code. And if you don't ask questions, I'll just be like, I know all this, and I'll just go fast. And you might not know it. And so the goal is to explain it as I walk through it. And there will be things that I kind of scratch my head at and like, oh, this isn't the code that I'm currently using. This is Rackspaces code, which I've worked on previously and have pulled and sent patches back and forth. But it's not what I use day to day. So Rackspace Private Cloud. Currently, this branch is for the Folsom release. It's tagged as version 3.0.1. So if you go to rackspace.com cloud private, you can download an installer that will go and install. It'll install OpenStack on 20 nodes, something like that. This code is being worked on by about 10 developers. It's good stuff. It has an open source Chef server embedded inside it. And they're just treating it like a JRE, where it does exactly what they need. The repo there, GitHub RCBOps, that's the team that's working on this stuff. So that's a bunch of their operational cookbooks, some of the other things that we're not going to really cover today. But it's all up on GitHub. Rackspace has been really great about sharing the work that they're doing. So you can follow along and make sure that if you don't want to use the product to deliver it, you can take the cookbooks and merge them. We have a whole community around Chef and OpenStack. We haven't talked about that. If we have way too much time at the end, I'll pull up a slide deck for a meeting today. But there's a lot of people working on this. RCBOps cookbooks is the actual Chef repo that you use to run this infrastructure. So I'm going to walk through that as we go here. This is a high-level architecture of the Rackspace private cloud product. They've added a new product called OpenCenter. It's kind of a controller for the nodes on top of it. It gives you a nice little UI. There's Chef server embedded there in the middle that is managing the state of pretty much everything. And then there are cookbooks that map to the different services. That's what it looks like. What's that? Sure, hey, look, there's Evan. Yeah, but then they won't get you on the recording. Oh, there we go. OpenCenter is not just an API. It's several pieces altogether that comprises the product. So it has an API, which is basically a state machine with a REST interface attached to it. And then there's also the CLI tools. And there's the GUI, the dashboard. And it's agent-based, as I believe is illustrated there on the slide. And then there is a component called the Solver that tries to determine how to get from the current state you're in to the desired state that you want to go to and to figure out what needs to happen to get there. So that's how you interact with it. Yes, it is the infrastructure level. It is not even up to platform level. So basically, you're starting from blank nodes that you want to actually put whatever operating system you're going to deploy onto on and go from that point forward. So then it will be using Chef through the orchestration layer to actually put on whatever roles you want on those machines. And your bare-mail provisioner would be razor. Right, exactly. It's much lower than a pass. Yeah, it's below pass. Yeah, this is? Yes. So if you're following along in your GitHub browser, or in your GitHub browser, in your web browser on GitHub, you can go right here. I've already cloned this locally. We're going to be walking through the roles, the environments, and the cookbooks. So everyone got that? GitHub.com, RCBOps, Chef Cookbooks. That's where the magic happens. And here comes the magic. So this is all the cookbooks that are currently get sub-moduled in there. There are quite a few. Some you might say, well, why do I need these for an open-stack deployment, such things as the AWS cookbook? Most of the cookbooks that are in here are supporting or the ones that are like, why is that there? Are supporting dependencies for some of the other cookbooks. So the MySQL cookbook actually depends on the AWS cookbook because it has a recipe for tuning MySQL on AWS. So that's why. That's there. Build Essentials there because we're building some Ruby modules from source. Erlang's there as a dependency of rabbit. Keep alive for the HA stuff that we won't be going through today. The monitoring, the monit, we're not going to dive into that. We're going to stick to the basics of open-stack. So an SOS report is your agent monitoring stuff. Monitoring stuff. Yeah, monitoring stuff, sure. Close enough. We also won't be talking about Swift for time. So the deployment follows the pattern of using an environment to set attributes, push them into the cookbooks, and that's how everything pretty much works. The environments, so the roles actually don't have very many attributes. We're putting the things that we care about in the environment level. So if you're familiar with Chef, the precedents, we're just doing overrides to say, boom, this is how everything's going to deal with it. So looking at the example JSON that is bundled in the repo, I've got a laser pointer. The name and description, of course, are kind of boiler plate stuff. There are no cookbook versions enforced by this environment. That is something that you may want to do. If you're doing a continuous integration pipeline with multiple open-stack deployments, you'd probably want to start locking down your cookbook versions. The JSON class, that's boiler plate for the environment. There are no default attributes, because we're using this strictly for overrides. But as we get into the override attributes, the first thing we see is developer mode falls. If you need additional debugging, developer mode true. It's pretty straightforward there. For monitoring, the Rackspace stuff is using collectee and monit. The cookbooks that I'm using don't, because different people have different opinions about monitoring and logging and that kind of stuff. So when we run into that in the recipes, they'll be like, I'm not using that in my stuff. Rackspace is trying to get to pull that stuff out a little bit more to make the cookbooks less opinionated for Rackspace. So other people can find them useful for whatever their choices are for monitoring. Glance, upload image true. The images, it's pulling these out of the attributes file for Glance. So the URLs are there for Seros and Precise. It's pulling them off of UEC and the Sero site. You can override these. So if you have local mirrors of those images, you don't have to pull them off the internet. I think you guys are pulling them off the internet, because they don't want to bundle Ubuntu images and Seros images in their deployment. These attributes relate to Nova. We'll see these again later. Whether or not to rate limit on the API volume, the vert type of QMU, obviously if you're going to use like LXC or something, you could change that. The networks. So this is actually for setting up the networks on the host. So when the host, in this case, has two ethernet adapters, one becomes the public network. That's what your VMs are going to connect to to go out to the internet or to your consumers of your OpenStack IS. And the private is the network that all the communications on the back end are doing. So the Nova to Swift, the Nova to Glance communications all happen on the back end private network. The reason we're labeling them is because you actually could make these exactly the same if you only had one NIC, but the goal is to have multiple NICs to segment your traffic. Yes? In this case, yes they are. No, no, they're VLANs off the same NIC, but yes. They are both E0, but if you had an E1, the reason we're naming them is later we could add a storage or a management or something like that. So you might have three or four NICs in a box, which would not be uncommon. If you looked at the crowbar stuff, they have the same kind of thing where they bomb them. But it's not that crazy. Over here are the attributes for MySQL, allowing you to log into it remotely. So you can create the schemas. OSOps network. So these are, these map you back to the same ones here. 192, this is the ones that the instances are going to use. And in this case, it's all the same. And then package component, Fulsome. So we're going to be using the Fulsome packages. So there are a boatload of roles in this Chef repo. We're not going to use all these. Obviously there are lots of different ways you may deploy this OpenStack using this repo. The reason there are so many is because we're going to dive into the all-in-one. So a single box, if you want to run everything on one box, you'd run the all-in-one role, which of course is going to have dependencies on other roles, which depend on other roles and depend on other roles. As you break apart that hierarchy, you can map those to separate machines. So things like the glance role, well it actually, that maps into Glance API, Glance Registry, and Glance Setup, you could have those on separate machines. So each one of these can map out to multiple machines. And then there's HA stuff in there as well, the HA controllers. And when you start doing multiple machines, you can just run single compute. So you could do n plus 1 and have single controller running everything, and then single compute just fanning out for multiple computer nodes. And that's a pretty common if you're doing 20 boxes. It's not exactly a super HA, but you have one box that does all the detail work, and then everyone else is just compute. Yes? No. Chef does not support multiple environments. Chef does not support multiple environments on a single node. Yeah, sorry. Yes, there's definitely a use case for multiple environments. I came out wrong. Question was, is there a use case for multiple environments? Absolutely, absolutely. So if you have different machines on different networks, and they're all talking the same Chef server, you just say, hey, you guys are in the QA environment. You guys are in the production environment. And then you set up the attributes different ways. Maybe in QA, the developer mode is true. You want all that verbose logging to see when things fall over sideways. And then when you go into production, you turn off debug, and maybe in production, you have four nicks. You actually have four nicks. But in QA, you only have one or two. So definitely, you would have multiple environments. And this is just the example. And then if you're doing a CI environment, you would probably use cookbook versions to lock the versions that actually go into production. So QA, you could unlock. But production, you'd lock them down. So stuff doesn't accidentally get in. So starting at the top, here I'm going to be jumping between the roles and then the recipes and cookbooks that they map to. So we'll be going between slides and actual code. The all-in-one role, we're going to take one box, and we're going to turn it into an all-in-one nova. So as it says, this will create an all-in-one open stack cluster. All-in-one just calls single controller and single compute. The roles are pretty smart. So single controller, all-in-one, actually when you apply this to one box, that's the run list that we're going to walk through. So we have a lot of recipes to look at. But that's what they are. Some of them we'll probably skip. So single controller, nova controller, non-HA. That's a lot of stuff, right? But if you look at it, it's just we're going to set up my SQL rabbit, and then we go keystone, glance, nova, and horizon. That's pretty straightforward. There's no nova compute in here because nova compute's going to run on something else. So if we're doing the one plus in model single controller and compute nodes, you put this on that guy. Definitely non-HA. So looking at the base role, this is going to get applied to pretty much everything. So the base role gets used a lot. This is what Rackspace wants every single machine to have. Anybody who's running, the base role is a very common pattern in the chef world. People have a base role. It's like everybody is going to do this. In this case, they're using OSSOP to use packages to set up the package repos. We're not going to look at that recipe. Open SSH, that's community open SSH. Make sure SSH is on there. NTP, very important in Chef. And with Chef, you can't have clock skew. SOS report is monitoring stuff that we're not going to go into. Our syslog, they're using syslog. This is going to make sure that it's set up. So all the boxes are using syslog the same way. Recipe hardware, I assume that's around Nick setup. Yep, Nick setup. And OSSOP's utils default. What is that doing? Let's pull it up. It's going to be exciting. So the OSSOP's utils, this is a utility library that Rackspace wrote with, well, back when Vish was working there, with Dell and Dreamhost. And it's a bunch of libraries for doing searches of IP locations, databases, and applying patches. It doesn't really fit in anything else, so it's OSSOP's utils. But the recipe, what was the recipe there? Packages, oh, okay, default. Oh yeah, sysctrl. So yeah, that's just tuning the OS. Not super exciting. Sysctrl's one of the cookbooks included. This is not the sysctrl that's on the community site, but there's like nine of them. And they all kind of do the same thing. Someone should clean that up. But this is just tuning for rabbit. Not super exciting. But that's the base role. Throws in some NTP servers, just so everybody behaves the same way. So, yes? I need to set the goal. Oh, okay, okay. So, back up real quick. The difference between a recipe and a role. And Chef, a cookbook is how you deliver a piece of functionality, like a particular application or a service. So, if we go back to our list of cookbooks, each one of these kinds of maps to something you're going to configure, right? And inside of the cookbook is the recipe. The recipe is the actual, the Ruby code that has the resources that you're going to set up, going to calculate things. So, as we just looked at that, the sysctrl settings, that was a recipe. There may be multiple recipes inside a cookbook. So, when we look at our run list, OSLogChutel's packages, we saw that, OpenSSH. These are just the recipes that are getting called. The colon colon is specifying which one of the recipes within that particular cookbook we're going to call, okay? If there's not a colon colon, it's default. So, a role is a way of saying this is a particular class of machines. This is something that anybody who has this is going to do these things. Roles may include other roles, but they have a run list. A run list that says these are the things that it means to be a base. If you applied this on a machine, well, sorry, I've got a single controller up there. This is the things that you get with a base. And a role has a run list and then it has attributes, default or overrides. Yes, somebody know Puppet can answer that. I'm not a Puppet guy, surprisingly. But that sounds right, modules. Modules are cookbooks, manifest are cookbooks. Okay. And then they have some concept of roles. So, in Chef terminology, a node is the endpoint, is the actual machine. And then that machine has a run list. And that run list may be recipes or roles or a mix of them. So, in this case, we see these are just roles in our run list. And this one, the run list is actually recipes. You can mix them, right? And then role is the class of the machines. And then above roles are environments. So you say, a machine can have multiple roles. It can have multiple recipes. It has a single run list in a single environment. Okay. Any other questions? I mean, sorry, I guess I assumed a little more familiarity with what we're doing. But that's the base role. Table stakes for being a node in Rackspace Private Cloud. So the first role in the single controller, or the first one we're gonna look at here, is MySQL master. This is just installing MySQL, setting up replication if you're using two nodes. See, it calls the base role again. That's not a problem. With Chef, when it creates that run list, it deduplicates them. So the first time it sees it, that's the first time it gets called. And it's okay to have it get called multiple times in the run list. So base is in everything, just because you may have a MySQL box. When you start federating this out to multiple machines, maybe in a larger deployment, you might have a box that, he does all the MySQL for you. That's the only thing that machine does. You still need that base role. So you can say, hey, that's the MySQL master. And he's gonna call MySQL open stack server. So let's take a quick glance over at that. So this is a cookbook that's just helper stuff for OpenStack. It wraps the MySQL cookbook. So it's going to pull things in, some of those libraries that we saw in OSOps Utils. It's gonna include monitoring, and it's gonna set up the MySQL Ruby cookbook. That just installs the MySQL gem. So Chef can make straight MySQL calls. These methods get bind end points. That's coming out of OSOps Utils. So what this is doing is just setting up MySQL to listen on where to listen. And then it just tunes it. Oh, look at that. I am first MySQL master. And then it's setting up replication, connection info, pulling the bind IP that we got, the username, the password which you can pass it in, or up here at the top, it actually just generates one if you don't have one. And then it saves it on the node. So later on, all your machines get their own passwords. Then, looks like it configures a slave. Hey, there's a new code. I haven't seen all this stuff. And then we'll see we officially, so up here it's setting connect to the master because we're the slave. And then we're gonna set an attribute, identifying us as a second master. Then, oh, if we were the first master, we've connected back to the second master yet. So it actually waits to make sure that stuff's set. And so, if you're new to Chef, that's a very common pattern is to use attributes to say, hey, I'm doing this particular, I provide this certain thing. And then people who are looking for that will just search for you. And then in this case, they'll just search for someone else who is MySQL master. And then when they get that node back, they'll look for the ID configuring a slave. And, hey, look at that. Clean up the craptastic MySQL default users. You know, the MySQL that ships from Ubuntu has a bunch of stuff already there. And so we take it out. And then we render the configuration file for MySQL. And then they do some monitoring stuff that I'm not really into. Yes, yes, so in your searches, you say, and ChefEnvironment equals. And then you say, node environment. So you search for everyone who's in MyEnvironment only. Was it doing that? Or is that in the OSOps utils? It's in the utils. The utils has search helpers. Right. Ta-da. Yeah. There is a patch that I think is going into like 11, six, or 11, eight that I think automatically adds it as like a search comma environment. So that just sets up our MySQL master slave. It actually didn't create the schema. It just set up MySQL ready to go for OpenStack. So the next thing that happens is Rabbit gets set up. Yes. Yes, yes. So what happens in Chef is this, we've assigned this role of all in one to a node. And then that node, I can try to do it and we'll see what happens. Ready, live demo. Now, actually I know this will blow up because it's a VM and I just, but it'll start and you'll see what it looks like until it blows up because I haven't set up multiple NICs on my VMs here. But you know, it's fun to watch it blow up. It's educational. Let me go clean install, don't save. So I've got my network thinking it over, thinking it over. Wow, that was a fast change of directory. So I have these two VMs. One of them is just an app cacher, NG. So I cached all my stuff. But if I say KnifeNode, show Ubuntu 2.vm, this guy, I've assigned the all in one role. So right here, this is what a node looks like. I gave him all in one role. He also has the Emacs24 role because I was writing an Emacs24 cookbook. But anyway, he's got all in one and so this stuff will get wiped out after he runs. So let's see what happens when a freshly provisioned VM actually runs. And it actually will probably be really slow and we'll just come back to it. So when you're on the box, you just say Chef Client and that's gonna connect to the server and says, hey, Ubuntu 2.204.vm, what am I supposed to do? And he says, oh, hi there. Here's the run list that you have. We're gonna synchronize our cookbooks. We're gonna download all the ones that we need to actually do this run list. So it's gonna go through that. I'm gonna switch off this but right now it's downloading from a hosted chef but you could have your own open source chef server or private chef or whatever. And it's slowly gonna do all that stuff. So yeah, that's, and it'll get to the MySQL master part of the run list. RabbitMQ server. This is gonna call the base role yet again. It's gonna call the default Erlang recipe. This installs Erlang. Not super exciting. We don't need to look at that. But I think it's pulling it off of the latest version of Erlang or the distro one if that's what you choose. And then it's gonna call Rabbit OpenStack server. This is just like the MySQL OpenStack. This is just kind of helper stuff. See, it's still going. So this is just an OpenStack cookbook wrapper. So here we're generating a password that Rabbit's gonna use. We're including the OSOPSU Tills to give us those helper methods. We're including the monitoring stuff. And then we're going to pull in platform options. So the rabbit cook, so it actually has a set of a hash mapped to that attribute. And we're gonna just pull stuff off of platform options because this actually works on Red Hat 6 or on Ubuntu 12.04. Then we might override our Rabbit MQ port. Which IP we have need to listen on all IP so we can use a floating VIP. Then there's some notes for their developers. And then there's some case logic here about since the upstream Rabbit MQ server does crazy things like install packages from random app revos. I fixed that. So their wrapper here is because the Rabbit cookbook upstream didn't know. Well, there was this period of time before Rabbit 3 came out that some distros were shipping 1.8 or 1.4. Some distros were shipping 2.6, 2.8. And now that 3.0 is out, it breaks everything too. I fixed all that in the Rabbit 2.0 cookbook. But what this is doing is just pinning that we only want to use the distro version of Rabbit. So this is kind of janky and you don't need this anymore. But that's what it's there for. Yeah, see, and I fixed all this stuff. Red Hat works. You're welcome. Yeah. Are there any other rabbits out there? So git settings by role is another helper method out of what was actually tells. And we're looking for other rabbit rabbits if we're doing clustering. And then this is just going to include the default rabbit recipe and go ahead and install it right there. And then it's going to restart the service. Holy, I fixed this too. So Rabbit has some weird, it doesn't have a good in it D setup. I don't go down, I go up. Yeah, I think I even told you guys I fixed it. I posted to the mailing list that Rabbit 2.0 came out. Yeah, it fixes all this janky stuff. Yeah, but at least here you see Rabbit MQ user. So in Chef, there are resources that are bundled with Chef. So resources like service, you know, manages services. It's part of Chef, it's built into Chef. There's some other resources in here. Wow, we're using a lot of custom ones here. But if you are getting a particular resource, if you need to write new ones that are not provided by Chef, we call those lightweight resource providers. You can bundle them inside a cookbook to add new functionality. So here we see the app cookbook is giving you the ability to set preferences for how app behaves. It's not part of Chef, but if you need to use app, cookbook can give you new functionality. As we see here, the Rabbit cookbook will do users for you. You don't have to go and type rabbit control user ad, blah, blah, blah. You just use the LWRPs that handle that. And so their monitoring cookbook is providing a Procmon resource, a metric resource. And that's just for those of you all who aren't familiar with it, that's where this, and then the underscore signifies coming out of the Rabbit cookbook. And so the Rabbit cookbook does users, queues, I can't think of, be hosts, and there's pull requests for everything else that you might do. And there's test, and it works with three and two, and yeah, and then they do their Keep Alive, VIP setup stuff for Rabbit. So you probably don't care about that if you're running on a single box. So that's, Rabbit now is set up. We've got MySQL, we've got Rabbit. Now we're getting into OpenStack itself. So this is kind of table stakes for our OpenStack install. We're gonna use Keystone. There we see the base role again. And we have, oh, I left out the Keystone API role. I bet I could tell you what it does. Keystone server is gonna go and set up the basics. So let's look at Keystone. Oh, look at that, it blew up. So here's my node running. So here's our long run list that we're slowly making our way through. This we're all the way on the third line. We are in Keystone. But then it synchronized all the cookbooks and compiled them, and then it threw a warning about some constant being reused, but it's just warning, so who cares. And then it sees that app get update. Lots of people called it all, and it said, hey, whoa. We don't need, we don't need, you cloned it. We're only gonna call it once up front. And that's kind of the behavior you want. And then it went off to the races here. Executed app get update. Then it called build essential. Build essential was in the MySQL, the Ruby. It needed to be able to build Ruby gems. So it installed the build essentials, which is your Ruby, it's your build tool chain. And so it installed all this junk, which is good junk. And then we saw it installed MySQL client. That's what MySQL Ruby also installs. And then the MySQL dev, and then it blew up because all the OSOPS util stuff was like, hey, you don't have any networks, you don't have any real networks that mapped all this stuff. So, and then at the end, oh, look at that, new version of control point. At the end, even though it blows up, when the Chef client runs, on success or failure, it gets caught and there's a report handler or an exception handler. Basically, that gives you the full context of the run and you can then take it and send it somewhere. So, in this case, I don't have any exception handlers, but I could have like, hey, I need to email this. If a Chef client run fails anywhere, email ServiceNow or whatever, or send a trap, or pager duty, or whatever. That's handy to have. There are about 40 exception report handlers on the community site, IRC, HipChat, Twitter, whatever. So, syslog. Do you use the syslog report handler? You should, because you already have syslog. Ha ha ha, there's also a really nice one for Splunk and LogStash too. Oh yes, Keystone server. So, here's our Keystone server recipe, generating yet another password. So, it's gonna set up syslog for Keystone. I don't need to go into that. It calls MySQL client, MySQL Ruby, which we're previously called. So, those don't get run again. Oh, so if your TILS doesn't get run again, monitoring doesn't get run again, but that's cool. And here, they've got some debugging stuff for, if we're in developer mode, it sets a known password, because I'm sure these guys debug this stuff a lot. There we pull the platform options, hash out so we can call things, rather than say, node, Keystone platform, Red Hat, on and on and on, they pull down just the hash. So, they're gonna call createDB a user. This is another helper method out of OSUXU TILS. It goes and creates whatever we set up as attributes for Keystone with our name or username and password. And then, go and get the access endpoint for MySQL. So, we can do all this. So then, MySQL, oh, they installed MySQL packages again. I got some refactoring here I'd like to do someday, because everyone calls Keystone stuff. And I really hate that Keystone sleep thing. Keystone behaves kinda wonky, so they give it a 10 second sleep to give it time to start up, because it's like, I'm ready for business, and it's not, it's lying. So, that's what's going on there. But, here we installed the Keystone service, pretty basic stuff. We're enabling it, which doesn't, isn't the same thing as start. We're just saying, hey, put it in a D, put it at the right run levels. It supports status and restart. And then, it notifies that sleep. So, it's saying, hey, we're enabled. If anybody restarts Keystone, you still have to sleep. So, anytime that Keystone gets run, it calls the sleep, which slows things down. Then, we set up some more monitoring stuff, create the Etsy Keystone, hard coding your Keystone users. Then, they call Keystone Managed DB Sync. Here, they're using these Git access endpoints, just the Ad Helper method again. The logic down in the OSOPS Utils will actually look and see if you have a management network, or a management network. So, that traffic stays off the public interface. So, rather than clutter up the recipe, you have helper methods to hide that away. This is package component stuff. I guess if you were using something besides Folsom. And then, they have this massive amount of stuff that goes into the Keystone conf. All these variables get passed in. Some of them, like these node calls, could come within the template. You can call the node directly. This just makes it easier to see all the stuff that is going into the template in one place. And it's a lot of stuff, but Keystone has a lot of settings. And some of them are more, some of them are worse than others, like the Nova Conf. Yes. Yeah. If you need essence cookbooks, they're almost done with Grizzly. There are Grizzly branches and forks of, lots of people have Grizzly already working or close to working. HubSpot, who gave the keynote the other day, they're using Grizzly on bare metal. And so there are, we had a session today to talk about kind of catching everybody up with each other and we'll be coming together back on Grizzly. Probably in about a month or so. But notice here at the end of this template. You know, so this, so template is just pulling in the, it's writing at your Keystone conf off of this ERB file, which is just a Ruby templating language. And then anytime this file changes, you know, it will regenerate it, but it's gonna send notifications to resync the DB and restart the Keystone. So remember, Keystone just enabled. It just said, hey, set it up, don't start it. It waits for the templates to get rendered. And so anytime you might make a Keystone change, Keystone gets restarted, and then of course the Keystone service then sleeps for 10 seconds. Yes. Well, yes. So, you know, yeah, yeah. I mean, you manage that as you would expect. And make sure you have a reporter exception handler that sends a diff that captures all the changes and sends them somewhere. So someone notices, oh, we accidentally did all these changes. It's a nice thing to have. Then they are deleting one of the files that gets bundled in with Keystone. This is the flat file for SQLite. You know, they're not using it. They're using MySQL. There are cookbooks that are using Postgres if you're into Postgres. And probably soon it'll support both. Writing out their Keystone logging comp, which again, restarts Keystone. Do you mean to have all the immediate leaves in here for restarting them? Because if you don't put the immediate leaves, it'll cascade and restart Keystone at the end. Actually, you do though. Yeah, anytime you touch Keystone, you have to restart it in case someone needs Keystone. Yeah, well. Yeah, everyone loves Keystone. Keystone like forever. Yeah. Then they're gonna go and create tenants for Keystone. So, the Keystone tenants are set. Oh, this is gonna be fun. So, let's look at the attributes file for Keystone. Well, no, it's not. This is all the things that you might possibly set on Keystone, page after page of stuff. But I was looking for tenants. Okay, so admin and service, that should be from a user's hash keys or something. I mean, there's a lot of things that you might do. And then, depending on which OS you're using, you might have different settings. Packages have different names. But you abstract it all out into your attributes file so it doesn't clutter up your recipe with a whole bunch of, if Ubuntu do this, if Red Hat do that. Yeah, so they're using, in Chef, there's Platform and there's Platform Family. So, Platform is more specific to the OS. So, Fedora, yeah, yeah. So, they could call it Platform Family of RHEL. And then that would also give you scientific and unbreakable or whatever they call their LSB tag. Cause you guys really wanna run on Oracle's Red Hat. Yeah, yeah. Right, I mean, that's why we have both. We have Platform Family for when it's really the same. I mean, Platform Family of Debian includes Ubuntu. And Mint and lots of other stuff. But when it's not the same, then you say case Platform and you can pick the ones you actually want. And things like Fedora, Fedora 12 is not the same as 18. So, then you might actually have to go into OS version. But it's nice to try to pull up as much of it into the same commonality as you can. And you guys probably haven't seen a lot of Mint installs of OpenStack. So, it's gonna have this for loop, nice each do tenant name and just create those Keystone tenants. Here's Keystone tenant is another lightweight resource provider that has a tenant helper that just goes and automatically creates these for you. It has a whole lot of attributes you can set, but it's a Keystone tenant. There's a lot of things you might have there. Same with Keystone roles. There's another loop over all the roles that are provided and then all the Keystone users. You could, you could. We put it into Attributes, so somewhere you could just have a data bag to Attributes dump. But yeah, a data bag will give you a nice file to manage it in, so would an Attributes file or a role or an environment. There are lots of places that just stash them. And all of them would be backed in version control, which is a nice thing to have. And then for each of the roles, it's gonna go and grant permissions for all the tenants. Then we have a whole bunch of Keystone servers and then we add Keystone endpoints, Keystone users and then some monitoring stuff. And then at the end, Keystone client patch because I guess there's some patch that they're mainlining because it's not upstream yet. So, their Keystone server is now up. Yeah, I don't have the Keystone API up here, but let's take a quick look at that. And it calls Keystone API, so pretty straightforward. And that's Keystone, Keystone API. And it calls syslog, osopsutils, monitoring, installs whatever Keystone packages are there, which this all happened before, but it's okay. It's okay to have duplication. I mean, it's not dry, but it could probably cleaned up some. This is all previously coded. Am I looking at the right? I guess this is if you put your Keystone API in a separate box. Yeah, yeah, so that's why they're separate. The Keystone includes Keystone API in case you had them on separate boxes, in case you didn't hear that. So, Keystone, now set up, fully populated, and we're good to go. So now, Glance setup is followed by Glance registry and Glance API. Probably, and each one of those, they call the base role again, so if your Glance machines were spread across two machines, three machines, it would all have that. So let's take a look into the fun world of Glance. And if you have any questions, please speak up, because I'm just talking to my laptop here. Yes? Yes. When you say redeploy? Anytime you converge, no, no. So that when you write an LWRP, you need to be item potent. So they need to be safe to be run multiple times. So I would assume in the Keystone LWRPs, it actually checks to see if that user's already there before trying to create them again. And if it is, it's a no op, and it doesn't notify anything. And so, if it doesn't restart Keystone each time, so after you run Keystone through once, all those guys are there. If nothing changes, it goes really fast because it no ops, no ops, no ops, no ops, no ops, and it's done. And so, the first time we run this on, when I run this on like bare metal, I've got some 16 gig machines with SSDs, takes about 12 minutes to converge this all in one. And I've got AppCast, RNG, so I'm not even downloading from the internet. It's just, it takes a long time to set up all these services, especially with all the Keystone sleeps. But the next time I run it, it's like 15 seconds. It's like, do, do, do, do, do, do. Everything's already there, the packages are already installed. And that's really important because then I can do things like dry run. I can say, well, what would happen if I were to run this? And it would say, oh, nothing changes, nothing changes, nothing changes, nothing changes. Or I could say, oh, here's a file that I would change, here's the def, and that's, yeah. Yeah, yeah, and so you can run dry run mode. Oh yeah, and then that makes it like, it's safe to run the Chef Client any time because it's not gonna make any changes. So, some folks will run Chef Client as a daemon and every 30 minutes it converges or whatever you like. A lot of people do it daily, or you can just run it manually. Yeah, and a lot of people actually run it on cron and they run it at the top of the hour or the bottom of the hour, and it hosted Chef, we get, it does an 11. Yeah, so Chef 11 has a lock and it's smart enough not to, so yeah, that first time you call the Chef Client and it's taking 40 minutes to install OpenStack. It's okay if you call it again because it has a lock now. But I'm sure OpenStack was one of the reasons we have that lock. Yeah, Java, anything Java? Yeah, so I was looking at Glance. So yeah, Glance, bunch of recipes we've seen before. Get roll count, what, what? You can only have one node with the Glance setup roll. Okay, so they're looking to see if anybody else thinks they're the Glance set or upper, and if there are, they're like, die. You know, there can only be one. Make like a Highlander resource. There can only be one. Yeah, and then some boilerplate for setting up users. There's that Python keystone thing that should be pulled out into just a keystone helper because we're gonna see it like nine times. Then we pull out the key, within Glance says, hey, I need to talk to Keystone. Go and get the access endpoints. And so this is that OSOPSU-Tills helper method. Then it's gonna go and say, hey, my SQL, go create my Glance stuff. Package curl install. Okay, might need curl, I guess. Yeah, curls for the images. Why not remote file? Okay, there's a remote file resource inside of Chef. And then they go and pull off of their, they install whatever, that my SQL stuff again. Man, this is some boilerplate that needs to be. Then they take out the SQLite stuff. Then they do more keystone stuff. Keystone tenants, users, roles, Etsy Glance, Etsy Glance logging, Glance registry conf, just writing all this fun stuff. They have some problem with that Glance console log. And then they call Glance DB sync. And Glance is now set up. After Glance set up, it was Glance registry. So they're two services to Glance. And a whole lot of duplication from the other guy, of the my SQL, the syslog, the monitoring. There's definitely some code cleanup that could happen here. But it's easy to throw stones, right? Yeah, it works, and everybody forks it. And here's a whole bunch of the same stuff we saw in the Glance setup. The reason, of course, is these might run on different machines. And it's safe. It's okay to have duplication. Then they have the Glance API service. And yeah, so we don't. And then there's, you know, set it up the Glance registry, Paste.ini file. So we're not gonna look at API unless people really want to. It's almost the same. So we now have keystone. We now have Glance. This is awesome. We can, you know, serve up images. So Nova setup. This is going to configure a box to be ready for Nova. Is this session over at 5.20? Okay. I mean, yeah, we can all kick off and go get some beers or something. Fortunately, no. So, oh, I got a company card. I got a company card. So Nova setup, that boilerplate stuff. Then we're gonna go get our keystone endpoints. Then we're gonna create our Nova DB stuff. Then we're gonna sync the Nova database schema. And then they've got a Nova plugin monitoring thing that they've added. So that's, you know, that's not standard open stack. So Nova setup has created the schema for Nova. It's not super exciting, but you know you have to have that stuff. Nova network controller is gonna set up Nova networking. This is the first step of setting up your Nova box. We're not using Quantum here. I know in the roles, there's some Quantum stuff that they started, I assume. Come Grizzly, they'll be having OBS or, yeah, OBS. And then, you know, within the Chef community, there's, you know, Medonet and Nicera and some other stuff. Oh, Nova network is a separate cookbook. Yeah, I was, yeah, Nova network is actually a separate cookbook. Why is Nova network a separate cookbook? Oh, this is where they're doing their Quantum stuff. Oh, and I'm sorry, I said Quantum. Nova networks. Open stack networks. Open stack networks. Yeah, we'll have to rename all this fun stuff. So, Nova network, Nova network. So, it's calling Nova common, which, how's that different from Nova setup? So Nova common sets up your Nova Conf. Oh, you guys have an LWRP for Nova Conf. Excellent. Are you using Partials? Awesome. So Chef 11 added the concept of partial template rendering. So multiple cookbooks can manage the same file. Open stack is like the most abusive case of that, of course, because everyone is sticking into Nova Conf. So, someday we'll have Conf directories, that'll be awesome. But until then, we have a Nova Conf LWRP to handle that. So, all they do is tag in the release, set up login for Nova, write out the open RC. So, when you create Nova, you can say, hey, here are the credentials I wanna use, or it'll go and generate them for you. I'll look backwards compatibility for Nova RC. And then they go and create a login for the Nova user. So that's what Nova common did, was set up that base stuff, and then they install the Nova network packages, install the Nova network service. They enable it, and then it says, hey, subscribe to the Nova Conf file. If anybody changes that, restart the service. And as you can probably guess, since it says delayed, they're expecting lots of people to change Nova Conf, and delayed means at the end of that, the Nova network service will restart. Which is what you need to do. Usually, delayed is actually the default, but for Keystone, we need a Keystone up all the time. I'll, every 10 second, wait's and all. So, yes, delayed is the default, but they're being very explicit about the defaults, just so as a casual user, you may come in and not know that delayed is there. Same with package install. Install is the default. Some people write it, you don't even have to list it. Yeah, it's easier to read, and easier to come back and change it if you need to be something else. I'm not hating, I'm explaining it. Nova Scheduler, the next part of Nova, it's gonna look a lot like what we just saw. So there's Nova Common again, making sure that our Nova Conf is on the machine, because maybe your scheduler is on a different machine from your API, from everything else. Go in, creating the Nova Lock file, installing our Nova Scheduler packages, installing the Nova Scheduler, which is subscribed to the Nova Conf, and it's also subscribed to the Logging Conf. The other one was too, right? Sure was, and then monitoring stuff, and then including a Nova Scheduler patch, which I'm sure is just a back port or something. Look at that. Affinity filters don't work if scheduler hints is known, so sometimes you have to patch stuff. Yeah, don't look at the sausage. Let me walk you through the sausage. I mean, you're saying, yeah, just enjoy the sausage. End of the day. So API, EC2, they're going to install both the OpenStack and the EC2 APIs. So here we see OS, the OS Compute API and the EC2 API, and then the storage API gets done later, and the Swift API, the sender API gets installed by sender. So the OpenStack API actually gets broken out more. Super exciting, but let's take a quick look at those. We're about to start getting faster. So there we see, absolutely. So there's a knife, there's a knife OpenStack plugin. If the slides from my talk Monday about the state of Chef and OpenStack kind of went through that, those are up on SlideShare, Matt Ray, if you want them. Eventually they'll have all the recordings for the talks, and you'll see me tell, say the same thing. But yeah, it works on Diablo and up, and it uses the OpenStack API, and it's been tested with most of the OpenStack vendors who are here. So even if you don't set up OpenStack with Chef, you can still run your infrastructure on top of that OpenStack with Chef. Right, right, so we have Knife OpenStack because we have Knife EC2, Knife Rackspace, Knife 36 other clouds, so. So yeah, they're the different API recipes. We'll just take a quick look at Compute and probably skip over the others. But it's, we're seeing kind of this boilerplate pattern of Nova Common monitoring package, creating the Python Keystone stuff, installing the packages, installing the service, monitoring, monitoring, getting the Keystone endpoint, adding ourselves to Keystone as a service registry, all that Keystone stuff, writing out our Nova API template, and registering our Compute endpoint. So the other, trust me, the other APIs are very similar. So Nova Volume is our volume service. Is this, yeah, I bet if we get in there, back to if we get in here, include recipe Nova Release Volume. So Release Volume, Folsom Volume, is now calling sender. So a little backwards compatibility. Probably when I release my grizzly stuff, I'll drop all this, because I'm like, there's no, there's no going back. But these guys have to support multiple installations. So again, not a fun problem, but so looking into the sender cookbook, recipes, sender, is that the first one? Sender setup, API and scheduler. So sender looks like lots of other stuff. You know, syslog, debug, mySQL, put ourselves into Keystone, find the API, find the various sender endpoints, create our sender stuff on the database. In this case, we have one mySQL and everyone uses the same one. You know, you could technically break this up if you wanted to, but we don't. I mean, you could have a mySQL tied to the service, but why? You know, set up the call sender DB sync, logging, sender conf. You know, there's NetApp support. I guess you guys are doing just NetApp? Yeah, they're doing NetApp. There's also sender plugins for LVM. What is the sender NetApp plugin do? So we've got an IRC channel, Chef OpenStack, OpenStack Chef. OpenStack Chef, that like, there's usually about 40 people in it, including a bunch of rackers. And so, you know, you can find these guys and they'll answer your questions. I was trying to look through the attributes to see if it was in here, but I don't see. It's because they target, yeah. Yeah, so there's NetApp, there's LVM, there's Chef, which is probably something else that I'm forgetting, but, you know, it's been a long week. So then, oh, I guess there's still more sender stuff. You know, sender sets up the API, registers itself with Keystone, and it's going to do the sender API, the sender scheduler, you know, Keystone, Keystone. Oh, it actually uses Rabbit, installs the sender API packages, writes out its login, writes out its comp file, and its API paste, deletes its SQLite file that gets packaged in the distro package. You know, so it sets up the three sender services, and then NovaVolume. So NovaCert is the Nova certificate service. Start up the, install the packages, start up the NovaCert service, you know, and it looks at the NovaConf and the NovaLoggingConf. Not super exciting, but one of the many services, and then it adds NovaVNCProxy, which is kind of cool, because, you know, it's nice to be able to VNC right into the box from Horizon. Goes in, installs all the console auth stuff and the VNC packages, sets up the VNC proxy service, you know, and then adds monitoring for it, you know, console auth service, more, oh, so there's the VNC console auth service, and the VNC proxy service, and so it installs both of those. And so you now have Nova running, Keystone, Glance, Nova, and Horizon. Horizon is a lot simpler, well, not a lot simpler, but it calls the base role, it calls MySQL Client, it calls MySQL Ruby, and then it calls Horizon Server, and the Horizon Server recipe is probably going to do, anyone, anyone, install Apache and Django and go and turn off SC Linux. Yeah, so there it's going to include the Apache cookbook from the community site has like 40 recipes because every mod, whatever gets configured. In this case, we need modWizgi, rewrite in an SSL, and then it's going to go and write out the ports conf to not listen on all IPs. They do some more SC Linux stuff. There's actually an SC Linux cookbook that has, well, maybe doesn't do exactly this. Create the Keystone endpoints, the database, install the packages, write out their local settings pi file, or whatever their local settings file may be. OpenStack dashboard syncedDB, not sure why there's a fix me. Then they're going to pull Horizon PEM out of the cookbook and write it to this file system. I'm not a Django guy, so I don't know all the stuff that's going on here. Stop Apache complaining. And then write out the right template for Keystone. Apparently has a different, or for Horizon, apparently has a different name and location depending on what it is. This could be refactored a little. Write out their dashboard conf, remove the Ubuntu theme. Do you guys install a Rackspace theme? Okay, it's vanilla. It takes off the Ubuntu theme, puts on, leaves the vanilla theme. So, we could add another recipe that you could throw in a graphic and you could theme it yourself, okay. Okay, Dave, if you use their installer, it will brand it for you, but this recipe does not. So, I'm sure that code's probably on GitHub somewhere. And then it goes and deals with a lot of dirty hacks. I'm sure everything's probably better in Grizzly. Grizzly looks really nice. Oh yeah, here we go. Write all this stuff and pull it off of Rackspace's CDN even. Look at that. So there's use of remote file instead of curl. And then they're actually making, keep going, nothing to see here. Horizon gets set up, boom. Horizon gets set up and you now have Keystone, Glance, Nova and Horizon. You have a dashboard working and you have your full single controller is done. That box can sit over here and anybody who wants to become a compute node runs the single compute role and it calls Nova Network Compute, which we already saw, which sets up the Nova Networking Service on that box. And then it calls the Nova Compute recipe, which is not, is the last thing we're gonna see. So it's gonna go and install packages. If you set it to KVM, it adds an additional package. This could get refactored into attributes. Cause now with Chef 11, attributes can be evaluated inside the attributes file. Just nice to have. Install the packages, write out your Nova Compute Conf, configure Libvert D. So this will get refactored a little bit for the bare metal driver, where we'll, I probably has already been refactored in HubSpot's branch or their fork, where there'll be an attribute that says which hypervisor you're using and then it'll call different recipes. But the logic will, cause I think in mine, I've got LXC. You guys aren't using LXC? Yeah, I've got, so my branch, my fork has LXC and KVM. Cause I was working with Calzada and they needed LXC. But then it's gonna install the Nova Compute Service, monitor it, install Nova Libvert, remove a kernel module, and then enable IP40. And then you have KVM running on a box, pointed at your API, and you can log into Horizon, create instances. You know, depending on your hardware, it could take, you know, five minutes. If you have all SSDs and blazing fast, or it could take an hour. But it's done. So that is, that is what all in one looks like. The Swift, I mean we got like five minutes or something, but Swift actually, there's a Swift all in one that will do it, or you can run Swift on multiple nodes as well. Just for an interesting time, I knew we wouldn't be able to get to that. The various forks of cookbooks that are floating around, we're gonna try to re-consolidate, and I'll submit a talk for the Havana conference, or the I conference in Hong Kong, about whether or not we actually got that to happen. So Rackspace, AT&T, Dull, Dreamhost, SUSE, HubSpot, Opscode, and others are all working on the same code base, essentially. We all have forks of this, and IBM. So any questions? Yes. Well in this framework, we'd have to have a recipe that configures VMware for you. Okay, if you already have a VMware farm. Right, probably there you would have an environment file that has, like there's the VMware, here's the creds. I mean, does VMware really need to know about? You know, we're doing a window log for VMware, you're gonna have to tell us where we are, or code a bunch of Rackspace. No, but you know where all the bodies are. I mean, you can find everybody, and he knows a little bit about VMware. But yeah, probably by the I Summit, there'll be bare metal, Nova, there'll be, there's already KVM and LXC, and then we'll also, SUSE wanted to add Zen, and that's probably where we'll stop. For the choices, we'll probably be supported in the mainline cookbooks. And if somebody wants to do VMware or Hyper-V, we wouldn't be opposed to it. So questions, yes? So the question is, how did Razor get to this? There was, these cookbooks were going to, that all in one roll was going to be applied to a node, and then that was created in Razor. So Razor was going to provision the OS, put the Chef Client on it, and the Chef Client was going to be given the all in one roll as its run list, which would have expanded into this, and actually look like all this, and if it was running on VirtualBox, it would take like two days, but you would eventually get Razor, well, depending on your laptop, or in your case, a tablet, so it would never happen. So to VBox's credit, 4.2 is slightly faster? Yeah. No, I mean, his tablet's never going to work. Oh. I don't know, is it an Android tablet? I mean, we can probably force fit something into Oregon. But that was the, yeah, I guess the missing step there was, we were going to take the Razor node and apply this, and while I talked for the last hour and a half, it would have finished. And you know, I figured with most people leaving the conference, it would have been better, and I tried to, well, not significantly though, I've been trying to spin up a VM for the last five or six minutes here, and it's been trying to pull down the precise 64 box, and it's, yeah, I'm still at like 2%. Yeah, so it does work. Somebody does a Razor Torrent client. So, yes, it does work. It just. Yeah, I've seen it work at least once. Yeah. Well, like, you know, this one blew up and gave us a nice stack trace and told us exactly in the recipe, hey, I blew up in, you know, cookbooks, OSOps, Utils, libraries, IP location, and I was called by my SQL OpenStack server recipe. Yeah. When your production stuff blows. Yeah. Yeah, yeah. In production. So, okay, so the dry run inside of Chef Client, if I say, you know, Chef Client-W, it's actually called Y run. All right, I mean, it's Dash W. We call it Y run because it tells you the assumptions it's making as it goes. And then the whole way along, it's saying, you know, hey, this file's actually here. Here's the diff, you know, what I would change. Here's what I would do. So it tries to get as close as possible to what is there, but you don't ever know what really happens until you hit, take off the Dash W and you run it. Right, and we're, the cookbooks themselves, that everyone has kind of collaborated on, we're gonna apply to be included in Stackforge. So we will get the CI for OpenStack and, you know, the Garrett code review for those of us who are working on them. And then everyone will slave off of that for their various configurations because, you know, it'll still only do, you know, a few nodes, but, you know, everyone has a really different OpenStack deployment, but it actually, you know, since we're running things through attributes and the patterns don't really change much, it's safe for SUSE on Postgres on Zen and Rackspace on Ubuntu on, you know. Well, I think for the AV guys, we'll go ahead and call it a day. But thanks a lot for sticking around. Hope you guys had a good conference and see you in Hong Kong.