 Hey, guys, welcome to Taming Open Stack with Ansible. So I kind of want to have a different talk. You always have your chefs, your puppet talks. And back when I worked for Blue Box, we did quite a bit of work with Ansible. And I thought it'd be kind of interesting, because I've worked quite a bit on the chef cookbooks that are in Stack Forge. I've done a lot of deployments with Chef. And sometimes it's a little daunting for people to kind of just step into those Stack Forge cookbooks and kind of see from point A to point Z exactly all the attributes that have to get set, everything that needs to get done. It's a little daunting. And unless you're really embedded in that community, it can be difficult. So I want to talk about Ansible. I'm mostly going to be going through Blue Box's Ursula repo, which is, in my opinion, one of the better Ansible deployment repos out there. So just going to kind of walk through that with you guys and show you what we've done, some of the patterns that we've used to solve some kind of ad hoc operational things, integration testing, just kind of a whole different kind of things we've done. So I'm John Dewey. I don't really have much to say about that. I've done a lot of Chef, a lot of Ansible, and Open Stack for the last almost four years. So has any of you guys are any of you guys familiar with Ansible at all by any chance? This is awesome. Very cool. So this is just something that I kind of just put together here. Ansible can do multi-tier deployments. It can orchestrate. It's obviously kind of geared to be quite simple. And you're writing the AML versus custom code. And if you want to write modules, you can write modules in any language you want. And so those modules I'll talk more about here shortly. This is just a brief kind of high level diagram of what Ansible does. You know, transports here. You basically, your transports are how you're going to SSH into the box, or they have other transports available, 0MQ. You kind of have a library of modules. And then it's just a matter of going out to those servers, performing the actions it needs to perform, and it does sequencing. So you can do it on more than one of a particular type of role. Or you can just go and you can tell it to do serially. So in a particular case of that, as Precona, if you're setting up Precona or Rabbit, sometimes you want to flip it onto Serial One. And it will perform the first action first. On the first node, do what it needs to do, then drop to your second node, do what it needs to do. And it's kind of nice. It's not the whole eventual convergence where everything just runs all at once. And if it fails, it fails. But the next time the agent happens to run, it'll kind of rectify it. So along with simplicity in mind, super easy to install. You really have very little dependencies. You just need Python and SSH. You can install it through pip, apt, yum, you name it, get check it out. And you just need to install it on a server that has SSH access to those particular hosts. Serverless. So that may or may not be a good thing for some people. There's obviously camps who believe there should be servers or there's other camps who feel it's not that you don't need it. But again, for being simple, it's not necessarily a requirement. And it's a pretty nice thing to have. You just can kind of install Ansible and run it out on all your servers. No server necessary. Agentless. So again, I kind of put an asterisk next to this. There's Agent Smith up there. Again, this is a topic of high debate. You have camps who really believe there should be agents and those that don't. I have not run anything at scale with 60,000 nodes where SSH to those hosts would be an issue. But I'm sure that there's people around here who have run into those types of issues. So that's kind of why I wanted to prefix it with a star, where it may or may not be a bad thing. It's kind of up to you guys to decide your opinion. And then it's all powered by YAML, which is pretty cool. So this is just a simple diagram here to show that we've just installed three packages, SMEM, Socat, and PSTAC. This is using a syntax. Ansible has a pretty nice language. And so it is YAML, but there is some loops here. And so we're just saying let's take these three items in that list and let's just iterate over them with item and install the package using apt, obviously. So if you're using Red Hat, then you'd need to change that. But again, there's a little bit better patterns to kind of clean that up, so it wouldn't be as hard coded as that. Here's kind of going over to different loops and the expressions that they have. So I really don't want this to be an Ansible 101 talk. I kind of just want to give you some information about Ansible. And then there's plenty of documentation that you can find online if you have questions about these particular loops. Conditionals, so if for some reason you want to do something when a result, when something failed or succeeded, there's ways. In Chef, you kind of have the guards as well. Puppet has guards. This is the same type of way you do it in Chef or Puppet or whatnot. But again, it's just listed in plain old YAML. There's many, many, many modules. 62 at the moment. And my last get check out just related to cloud. 229 for various tasks. F5s, load balancers, line-in file if you want to do kind of like a nice item potent, search and replace for text in a file. Basically Ansible kind of has the notion of let's put everything inside the core library and let's support as much as we can versus the other option of let's give you some. And then if you're looking for other things, you kind of have to search different sites or different Git repos and kind of find out what happens to be the current community hotness and kind of pull that in. For example, recently I was trying to do a simple pull a GZIP file down, uncompress it and stick it in a directory. That oddly enough, I had to go search the internet for that for a particular different CMI I was using. So this kind of wanted to just go over a module file. Now kind of the important thing to see here is that you're registering a variable and then you're using that variable later. So this kind of, this small description here is saying to on local host, we're gonna go ahead and use a local connection and perform this key pair. It's calling the Nova API and generating a key pair. Once it generates the key pair, it saves it off and then you can later use that key pair and other tasks that need it. So we could have obviously upfront just created a whole bunch of variables and said use these variables, but this is kind of a nice, since we're at OpenStack, we figured we'd talk about creating key pairs, spinning up a VM, setting security groups and then using that key pair. So I kind of just wanted to highlight that in this particular example, we're registering a variable with the name of the security group and then when we're over here using Neutron to open ports, we're using that security group that we had created and then later down here we're associating a floating IP, registering that floating IP's address and then using that when waiting for it to boot. So you can just use IP and it's kind of nice. Ansible has a nice clean way to just wait. Once port 22 comes up, which we've defined with the delay of five seconds, once it comes up, you can assume that everything is at least good to go and you can continue on. And this is generally the pattern that you'll find in most Ansible playbooks or roles. This happens to be a playbook, I can talk more about that here shortly. But I think that's all. Oh, and also modules are chainable. So what I mean by chainable is I've chained the output of this one into the input of this one. And so if you write your modules right, you can make sure that your modules return data that's relevant to another, or that's relevant. And so in this case, this happens to be a module that I got upstreamed into Ansible and it returns the floating IP so you can then use it later. This is just kind of a, obviously we're not gonna go over this whole thing, but this is just an example of what a module looks like. And so this is the key pair module. So it can be written in shell, Python, Perl, whatever you really want. So long as you, with the Python, for example, in the Python module, so long as you return JSON, you're good. If you were to do shell, you'd have to return certain variables or certain key value variables for Ansible to then know what to do with it. But it's pretty much as it says here, this is black, you know, this is magic. So Ansible kind of takes care of it so long as you follow their conventions. Briefly, kind of talking about the way you configure Ansible, it has a notion of hosts in this, right here is a group. So in our example, we've defined an identity group and that's where we're like, we're gonna install Keystone or whatnot. You do have to have SSH access. You can pseudo access or not. You know, it's fully configurable. There's a notion of inventory so inventory is simply how you wanna set particular variables or it also kind of Ansible also discovers facts about your systems. So people familiar with Factor Ohio, you can also use Factor Ohio with Ansible. But by default, it doesn't use those. And so if you were to go out and deploy, it goes out and generates facts about that system. For example, interfaces and IP addresses and then you can use those inside the YAML for or inside templates or inside other things that you may want to then populate your templates or do what you would normally do with any other content management system. So in this example, we're just simply defining that identity which is our Keystone system has the following ports and we're using UID for OS strategy. And then these can later be used in other playbooks or roles to then grab that information and populate templates. Playbooks are kind of what Ansible uses. Everything essentially comes down to a playbook. You may have roles but you need a playbook to say what host either a task is gonna be run on or a role is gonna be run on. So in this example here, we're saying let's install base memcache Apache 2 and the Keystone role. And as you see here, this is kind of an interesting syntax. So this is where you can kind of override values that you had defined in your inventory with particular overrides that you may need for the deployment you're doing. So if maybe this is a dev deployment, you may want different role or different port for Keystone. And so this pattern is followed pretty heavily by the Ansible guys. I've never really gotten into it. I hadn't used it too too much but as Galaxy got released and Galaxy is kind of a site where everybody's trying to share their roles and kind of get, it's almost like the community site for sharing reusable ways to do things. So you have my SQL role, you'll have your Apache role. It's a way to share common Ansible tasks. And so I myself have never gotten into, I have never gotten into this particular pattern but it's used quite a bit in the upstream community and so I thought I'd kind of highlight that here. And again, this is how you target a role. So before when you saw here our identity group, over here when we're using the playbook here to actually execute it, it's gonna execute on the identity group. And again, I'd already talked about this. Here's a link to Galaxy for those of you who haven't heard of Galaxy before. And so here's, we'd already talked about roles briefly but this is basically how you'd find a role's layout, much like you would a cookbook or whatnot. They kind of have a general hierarchy to lay things out. If you were to use Galaxy Ansible, you can actually use Ansible to generate a scaffolding and you'll get this very layout here. And you have everything you need. So Ansible pretty much ships with everything you need. You just need SSH and you're good to go. Pretty much like a semi with a cruise ship, right? I mean, there's everything you'd ever need there. So you're not gonna see this. I couldn't fit this all on the page. This is somewhat scrunched down and there's more to it than this but this actually right here is the Ursula deployment of their stack and so their open stack stack and so it goes, you know, there's much more to this but I kind of wanted to just give you an idea that there is orchestration involved. So the biggest problem that I had when deploying open stack is you kind of have prerequisites. So you need your rabbit installed if you happen to be using rabbit. You need my SQL installed if you're using, if you have Precona set up, you're gonna want your cluster set up. If you're using rabbit cluster, you want your rabbit cluster set up, then you can finally get to Keystone. Once you have Keystone installed, then you can start deploying kind of the rest of your stack and so the biggest problem that I always had personally was just getting things to converge in the right order and getting things to happen in the right order. So with Ansible and Salt has the same kind of overlays or the same similar system, you can kind of define how you want things to occur and based on that, then you can make assumptions that okay, by this time, everything's done and I can go ahead and install the rest of my computes and so from when I usually try and get people into open stack, I usually point them to this so they can see what occurs when and also it's easy to grok. You can just look at it and say, oh, I get it. I'm not hunting around the repo to find what gets installed where and I'm not looking through all kinds of other places to figure out what happens. So just to kind of highlight some of that, so this is installing the rabbit controller. Over here we have, and setting up Precona. And this is all, I have reference in the slide so you can go look in the reference and look at Ursula and you guys can see more of this but I just kind of wanted to detail some of it. This is kind of where the orchestration is occurring and then highlight some of the kind of the relevant parts of that that would normally be a little more difficult to do, how it's kind of handled in this particular case. And so here we have an arbiter, we have two MySQL server or two Precona servers in the cluster along with an arbiter and then at the bottom we've set up a backup as well so we're performing backups. And those all equate to roles and those roles are all in the repo and they're managed all through, currently they're not upstreamed but Michael Dahan has actually had interest in us upstreaming those. Again, we're highlighting more of this, we're installing Memcache, dealing with Keystone, setting up Neutron, and then restarting OVS. And here is some more work around Swift. Set up Glance or set up Swift and then Point Glance and then set up Glance and Glance happens to use Swift. More on orchestration. So an interesting thing you can do and this is kind of a weird crazy cuckoo thing to do but people do do it. They're orchestrating using Ansible and orchestrating Chef with Ansible and some may believe this is the wrong way to do things but I don't really see it as a bad thing at least especially in the case that maybe you're wanting to switch from one to another and you just are, or you're trying to get around problems where maybe you're saying let me just use fabric to kind of do things in the right order. I personally don't see too big of an issue doing this and in a particular case and on that right now we're trying to switch to something else and I see this as a good pattern to follow where you can do it for the time being, have Ansible orchestrate something else and then maybe start switching over to Ansible and piecemealing things as you kind of get more familiar with it or as time permits. So again, this may be crazy, some of you may not like this but it's worked pretty well for us in the past. So along the lines of this kind of, I'm not necessarily talking about open stack deployment anymore, I just wanted to highlight the different ways that you could do open stack deployment and kind of point you guys to a repo and give you enough information about Ansible where you can kind of then start poking at the repos and get more familiar with it. This is now more along the tasks of why, what are other reasons you would use Ansible and I ran into this particular use case not too long ago where guys are writing fabric to do this and it's not item potent, everybody's kind of rolling their own thing, this could have been a shell script, right? But kind of the key is why necessarily do that when it's kind of already been solved for you, you can turn that into this. And this is just a task set up a network and this you could just have Ansible execute for you. So I kind of like to show the comparison because I feel like that is kind of nice, people have already done the hard work for you, you just need to just make sure that you are using it. It's also good for one time tasks. So this is a bit of a contrived example this would very likely go into Cron if you wanted to clean up your Keystone tokens, your old Keystone tokens. However, we have run into situations where people do go into the database and they twiddle stuff around, next thing you know your database is blown up and you're spending the next six hours fixing it. So I kind of, this is contrived in the sense that this isn't something you'd necessarily put in a one time task, but you would do something similar if for example, you wanted to do something else, you could stick this into a task, get it through Garrett or whatever type of, if you're using GitHub or whatever you're using and get people to peer review what you're doing and look at it and say okay, that's something that's okay for the operations people to run anytime they're experiencing a particular problem and then you have eyes on it, people can see exactly what it's doing, you could even name it correctly so when others look at it, they know what it does and then you go ahead and once that's checked in and people have agreed with it and peer reviewed it, then that's what the ops guys happen to run. There's none of this jump on the server, let's hack around and then break the database like we did. More one time tasks. So this is just another example of things you can do with Ansible and again, we're registering this here, we're just performing it so with Ansible, if you wanna register a variable or the result, you actually have to do, as far as I know, I may be wrong but I'm fairly confident you have to kinda do a shell command, it takes this and basically saves it as a variable and then later, you can do this, this will only get executed if this is not zero, right? So that's kind of the, I'm sorry, if the return code is non-zero. So these are again, kinda nice things that you can do, it's pretty easy and it's a little nicer than just shell and you can cut, nobody wants to write a bunch of if then conditionals in shell over SSH when you can just do it here and you get that flexibility. I just had to throw this in here at Heartbleed, right? I just kinda had to just throw it in here. This is again, a kind of a nice example, I grabbed this from somebody at Art, when Heartbleed occurred, this was already posted, this was already out there but it's, the thing about this that I thought was nice is you know, sure you could just for loop, you could just go ahead and bang it out there but there's a little more, it's a little more elegant than that, you can just, it's again using the register, it's a little more of a, I guess I would say, it's a little more of a cleaner approach and the Ansible guys release this as soon as Heartbleed came out and I just felt like I had to throw it in there. And then integration testing, so this is kinda one of the parts that I thought was pretty cool. So when we deploy OpenStack, you either, it's either working or parts of it aren't working or you kinda wanna validate that what you did in your roles and playbooks actually perform those actions on the server. So sure we can run any other, we could have run Tempest for example when we do but it's nice, this is kind of an example of for everything on the first controller here, go ahead and run these commands to validate that at least that bit of functionality that you had done is working. Whoops. Yeah, I'm late for stand-up guys. But I find this actually pretty cool. This is actually doing things kind of the old way, Ansible, I'll get to that next. Ansible has actually has a way to deal with this with they've added some nice assertions to the language but this is the way that we're doing it currently and haven't migrated to the new assert language. But I actually kinda like this. I think it's pretty reasonable. Obviously there's a lot of shell commands but this is usually kind of a one-time thing that you run after you run your set of playbooks and so I thought this would be nice to highlight. Here is the new assert language. So Ansible's kind of added this whole assert that and then this particular result of a variable. So they save off this register here, so this d-package result and then later they use it in their assertion. And so this is used extensively throughout the Ansible roles that they provide. And this, eventually at some point, I guess if we wanna do it right, we should start using this language in here. So common patterns. Obviously this isn't too cool here. I don't like the fact that I need to take all of this and copy-paste of this into five other places throughout the code. I kind of wanted to take this as a point to kind of say, this particular pattern that I have, I don't know how to solve and it's, for example, like Chef has libraries. You can have library cookbooks. I could easily hide all of this into a single library cookbook and in my template just invoke that method call and get back my list of rabbit servers. If any of you know how to do this in Ansible, haven't figured it out. I don't know a clean way to solve this particular problem without copying it all over the place. And there's plenty of other patterns I have like this that tend to get placed into other parts of Ursula. And so I kind of wanted to highlight this as Ansible's super awesome, I like it. There's just certain patterns that I don't think it solves. It doesn't solve everything as far as library patterns. And variables can be tricky. I've spent so much time dealing with variables and trying to get things to follow, I guess, the mindset that I had when I had used Chef that it doesn't exactly equate right into Ansible. So with Ansible, it's the variable defined closest to the role generally wins. So way back, that's probably a bad idea, but let's do it anyways, I got time. So way back here, right here, anything defined in here always wins. So if you were to want to override this, the only way to override that is to define it is to override it here. And the reason that is is because you want to look at your, I guess, well, the main reason that that was there is you want to know exactly, you don't want to be looking at something saying where that variable gets set. Kind of looking here, here, here. You want to be able to look right where it was executed and see where it was defined. So it's kind of a simplicity, it's a take on simplicity. You know exactly where to look and you're not hunting all over the place to find out where things were defined. With that said, though, it has turned and it has been a bit of complexity on my end. I'm not exactly entirely happy with the way variables are. And maybe that's just me, I need to change my particular outlook on this, but when sharing roles amongst the community, it'd be nice if there was a way to better handle variables. And so in this particular case, so in this particular case, I've defined memcache on this particular port and kind of wanted to highlight this. If in Horizon, I wanted to have Horizon use memcache and pull in those variables that were defined in this particular role, I would have to add memcached to that playbooks run list, essentially. Otherwise, those variables are out of scope. You can't get to them, which has unintended side effect of installing memcached on that host. You may not want memcache installed on that host. So again, this is just a scenario that we ran into where I may not want memcache installed on here, but I kind of want to be doing the right thing and I want to use the variable set in this role. So there's quite a few answerable people here. This may be a problem that's already been solved and you guys are saying, I may not be doing it exactly correct, but this is a use case that we ran into that was somewhat annoying. And again here, this is again a slightly annoying problem where you just have to be careful with how you use variables. It's very simple, Ansible is simple. The playbooks are simple, but you just have to be cautious with what you do. In this case, we're saying in this inventory, we're gonna cluster rabbit. And so when you run Ansible, it's gonna parse the inventory, it's gonna parse those group variables and it's gonna say, okay, we're gonna cluster. Down here, if you happen to redefine your vars to say the default file that rabbit had, it would then potentially reset this to false. And so this is probably very likely a bad pattern to do. Don't do this anyways, but again, you wouldn't know this unless you actually did it that this is kind of a weird situation where how things can get rendered depends on where you are in the site file. And so the way we solved variables is we decided let's just stick everything into inventory. We wrote a custom plugin for it and we make sure that everything is defined in one spot. We don't have to worry about if things get defined in roles or whatnot. Define it in one place and we ended up making a plugin so, well, I didn't, Craig actually did, but we ended up making it a plugin. So for the pattern of, we were deploying many different open stacks for different customers. And so you have a lot of common things that get repeated over and over again. You have your common ports, memcache runs on the same common ports. Most everything kind of stayed the same, but there was other things that had changed. So we wrote this plugin that basically just combined everything, merged the hash into the way we liked it, and then Ansible uses that. So it solves a lot of the problems where defaults are defined here, overrides are defined here. We just kind of do it in one place and it all trickles you down to the rest of the system. Ah, DevStack, I thought I deleted this. So this is kind of a, we could obviously discuss this till we're blue in the face, but I was thinking kind of getting your feedback at, you know, DevStack's pretty much all shell code. It would have been kind of interesting if DevStack might be implemented in Ansible. It's just a thought. I'm not saying it should be done, but just kind of to get your feedback on it. I think, you know, if DevStack was written in Ansible, or most of it was, you'd already have, you know, you could switch some variables and, you know, deploy from packages if you wanted, and you would already have a open stack that's deployable out of the box because you're deploying it on your local box using Ansible. So you kind of, it was more of a thought and just thinking that might be an interesting, you know, thing to do, but it by new means was intended on something that we have to do. It was just kind of a thought. And that is all. Thank you. Questions? How was that good? No questions? Got one guy right here. You can go to the microphone. Thanks for the talk. That DevStack thing is a crazy idea about us thinking about it too, so. It's a total crazy idea, right? Just for fun, it could be fun. Quick question, do you have any recommendations on how you might share roles? I know about Galaxy, but inside a private team without putting it on the public Galaxy site. So my buddy Craig actually wrote this pretty, I'd have to find the link, but he essentially wrote this tool that you could give it a repo and you can, it's almost like a Burke shelf, but not for pointing in and saying, here's a repo, here's either a branch or a tag or a get URL, and pull that into your Ansible repo. So other than that, there is no, that I'm aware of no other way to do that, other than pulling straight out a Galaxy or having some other solution. He actually wrote the solution. It's actually pretty good. And you could actually, you very likely use it with Galaxy if you just have to update it to understand the Galaxy API. But other than that, maybe you could sub-module if you knew the get URL, you could sub-module it, but some people don't like sub-modules, so. Great, thanks. Sure. You got anything? No? One of your examples, you were a validating state of your deployed cloud, sort of like doing, you know, post run tests. How does Ansible handle failures in that case? Do those tasks, when they failed, does it exit out or does it log in and continue? Yeah. If anything fails, it errs and exits out and then it leaves you at a state where you can, basically leaves some files on disk that you could then resume from. Yeah, cool. Thank you very much.