 Alrighty, good afternoon, everybody. I hope you all had a good lunch. So this is the session about making an OpenStack Ansible work for you. My name is Andy McCray. I am the current PTL project technical lead for the OpenStack Ansible project. And I was the PTL lost cycle as well. So for O'Connor and Pike, I've been the PTL. I'm a software engineer at Rackspace working on the private cloud team. I've been involved in OpenStack since around the Essex cycle. My first summit was actually in San Francisco, which was the Folsom Design Summit. And during that time, I've done work as an operator. I've been a systems administrator. And for the past couple of years, I've been doing development work mostly focused on deployments. I worked on the OpenStack Chef project for a while and lately the OpenStack Ansible project. So it's really just been about deployments and repeatable deployments and deployments at scale. So I'm really excited to be here and share my passion for deployments with you. And I'm glad you could all be here and I hope it'll be really useful for you. So I don't know if you were at the keynotes yesterday when GE did their talk about their OpenStack deployment and had the one slide that said, this platform must be made of awesome. And as the PTL of OpenStack Ansible, I can confirm that it in fact is made of awesome. The GE deployment is based on OpenStack Ansible. So all the OpenStack Ansible work that we've done is what was used to deploy the GE deployment. And it's not just GE. In the OpenStack Infra team where they have hardware to do gating, there was a very large OSIC cluster, which has now gone away. But when it was there and it was deployed using OpenStack Ansible and it was a 500 node compute cluster, it had some ironic in it. So really it can do scale at high performance. And it's really just about getting production workloads for OpenStack and deploying in a way that allows that to happen. So in February this year, I wrote my first corporate blog. It was less technical than what I would normally do. And it was more for a broader audience. And so one of my friends, Kevin Jackson, retweeted my tweet with my blog in it. And one of his followers on Facebook had this to say, I use it because it works. Everything else, not so much. Which I took to mean your blog kind of sucks. But OpenStack Ansible is pretty great, which is fine. I'll take that feedback. Actually, it's exactly the kind of feedback we want. It's great to have feedback that suggests that what we're doing and what you're deploying with is really great. And you know what, Sean, if you're about to come to me afterwards, I'll buy you a drink for sending out one of my slides. And I'll be working on my blogging skills in the future. But it really highlights our approach towards deployments, which is we don't want fancy flash work. We want to do things that work. So we want everything we do is to provide the best mechanism for deploying and managing production OpenStack environments. And that's at the heart of all our decisions. So we want to take into account upgrade, scale, customization, manageability, repeatability, all these things that are important to production deployments. We don't want to use the new fancy thing. We want to use whatever is best, whatever works best. And we'll get you the best production deployment that we can create. And that's to say that if some research showed that Windows 95 was the best operating system to deploy OpenStack on, we would start working on getting that as our stable deployment. Of course, it's not. But it's just an idea of how we do things and what drives us when we develop things. And it's been the key since we started off OpenStack Ansible as a proof of concept in Havana with the aim of deploying some infrastructure services and containers and seeing how we could make that work. So everything else is secondary. I mean, even if you look at the name OpenStack Ansible, it doesn't really take an analyst to tell you what OpenStack Ansible is. It's a project for deploying OpenStack using Ansible. There's nothing fancy or flash about it. It's just a key to what we do. So this talk is really about helping you to get the kind of great experiences that these quotes suggest that the people using it have had. And I'm hopeful that I can get some more quotes in the future that I can use in other slides because the more people that we get using it, the more positive feedback we get, and the more we can help build this project moving forward. All right, so who's this talk for? So anyone who likes terrible metaphors or triathlons or both, this is a talk for you. So it's really about the kind of people who can contribute and benefit from using OpenStack Ansible and what we can give you and what you can give back and how you can utilize OpenStack Ansible to achieve your goals. So I've kind of split it into three groups. So we've got the cyclists, which are the operators and the deployers. They've got their own hardware. They've got their own tool sets. They just want to get on the bicycle and they want to ride. So they just want to take OpenStack Ansible and they want to deploy it straight away and they want to get going as quickly as possible. So yeah, that involves using their existing hardware, not being forced into using specific things by the project itself and being able to customize things and use what they already have and build on that. We then have the runners in the middle there, which is the developers from other existing upstream projects in OpenStack or outside of OpenStack. And what they want to do is they don't want to get too deep into OpenStack Ansible itself, but they might want to implement their project in OpenStack Ansible, get more exposure, get more people deploying it. And they could do things like test or develop using OpenStack Ansible, which are some of the things we've been working on improving. But really what they want to do is just keep doing what they're currently doing, but maybe get a little bit of a help from the OpenStack Ansible project. And then lastly, the swimmers. These are the new contributors that are interested in developing on OpenStack Ansible or contributing documentation or ideas or anything like that. And they want to deep dive into OpenStack Ansible itself. So before we get into specific examples, I want to talk about the building blocks of OpenStack Ansible and around the structure of how it's built. And the main aim for this is to give you an idea of what we're working with. And I'm hoping that that will show you how you can then use that to customize and achieve the goals and the requirements that you have. So OpenStack Ansible is made up of a whole bunch of repositories. We have, for example, an OpenStack Ansible OS NOVA role, which is in its own repository. And that's really the OpenStack Ansible role for NOVA. Similarly, we have one for Keystone and all the other projects and a couple for non-OpenStack specific things like your database or memcache or stuff to set up Alexi containers. But we do have the overarching OpenStack Ansible repository. Now what that is is a kind of opinionated idea of some variables and some settings and some playbooks, which are a bunch of a way to run roles against hosts and documentation. And this kind of gives you a kickstart. Like you can go in there and try this thing out and get it going. And it'll give you some base defaults of what we think is a good starting point. But that's not really where the power lies. The power lies in the fact that you can customize all of this. So the documentation that's there does already tell you how to customize some of this stuff. We have examples, commented examples for how you'd set up your inventory. We have guides on how to do deployments and not just a quick start, like a full deployment and various other guides. So as an example, your hosts don't have to be containers. They could be on metal. They could be VMs. They could be anything, really. All that OSIR and Ansible cares about is that you have a host that it can target and log in and do some work on it. So set up the service, install the packages that are acquired, set up configuration files, these kind of things. And that makes it incredibly flexible, because we're not telling you you have to deploy containers. You have to do this. You can decide that you don't want to deploy containers, that you maybe want to do things on metal. And there's a really great example of that in the community at the PCG. One of our operators came to me and said, well, we've decided we want to deploy Swift as a standalone. And we're going to do it at scale, and it's going to be a large deployment. And we're going to test out Swift on PyPy. There was a talk about that a couple summits ago about how it can improve your performance. But it's a little bit more performance-intensive. And they found that putting the Swift proxy service, which by default would go in a container, wasn't good enough. It was a bottleneck, and it was slowing everything up. And so they wanted to move it on to metal, and also allows you to do this. You just tell it that the host to deploy Swift proxy on is a physical host, and it goes in and deploys it. And even more than that, they wanted dedicated memcache servers for each proxy host. And so they told it that memcache should live on the proxy servers as well, and it did it. So there was no needing to rebuild some container that isn't been done before and then deploy it. It's just tell it where to deploy the things, and it will deploy them. Now, if you have containers and you want it to be in a container, OSU will set those up for you and then do the deployments and everything for you. But you don't have to. It's very customizable. And so at the end of the day, our role is just a set of tasks that target a group of hosts and handles deployments and orchestration of the specific service. And you can specify variables to change how it sets it up. You can do that on a global level. So for every container, you can do it on a specific container level or on a group level. I could set all keystone hosts to, for example, mount a specific file system into a container because it contains something that I need. And then it would only apply to those specific hosts. All right, so now we're ready to go. We've got the building blocks. We kind of know how it works. So let's do some simple examples of you're an operator or a deployer and you have an existing, let's say, database. You've got database administrators. You've got a full cluster. And why would you want OSU to deploy your database? Like, you wouldn't. You've already got an existing thing that works, right? And you can already do this. So there are a couple of variables that you'd have to set. So there are two options. You could either set it up with the database, set up all the databases in advance. So for example, set up the nova database with a nova user and provide OpenStack Ansible with the password and database names to those databases. Or you could allow OpenStack Ansible to access the database via, by giving it the address of the database server and then giving it the root password and root user. And then it will set up the required databases and users and everything with the right permissions for you. So there could be a little bit of heavy lifting. But the idea is you could then set up your own playbook and call, for example, the nova role. And in that, you could tell it to create the database in your own way without having to do any deployment of databases to use your existing infrastructure. And logging is pretty similar. So we have an option, which is our syslog client user-defined target. And there's documentation for that, so you don't need to remember the names. But essentially, you can specify an external logging host that all the OSU services will then ship logs to using our syslog. So you could have like a Splunk or a Logly server that's set up externally to your infrastructure. As long as you can connect to it and ship logs to it, it will work. And it will just set it up for you. So it's great if it's already enabled, right? We already have database and logging support, so yeah, that's easy. You just set the right variables and maybe do a little bit of setup before, and then off you go. It's all set up. But what if it's something a little bit more complicated, like a brand new service that you want to integrate? Or you've got existing infrastructure that isn't something so simple. How would you do that? So rather than talk about hypothetical things, I'm going to give a real example. So in the OcataCycle, we added SIF support to OpenStack Ansible. So what do I really mean by this? Because you could always deploy OpenStack Ansible with SIF even before Ocata. But we added automation around deploying it. So we did all the heavy lifting. I mean, really what that is is creating playbooks that will run against our hosts and environment files, which is essentially the definition of hosts and groups so that you could then run, for example, the SIF-mon hosts in containers if you wanted to. We added integration variables. So the kind of variables that we'll integrate with, say, Cinder needs to know about SIF variables. But I don't want to have to specify them twice. So now Cinder will pick up your SIF variables. And Cinder will automatically use SIF if you have it in your environment, things like that. And then automation to clone the SIF role from upstream. And finally, and definitely not least important, testing. So we have a full deployment testing on our OpenStack Ansible commits that will test a SIF deployment with OpenStack and then do some workloads on that. But what we didn't do is rewrite SIF tasks and create a whole new SIF role and do all the SIF things. Like, there's no need to do that. There's a really great upstream SIF Ansible project. It's really well maintained. And it works really well. And we just integrated with that. So we tested that it would work for us. We fixed a couple of bugs upstream. And then we made sure that the community in the SIF Ansible world was receptive to commits and we're keeping it maintained, and they were. And so we just implemented and utilized it. And the way that OpenStack Ansible's designed, we can link into those roles, the idea of roles, the idea of host groups, and we can easily get things going together that way. It's really designed to be very modular. So how can you possibly know all this, right? Like, it's easy for me to stand here and tell you, oh, there's this thing you can do and you don't know how to do this. Well, you can do it this way and it's really easy. But we'd prefer that it doesn't work that way. We don't want people to sit there and go, oh, well, maybe I could do this or I don't really know what I can and can't do. And I really don't know how I would even go about doing that if it's even possible. And so we've put a lot of focus on documentation. A huge effort. In the Newton cycle, we restructured our deployment guide and we were the first deployment project to have a deployment guide in the newly found deployment guide section on docs.obstack.org. And what that is, that's not a quick start guide. So that's not a, like, click one button and it deploys an AIO. This is, how do I set up my networking? How do I set up my infrastructure? What kind of things do I need to do to my hosts to ensure they're ready to be deployed on? What kind of options can I do to set up containers or not containers? How do I do that? These kind of things. So we put a lot of work into that. We strongly believe that making OpenStack Ansible easier to use and deploy and helping the community will result in more contributors, which will in turn result in a better tool for everybody. We actually have two core reviewers that are dedicated to doing documentation in OpenStack Ansible. And their job is literally to help with documentation patches, help how the documentation is structured. And we think that that's a show that we really are interested in making this easy to use and operate and a community project. As a developer, I can write documentation and I can put that up there. But when you look at the difference between what you put out in documentation as a developer and what comes out the other end when it's been through one of the people who specializes in doing technical writing and documentation, the difference is amazing. It really is completely different. And we think that that's helped a ton. And then coming up in the Pike release, we've added an operations guide, which I'm super excited to say we've actually moved from draft to being a non-draft guide now. There's a little bit more work to do and it will be ready by the end of the release, but it's at a level where it's readable and there's some useful information in there and it's been agreed by the docs team that they can move it from a draft mode to ready to release. So some of the sections I spoke about before are already in the docs. So there's a section on integrating Ceph Ansible with OpenStack Ansible, using OpenStack Ansible in an existing Ansible structure and there's also a section on including OpenStack Ansible in your project. So the difference between that really is if I have an Ansible structure already and I want to include OpenStack Ansible on top of that versus I have OpenStack Ansible and I want to include roles in that as my base structure. But there's documentation on both of those that were pretty much put forward by the community. So there's a member of our community who actually installs his base set of things and then with OpenStack Ansible on top and he wrote the docs for that. The database logging and monitoring integration that I spoke about before, there are three of the pieces that are kind of to do before the end of Pike for the operations guide. So there's actually a section there already but it's empty at the moment for the monitoring one. All right, so moving on to the developers of existing upstream projects. So in the last couple of cycles, we've had a ton of new roles. Last cycle alone, we had Dragonflow added which is a network flow tool that plugs into Neutron. We have Octavia added last cycle. We've had other projects like Moltenion and Almanac. This cycle, we've got work happening on Searchlight. Carbore is on the way. But the problem is that that's a really daunting task. I'm an upstream developer, I work on Python. I don't necessarily know anything about Ansible and I don't know anything about OpenStack Ansible. So how can I ramp up quickly? How do I get this going? So this is another thing we're putting focus on. It started last cycle with a cookie cutter repository that's actually in my GitHub repo which is essentially just a skeleton of what you'd need for a role and it includes a couple docs on what you need to do to kind of get it to a workable thing for your role. And after you've done that, you'll have something that passes tests and passes gating and it'll just do a convergence test but it's a good start point for someone trying to develop a new role. And then you can just add tasks. You can add the playbooks and environment files and you can get going that way. But that's a little bit like copy paste and it's a little bit manual and it's not the best way to do it. So we're evolving that slightly and one of the people on my team is creating an Ansible Galaxy role which will be templated. So you can pass variables for the role you want to create and it will go and create a structure for you like every other OpenStack Ansible role and will include some kind of task structure as well. So the kind of set up tasks and install tasks that you would normally do. Of course it won't have packages set up or anything but it'll show you where they should go and it'll give you a really good idea of how to start. There's already some documentation up in developer docs on adding new roles. And then there's also documentation for how we do like tagging in OpenStack Ansible which is kind of the concept of you have tasks that are grouped by tags. So you can run against a certain tag and we have kind of best practices for that. We also have best practices for how you would do your YAML. So in the tasks and in the playbooks how your YAML should be structured to match the rest of OpenStack Ansible. And we really wanna add to this as well. So if you're looking to add a new role or if you're already trying to add a new role please give us feedback on what is missing, what would help you, what the blockers are and how we can make it easier for you because again it all comes back to being in a community where we want to help other people get to use OpenStack Ansible. We think there's a huge benefit in that. If you've developed a new role we've had really great success with people coming in and putting a role and then continuing to maintain that role because they like how it deploys so much that they use it for their own kind of sake. And we think that's really the way forward in terms of maintaining projects. There's something like 60 projects in OpenStack it would be impossible for us to maintain all of those projects and so by allowing developers to come in and by helping them to get on board with OpenStack Ansible we think that they will help us to keep these maintained and up to date and so far it's worked. So this is something really new that we've got a proof of concept up for using Keystone as the role that currently works in this flow. So we've always had a development workflow which was essentially because we deployed from source you could tell OpenStack Ansible that your GitHub repo for Nova is your own repo your own fork of Nova and you've made some changes there. That's a bit of a clumsy workflow though because you'd have to develop, push to get then run OpenStack Ansible then deploy and then it'll get your code. It does work but it's not great. So what we've put in the Keystone role is a way that you can dev locally it will then mount that into the container and install off the local copy of the repository that you've worked on. And that way you can literally do code run a playbook and it will deploy Keystone again for you and there you go you can now test the code. So the old path still exists for all the other projects but we're looking to make it more generic so that you could literally just put any project in there and if it isn't there it'll just get it from Git and if it is there it'll use that. The benefits that we think come from this are that you can develop functionality that you can't currently develop on using for example DevStack. Using containers you can do really cool things like test upgrades, test multi-region Swift, test a federated auth with Keystone and various other scenarios by just setting up multiple containers and putting them on different networks if you need to but that kind of path exists. And I think there's a lot of application that could be used for numerous projects to do that and we think it would be really cool. And again it comes back to the idea that if we can get people developing using OpenStack Ansible as their base they're gonna add features for us because hey I wanna test this feature that I just added here's a way to do it in OpenStack Ansible added it to OpenStack Ansible done. So this links in really nicely with testing your project using OpenStack Ansible. So this is something new that we're looking to do since the PTG we had a chat with the Keystone people and this is kind of the reason we added the Dev workflow. They're interested in doing zero downtime upgrades for Keystone as a gate check on their commits and OpenStack Ansible already has a gate check that does zero downtime upgrades for Keystone and checks that it literally does a test as the upgrade is happening to ensure that you can still hit the API throughout it. So we have this test already and so why should Keystone go off and write another set of tests within DevStack to do the same thing that we can already do? And so as part of that we've set up the kind of developer workflow for Keystone so that we can then use Zool cloner to use the version of Keystone that you've made the patch on and then do the upgrade testing. But we think that this could apply to other projects. So like I said before like Swift multi-region that would be awesome. We already have a gate for that. We could set that up. So it would clone the Swift source code that you've put the patch on. It sets up I think three containers per region. So one proxy and two Swift storage containers per region. So six in total and then it performs things like it performs the functional tests that Swift have. But also like Cinder do upgrades that are supposed to be zero downtime. If we could test those by adding in a scenario for OpenStack Ansible and then all we'd have to do is enable that gate to run against Cinder commits as well. And even if they're non-voting that's still a really powerful and great way that we can give back to the community because we're doing that work already. And then in turn we can get the benefits of being part of the upstream community as it grows. Like staying at that edge where they're putting code and then we're testing it straight away. To us that's a really key aim. We think that if developers get on board with Ota and we help them we're gonna have this amazing relationship where it's just gonna, things are gonna move a lot smoother for us and for them. All right, so OpenStack Ansible and OpenStack in general can feel pretty daunting as a new developer. I think we've all been there. We wanna make that easier if we can. So we have a couple of things that we do. There's a bug triage once a week which is great for operators as well because you know that if you file your bug the following Tuesday it'll get seen to and at least comments it on and maybe even fixed. We have a pretty active ISC channel where there's always people willing to help. That's operators, developers. It's hashtag OpenStack Dash Ansible on a free node IC. And the focus on documentation that I've been talking about, that to me is a key thing. We're aiming to help new contributors come on board and new users get on board quickly. And improving the documentation is a really, really great place to start. And it's a really great place to learn about a project and get in-depth knowledge on it. Finding and fixing those documentation bugs is one of the best ways to get involved in a new project. I know that I've done that before on brand new projects is just get involved, start reading the documentation and start fixing bugs for that. And it's really great for us too because there's actually one of, a person in the community now who's going through the documentation. He's got tons of experience in Ansible, but he's decided that the best place to start is with the documentation. And he's found a whole bunch of things that we can fix and improve in the documentation because he's got a set of fresh eyes looking at this project and you. It's easy for me to say, this all works and the documentation makes sense and this is easy. But when I do it every day, it is to me. But you coming in as a new person, it's much more difficult and we'd like to know where we can improve that, where we can make that better. And to me, that's a really great place to start. Because at the end of the day, if a user can't deploy OpenStack Ansible after reading the documentation, then the docs are broken and we need to fix those. But we don't necessarily know where they're broken or how we can improve them. And so getting people going through that and filing bugs and fixing them is super important. On that note, there is an onboarding session tomorrow at 9 a.m. in room MR 101, so downstairs. So we're going to be running through the kind of structure of OpenStack Ansible and hopefully talking you through ways we can get like a first commit in for OpenStack Ansible and kind of get people onboarded if possible. So if that's something you're interested in and you're a new developer looking to get involved in OpenStack Ansible, please come along. We'd love to speak to you. We'd love to help you get on board with that. So I hope to see some of you there. So I think my key takeaway here is that we can all be contributors. In a triathlon, you don't get to do the running or the cycling if you don't pass the swim. And really, contributions come in very different shapes and forms, and they don't always, and they're all really useful. I know that people always think contributions and they think, oh, it must be a developer writing code. And that's really not it, especially for deployment projects. We rely very, very strongly on our employers, on our operators to let us know what's important, to file bugs and let us know things are broken, to tell us their use cases and where we can improve them, to let us know what they think we should be doing moving forward, to let us know that our documentation is good or it sucks or there's something missing, and we need all of these things. And that you can do without even signing up to Garrett, like you don't even need to be able to submit code, you just need to let us know and file bugs. And on that note, actually, there's another session which is literally straight after this downstairs, which is the operator feedback session. It's a forum session, so it's not gonna be me just talking, which is probably a good thing. But we are really off the feedback from any operators about what we can do to improve and make things better moving forward. So please come along if you'd like to give your feedback or let us know what we can do to improve. And for developers, they can help our improve our project by adding features, by just using it, and developers by their very nature like to fix things, like to improve things, like to add features. So I firmly believe that if we can get kind of more developers on board using either implementing new roles, developing or testing, using OpenStack Ansible, that we will get more, like our project roles will be more close to how upstream is, and we will have less of a hard time maintaining things. And that's really the aim for us is to kind of give back to the community and to have the community help us in return. And we'd obviously all love to see more OpenStack Ansible developers. But yeah, we firmly believe that if we can create something that works and is useful for everybody, that then you'll want to help us improve it, and not just for yourselves, but for everybody. And we've seen that work in the past, and we hope to continue that focus. And if you'd like to know what we're doing in the next cycle, and this cycle, and the cycle after that, our project update is tomorrow at 1.50 PM. So please stop by for that as well. I'll be running through what we've done in the Pike Cycle or what we're doing. And some of the things we're looking forward to doing in the Queen Cycle and beyond, and kind of where our focus is lying and what we're thinking of doing. So that's pretty much it. I think we've got about 10 minutes left for questions. So if there are any questions, or if you have questions that you remember later, feel free to reach out to me on Twitter or on IC or in person, of course. So yeah, I think, again, the key thing is like, we wanna be helping the community to help us. And we've done a really good job of it so far. And I think that we can help you to get your use cases done, and to deploy production workloads of OpenStack so that we can give you great experiences. And in return, you can help us make the project better. So thank you. Onto those, yeah. Okay, so just to repeat the question because I know it's been recorded. So if you have existing infrastructure that you've already deployed, is there a way you can kind of hook in that so that it implements your inventory? That's pretty much a TLD other question, yeah. So we don't right now. What we do have though is a bunch of operator playbooks. So this is kind of new. We've created an operator's repository to put kind of playbooks and tasks that operators specifically use. As far as I'm aware, that isn't in there. Although, what I would say is that I operated a small lab, it was about 10 nodes just for testing and developing. And I did utilize an inventory that essentially deployed the physical hardware using a Pixie boot system, and then took the same inventory and ran against Ansible because at the end of the day, it's just Ansible inventory, right? So it would be reasonably trivial to create that as far as I know we don't have anything that's directly built in. We do have ironic support, for example. So one of the aims with the ironic role was to create a set of hosts that are your ironic servers, if you like. So not the service itself, but the servers you're deploying to. And so if you use those to deploy onto you, then have knowledge of those nodes to then do other things on. But it's not directly there right now, so yeah. To be honest, I would speak to one of my colleagues, Kevin, because I know he deployed huge labs, including ironic labs, and there's no way anyone's really gonna go through and manually do that, right? I mean, once you've got like a 500 node cluster, there's literally no way you're gonna sit there, typing in 500 nodes. So I'm pretty sure the functionality already exists, but I'm not sure it's already in the upstream repository that we have. So I can double check for you, though. I'll get your name afterwards, and then I'll follow up with you. I know. Okay. There we go. Okay. Anybody else? Again, if you remember something you wanna ask afterwards, or you think of something, feel free to come speak to me or anyone else on the Open Stack Ansible team. We do pride ourselves on being a reasonably friendly community. So, see ya. Some of us find another, yes. Cool. All right, thank you very much, everybody.