 Okay Good wow full house Nice feels good Okay, let's get started so So what I think that I didn't Anyway, so welcome. Thank you for coming This talk I'm going to describe how I've been contributing to OpenStack for the last almost a year And I did not install a single DevStack VM in that year So start telling you a little bit about myself. I've been working with Rackspace for a little bit over a year A couple things I contribute to OpenStack mostly the heat orchestration project and I'm also in the team that Created OpenStack Ansible, which is the project. I'm going to talk about today I Do a little bit of contributions in other areas Keystone horizon I am an O'Reilly author. I wrote the flask book the one with the dog the cover that you may have seen So I tend to gravitate towards things that are web development that that's that's what I like and I feel more knowledgeable and I have a blog also where I blog about flask and web development on Python and that type of thing so Just to give you an idea and if if you you think this is not interesting you can just go find something else to do I'm going to talk about the problems I have with DevStack. I'm gonna start with that Then I'm going to show you this this project OpenStack Ansible. Give you a quick description I'll tell you how to install it Which is very easy. I have a demonstration and Then in the end I'm going to give you some pros and cons This isn't a project that you can use for every type of OpenStack upstream work So I'm going to tell you what works and what doesn't so it's not really, you know, the Full replacement for DevStack if you were hoping for that. I'm sorry Okay, so what's wrong with DevStack? There's a much that's wrong in my opinion But a few things to bother me So for starters when you install DevStack, it's an all-or-nothing distribution You you have to install you have to select what you want you get it installed and that's it basically, if you need to change something you have to start over right you have to unstack and then You don't change the config and run the whole thing again So you cannot mix and match you cannot modify after you do an installation You get everything installed in in the same environment, which usually a VM or a cloud server so all the projects share the dependencies the Basically you have to make sure that everything is consistent if you You're working on one of the projects So you update your requirements to match that process that that project You're progressing and that screws up everything else, right? So you're forced to go and fix all the other projects So a lot of problems Some extent I have to be fair Recently DevStack added support for virtual environments, which alleviates some of these problems So I should mention that I'm not sure how how many people know about it But you can set up virtual environments per project now Which is it's a good step forward. You're still sharing packages at the OS level though so So open stack Ansible this used to be called OSAD open stack Ansible deployment Now that we were accepted into big tent. We renamed it to open stack Ansible or OSA It's all the same thing. It's just how things progress and tend to change very often I'm sure it'll be named differently for the next summit. I'm pretty sure so In in one sentence what this project is it's just a set of Ansible playbooks that deploy open stack So that that that's all we have It's an initiative of Rackspace We use it to deploy our private cloud product. It's built on top of this But this is completely vendor free it installs in the same way as DevStack It gets the sources out of the the get repositories and installs a pre Pre-generic cloud if you if you get it from this repository open stack Ansible on the open stack namespace So we don't have anything that's Rackspace specific here It's a pretty standard open stack and the big difference between this and DevStack is that Services are installed in LXC containers So For some of you if I say containers you may think Docker. So LXC it's it's a different option which Some of us think that it's more powerful When you create an LXC container you get like a VM. It's not it's not an application container It's basically a VM or you can do whatever you want. You can start multiple applications It works in the same way as a VM, but it's a container So that's basically what we do here. You have a little diagram So this is not the way you use it to develop but this this is how you get a private cloud that that can be small or huge Basically you have a the horizontal bar. That's your switch and then you connect the external networks top on the top left You have load balancers next and then in the bottom the first box with the Many gray boxes. These are your infrastructure nodes. You can get many of them and Each gray box inside it. It's it's a container So that that's how you get redundancy. This is you know production ready Next you have the compute nodes where your instances live and then the last two set of boxes are for storage So, you know, it's a pretty standard open stack cloud But of course if you see this and you're thinking I mean this this is as far as as it can be from Dev Stack, right the single node where So What's the trick? When we work on open stack Ansible we For for convenience we work on a single node So it is possible to install open stack Ansible in a single node We use it for our gate and for our development every day So it's fully supported. We have scripts that perform installation for you. So you don't have to worry about it You can deploy it on a cloud server That's what we usually do when we work on it or when I work on on open stack heat I have a rack space public cloud server and you know, I just put it there The requirements are a little bit higher than for that stack. You need 16 gigabytes and 80 Seen a bit of RAM and 80 gigabytes of this space and that that's for the container, you know overhead Right now we only support Ubuntu so Ubuntu trustee is the only platform where you can install this at this time Hopefully that will double change And One nice thing that you get from containers is that even though If you install in this way, it's a single node you you can get redundancy so you can install You know a load balancer you we install HAProxy to load balance multiple instances multiple containers of the same service And I'm going to show you all this in a little bit. So you'll see how that works so I've never described the process the installation process, which is very simple So hopefully when this ends you all gonna go running to you know, start a cloud server and try it So you need a 1404 fresh VM Or cloud server So you need to clone the project so open stack slash open stack ansible We put it in slash opt slash open stack ansible but choice is yours and Then you have a Set of three scripts that you need to run the first one which comes with the project is called bootstrap AIO This creates the the configuration files that are proper for for an only one So you don't have to worry about knowing how to configure this Which you can if you want to deploy it in a different way But but we have a script that does it all for you Once that script runs you have a directory it see open stack deploy where all the config files are So you can go and make any changes before the installation occurs So for example, what I typically do is I go set my admin password any passwords that you don't set the randomly selected for you so You can you can use a random password and then you go check the config file to see what it is But to save time I just set my my password there so once your config files are ready to go you Bootstrap ansible that this is another script that installs ansible and a bunch of extras and wrappers to make it simple to use Just runs it doesn't need any configuration and Then the most important step is to run the ansible playbooks So this is the last thing you do and this using the config files it goes and installs open stack On On a public cloud server this takes about 45 to one hour for five minutes to one hour and and you have a cloud that's ready to go With all the major services installed So I'd say one hour one hour is a fair thing to say So Suppose many of you are not familiar with LXC So this is a simple list of the most useful commands the ones that you're using all the time If you want to see the list of containers LXC LS Dash F it's like it and S the dash L for LS it shows You know that the one line a one container per line nicely formatted If you want to log in to a container you use LXC attach and provide the container name Which you can get from the LS listing The containers are all started so you normally don't start a container But that that's the command LXC start that you use if for any reason a container has been stopped So you can restart it And the D by the way that the dash D is to start it in demon mode So it's in the background it runs in the background. You still you stay in your host after you start the container And then a couple options that I'm gonna show how to use them or a use of for them later Is how to stop and destroy a container? It sounds horrible, but it's actually quite fun So I'm gonna show you that So you you if you need to destroy a container you stop at first and then you destroy it And in both cases you provide a container name So, okay one more slide before the demo. Sorry, sorry about this So this is my workflow. It's not the only workflow by the way You know, you there's other people my colleagues that do it differently So if you look around you're gonna find, you know, it's not the the only way to do this But what I found to be the most useful and convenient for me is I start by deploying one of these things So I get it in a public cloud service Once it's running say I as I typically do I want to work on heat engine for example So I do LXC attached to that container So now I'm logged in that container that container is running a stock heat engine that Ansible installed for me Right, but that's not really good for me. I just want to install my development version So what I do is I stop it and this is this is like a mini Ubuntu So I say stop heat engine. I'm sorry service heat engine stop, right? It's an Ubuntu VM. It feels like one so you stop it Then I just get my version which could be from GitHub or maybe I have my fork doesn't matter You just install your version In case there are any differences between the version that Ansible installed versus your development version You can refresh your dependencies. You can update your database If you need to change the config files the config files are there because Ansible installed a Running version of the service. So in most cases you you're ready to go You don't need to touch the config files, but if you if you're working on something that requires Changes to the config file. You can make them and then after that you just run it You run it manually and you know logs to go out to the console. You just running like a Python application and That that works really well for me in a way it matches the way things run in in Dev stack Where or you have on that screen thing that you get, you know all those sessions The difference that you only get the one that matters the one that you're working on You don't have all the other stuff, right? So you you get your single session and then everything else is installed for production unlike Dev stack where everything is installed As if you were developing So So I'm going to apologize the so I have a recorded demo and it's tiny characters But if you go to that that URL I recorded this the same session the session that you're going to see here It's recorded and you can view it now along with me or you can do it later It's it's out there. If you're not familiar with Askinema. This is like a YouTube for console sessions And it allows you to copy paste so it's really text that you see in your browser. So It's actually very useful. So if you want to play with this later and see what I did it's all recorded So let's see how that works. So this is what I have. I'm sorry about the size Sure, yes Okay, so let's see if my magic button works So first thing I did Show you the container list so LX CLS dash fancy and That's that's the list of containers on on this installation. So if you do this you're gonna get a list exactly like this one or very close The container names are gonna be different because they're randomly generated, but other than that you're gonna get this So these are all containers are running in this single host. This is a single host It's hosted on a rock space public cloud VM Of the specs that I mentioned before 16 gigabyte of RAM 80 gig of this So you're gonna notice that all the usual suspects are there you can see Nova keystone glands all those But notice that some of them are redundant, right? We have how many keystones? We have two keystones. We have three rabbit mqs. So HAP Is it three? Okay So you can see three galeras. So we have a redundant database here. We have a galera cluster It's all running on on on my little node, right? And it's basically we installed it as a production service, right? It's a production deployment the production cloud, but it's running in a single node So let's continue What did I do next? Yes Very important. You enter is fully supported So here's Alexi attach. So So I picked a container. So I'm gonna stop right there and tell you about this. So this is the utility container So this is the only if you if you produced that list very carefully When you got to the last one with utility containers, so what's this, right? So utility container is a container that we install where all the command line clients are set up for you So when you need to I mean the same way you go to horizon, which is also installed you go to horizon to do your GUI You know access to open stack you come to the utility container to do your command line Access to the cloud. So here I logged into that container with Alexi attach a Couple more enters. Yes, that still works So I'm going to source my credentials which and she will put them there for me. So just source open RC that's ready to go and I'm gonna show you some commands which I I tested a little bit of everything So the open stack command line is installed. You can get use a list Then heat stack list, which is gonna be empty because this is a fresh install And I did one more glance Which is also empty. So this is a very new very new installation So I think clear works. I divided the screen now I left so I I split the screen now. So I left the utility container. It's at the top because I'm gonna use it again in a little bit and Here I'm gonna show you How this host is structured. So the bottom one. I'm not in the utility container. I'm in the host So it's very useful that all the file system of the file systems of all the containers are Files in the host. So you don't have to Log in to a container to see the files that the container sees as its root file system So you can see here that there's a directory for each container if you go inside that directory There's a root FS sub directory inside that you have the root file system of the container So for example, if you are a say you are a pie charm person You like, you know full-blown ID ease You can install it in your host in a container it will be in practical right so you install it in your host and you you can share the source code between the container and Your host by going to this the corresponding directory Same thing happens with logs I'm gonna show you right there that the logs are also all hosted in the in the main host and they're mounted on each Each container mounts its own logs So this is opposite slash log same thing one directory per container and here I'm doing a little demo of The host of the logs. I'm gonna tell glance API logs So I just go to the glance There you go glance container glance API.log That's log file Now you can see me going to the utility container on the top half and as a running command I can see it in a log. So It's kind of a dyi of a dev stack But you customize it to the stuff that matters to you. You don't have to get the whole thing configured this way So Okay, what's next? I'm gonna show you how to set up this box to develop on heat, which is what I do most of the time as an example so I'm gonna start by logging in to the heat engine container Alexi attach the shen container name Then as I said before I'm gonna stop heat engine because that's a service that I don't care about I'm gonna install my own heat engine So stop it You probably don't believe me. So I'm gonna go to the top and Show you that heat is dead effectively dead. I mean, why would you believe me, right? You don't know me But there you go instead see So now I'm going to clone heat from github. So there's a virtual environment Installed that I'm going to activate. That's where all the Sorry, all the dependencies are In all containers the virtual environments are in slash open stack slash VMs So I activate that that one now here comes Something that I'm gonna stop right there. I need to explain this Because open stack Ansible is a production deployment. It does not go out to the globally available Pipe I to get packages open stack Ansible sets up its own package repository so that every time you Install or fix an installation or change it You have consistency all the same packages are installed. So that there's a What I think there's really two I mean if you go back and review the list of containers There are two containers that are repositories for Python packages So when you're working with a developer version You may want to go out to the you know the big pipe by the the globally available one because you know as the project Changing the there might be requirements that are outside of what was generated at the time you installed this deployment So what I'm going to do is edit the pip configuration now, which is in all containers have it so It's configured to not go out to the cheese shop And I'm gonna tell it to do go out to the cheese shop by setting that thing from through to false This is in preparation for installing dependencies, which is what I'm gonna do next So now that I enabled access to pi pi and if you don't do this It doesn't matter right if there's a package that it's not there You're gonna get a failure and then you oh, okay. I need to do this you can do it later So now refresh all the requirements in that In that container Which usually When you do it like I did it where I installed the whole thing and then this doesn't happen to be much different But there's one that there's one update there So pretty pretty much everything else was already installed so this is kind of boring I think there's just one that's gonna be a little bit slow to install the XML one that's super slow Yeah So I'm just gonna See if I can skip that save some time. There you go. Yes Yes, it calculates, you know the consistent said Sorry It creates a consistent set of requirements for you know for all the projects that you install. Yes, that's correct But now since I'm installing a developer version, I may need to go outside of that Which for production you don't do but for this it's okay. Okay, so I need to Set to sync the database in case there are new migrations So this this one exercise I was lucky There were no migrations But you don't know you have to try it and I'm gonna talk a little bit about migrations later They're kind of a headache for some for a reason I'll tell you about it so Database was synced and now I can Install my version of heat in the virtual environment for development So I I do Python setup to apply develop For those that don't know This installs the package but doesn't move it to the virtual environment it installs it in place so you can edit the files and The changes are automatically picked up because Python put a link the same link in the virtual environment that points to the source files So it's very good very convenient Sorry, I'm not sure what's going on with my throat today. I apologize So now all that's left is to run it. You can just go ahead and run it it's it's a fully installed development package and Something that I particularly like is that when you run service this way The logs go out to To the console and of course you still don't believe me, so I'm gonna go to the utility container to show you that it's It's working again. There you go So and one more just for fun So it's really running it's running out of my source code so I can start working on it and each time I can control see this Make some changes run it again, and that's how I work. That's how I do all my contributions to heat so I'm gonna stop this and If you remember one of the things I mentioned I Promote the at the top the utility container so I Mentioned that with Dev stack. It's an all or nothing. There's no way to fix things so I'm gonna show you what happens if you mess up, right? I'm very good at this by the way happen many times So the container that you're working on starts working erratically You want to fix it so you can sit down and Debug it or you can just trash the container until ansi will make me a new one That that's the very nice thing that Dev stack. I mean it will take in taking years to support something like this I think so This is when I stop a container and destroy a container which is very exciting to do. I always love to do this Feels like you're doing something forbidden, but you know in the end It's no big deal. You get used to it. It's quite fun so There you go So Lexi stop to stop it you have to stop it destroy doesn't destroy a running container So you have to stop it first So, okay. There you go something. I didn't know thank you and then you just run the answer will playbooks and Because of sorry Because of the way answer will works answer will detect that the container is missing and It will Just make a new one as if nothing happened now If you remember, I showed you that there's a run playbooks script So you can run that and that's gonna go through every little thing you have to do to install open stack You know, it's gonna say oh, this is done. Okay, let's move to the next one This is done to it. It'll keep going and it'll take a while just to realize that most of the things are still there So the option that I used here for some reason that run playbook script doesn't have doesn't take command line arguments But ansible does and with ansible you can limit the scope of what you want to do So here I I used ansible directly instead of through that wrapper script And I'm gonna tell it that only perform the tasks that are related to hit engine Which is quite faster. So opensack ansible and Opensack ansible basically imports all the config files It's a little wrapper around the original ansible and I tell it that shell hit engine So now it's gonna start finding all the things and it's gonna realize that the containers missing. It'll make a new one and if you Let's look at this. So this is 1150 ish So 1150 so I'm gonna first forward this so It's it's about two and a half minutes it took for this and you can verify it in the In the screencasts if you want if you don't believe me. It's almost done there There you go finished it Created a container and now I'm back to normal so Think we're running a little bit short on time. So I'm gonna speed this up basically what I did here is Just connect again and make sure that everything works Okay So I mentioned this I'm gonna go really fast all the locations of things If you you can get the slides for reference I Didn't show you h.a. Proxy, but the h.a. Proxy is installing the host so you can You can modify the configuration there, especially if you're working on one of the Services are redundant. Usually you disable all the ones you you don't work on and keep them as backup I typically do that if I screw up instead of going you know to destroy the container I have in the case of keystone. I have to I can screw up one and then I can move to the next so I mentioned the cheap shop, so I don't think I need to repeat that I showed you in the screencast Debuggers if you work with command line debuggers, they they work great on the container So my favorite one is pu db, but pdb works, too pu db is it's a It's a GUI debugger, but uses terminal so it's text mode GUI and that that was really great I use that all the time If you want to do remote debugging with pie charm I told you how to get to the sources from the host so you can install it in the host do remote thing through SSH That all works, too If if you're doing Apache so keystone and horizon they work on Apache So it takes a little bit It takes a trick to make it work with a development version the home director needs to be set So you have to remember to do that Database migrations if you get a migration in your development version, and then you need to go back your screw basically It's not this project's fault. It happens with Dev stack or with everything else And I I mean I don't understand why people thought it was a good idea to not support downgrades It's a horrible idea. I think So basically the nice thing about Ansible is that the option is you can delete the database and then run Ansible again And it will make a brand new database for you Empty yes, yeah Of course, that's why I'm so pissed. I I mean I want downgrades. I always create my downgrades on my projects So you can back it up, of course, it's a pain in the butt Galera is tricky to restart so nothing to do with this project. That's something you need to be aware of I work on it Public cloud server. So for me it never goes down But if you're using a VM every time you bring it back, you have to for the first node You have to take its document and you have to run it in a little different way Just so that the cluster starts So the pros of this No dependency headaches. So dependency problems is the thing of the past for me. It's very easy to modify things So I create one one of these and it lasts for a very very long time I can keep it going. I don't have to go make a new one very often like I used to with Dev stack It's really closer to the real thing because it's a production deployment Except for where you put your development version The cons is that this is a new project. It's young. It only supports the core services at this time So I mean any of you that if you work on a project that's not in that list and You want to write answerable playbooks for it? We will welcome you with open arms. So keep the contributions coming and Then for neutron guys any neutron developers here No, yes, okay, so It the way we deploy neutron is very opinionated. We use Linux bridge. So I suspect it's not very useful for development You have to do a lot of manual work to configure Whatever you want if it's not this so I don't see it as a Usable for and so usable for for neutron So as I said, we we would like to take contributions we We want to grow this project. We are an open stack Ansible on free note And that's what I have There's a blog post about this same talk, but it Block post version of this talk if you want to read it later that's the URL and That's it Question I supposed to give you Mike. I think It's probably off it's on now about Opens stack Ansible by itself is is this just a set of playbooks to set up a development environment Or could it be used for setting a production environment as well? You mean the open stack Ansible command that I used no that that's using production Basically, it's a wrapper around Ansible that just imports all the YAML files that that you have in the ETC or open stack deploy This is the place where the config files are so you don't have to I mean you can run Ansible Normally, but you have to include all these variable files by hand So this script doesn't for you. It's very simple actually When it's contributed I guess I don't know But we would like to we would really like to support red hat Would you be able to say how it differs from cola are they similar? Cola it's based on docker. I'm not I'm not familiar at all with cola. I know it's it's docker based so We find LXC much more flexible LXC basically you create VMs they feel and act like a VM So you're not restricted to deploy single applications per container You really get a an Ubuntu machine in each container. So we find it more flexible, but I'm not familiar with cola. So So this is a single node for development. This is the The playbook supports. Yeah up to thousands if you want. Yes, it's a production deployment We deploy for our customers to one more. I'm sorry. It's off. Here is the question when I use DevStack I always need for my purpose to disable force config drive on lower conf and This is a headache for me how with your project. I can atomize it or minimize my Specific actions. I don't remember specifically if we have a configuration option for config drive Okay, that guy says yes, and he knows yes, you can disable so after you Bootstrap ansible that first step you can go in the config files disable it and then when when you run the playbooks It'll start disabled. Yes So these two guys are experts by the way, I'm just a developer, but they know so One more you mentioned that you contribute back to heat. Yes, and the gates in DevStack At the gate So I can reform. Oh, oh you mean that the gate is set. Yeah So the problem is that when you do this stuff as you do you are running not the latest masters of all the other stuff and So there are Maybe I mean it's still useful, but probably something to be aware of that. That's a good point Gate might not accept your change because I mean something changed. Yes. So to be honest, I Guess it depends the kind of work that you do. Yes For the things that I've done that that was not a problem for me I run, you know the unit tests in the container and You know and then you know pray and send it Yeah, it's been good to me. Yeah, so probably when you Making a new resources or stuff like that on heat then when some API of Nova change or Maybe I have seen their change you move to see the well, so the config files in the In the ETC directory that I showed you can change what version of any project you want to install and you can upgrade any project So you don't have to you know make a new one You don't have to operate all the ones Change it run it and yeah And it will be much faster than upgrading everything. It's like minutes. Yeah, like I showed and two and a half minutes Yeah, so it's you can still know make it work. I think thanks. Yep. Yeah, we have to go Thank you Thank you