 All right. Good afternoon. Thank you for coming to the session. Thank you for voting for the session. My name is Keith tensor. I'm a solutions architect at Red Hat, and I'm mostly working with our enterprise customers in Germany So I'm going to talk today about operational automation what it means from a solutions perspective instead of from a product perspective and Really glad to see it at this summit. There's a lot more sessions that are talking about solutions I think it shows a lot about the maturity where we are So I do a lot of blogging some of you may have come across my blog potentially so What I'm going to talk about today is the concepts behind Automation that we have that I've worked with our customers on and The technologies behind this and I'm going to show you actually how this works and do a demo Hopefully that works the recipe for the demo if you guys want to do this at home Hopefully you do it at home before in a production environment It's on my blog at Keith tensor.com. So I show you guys real quick if anybody wants to do everything. I show you today. It's it's all documented So before we get going a couple key takeaways like so if you leave with any anything I would like that to be That we come away with an understanding that operational automation really means two things it means provisioning So a provisioning platform and what that automation looks like as well as an application deployment Until we have an application deployment. We don't really have any business value. So just doing it like Amazon isn't really that exciting So the second thing is we really need to decouple The provisioning automation from our application deployment. The reason is is while open stack is wonderful and awesome there are also other provisioning platforms out there that our customers use and The customers from a business perspective. They don't really care about it and Getting systems they care about their applications. That's their business value So they are willing to invest a lot in those blueprints for how to deploy those applications And they're going to want the return on investment meaning they're going to want to be able to Deploy those applications not maybe just on open stack but on Amazon or other platforms that they have and in the future Who knows what other platforms might be available? For the context of this discussion This is about open stack. So open stack heat and ansible are the two technologies we'll be using and talking about well and So before we get into that I would like to talk about a concept of that's going on with all of my customers right now And this is the idea of industrialization of IT. So if we look at the auto industry, I think there's a lot We can learn actually they've gone through this process already that we're going through today So I don't know if you have all seen the episode of the Simpsons or watch the Simpsons Probably a lot of you have where Homer gets to build his own car It's one of the one of the most funniest ones I think one of my favorites and Essentially Homer, you know gets appointed to build a common car for everybody and so he goes off and he builds this car that this perfect car for Homer Simpson, right and Unfortunately Homer Simpson is the only person that actually would want to drive this car and it cost also $80,000 so This project obviously failed miserably and there was zero reusability So this probably sounds familiar about how we started off in IT and how we build systems build systems and how we some Of us are still building systems today So, you know, I've gone to customers All over the place and I see over and over again Areas where you see, you know post-gres 100 databases and 50 permutations of those deployments, right? So we need to get and I think we've gone come a long way there But we're still not everybody's there and still not all all the applications are there to think about t-shirt sizes and standardization It comes down to the 80-20 rule. It's not about the best thing for each individual application It's about what's good enough, right? So we can invest more time in actually innovating and not building stuff So auto industry started there as well as IT then things evolved in 1914 this concept of an assembly line came we all know this Ford built basically created blueprints for how to build a model T car with a couple of permutations Ran that through an assembly line and lots of people were you know screwing things on and changing stuff and doing things And if you think about how we do it today, that's a lot of very similar, right? I mean we we we build systems that it goes, you know network guy does something It's the word guy does something an application guys come in and there's there's there's a blueprint for doing things But the level of automation is still fairly low We we don't have let's click a button and not just deploy You know the shell of the car here where then we have to do a lot of work to to get it where we want But let's deploy an entire application That's running out of the box, right? so in This assembly line approach, you know, I think it has a certain limitation, right? You can optimize things to a certain point, but then it will not scale and What really the auto industry has done and what really we're after with this industrialization concept is moving our Bulk of our workforce and our efforts in IT from running the business keeping the lights on to Innovation and innovating and if you look in the picture over on the right if you can see that's how future cars are being built today Nanobots automated. There's very few people There's actually one guy sitting up on the left and he's probably just admiring the wonder of this automation And so all of the people today that are building cars a vast majority of them are working on the innovation side and not actually producing and building them and One interesting stat I looked at from the auto industry is in 1914 Ford which totally smoked everyone as far as how they could build and how fast they could build cars They were able to do five cars per employee that they had that year today. They do 500 That is because of automation if you think about it That's where we're trying to get to with with with IT. So at this point I just wanted to kind of Conceptualize things a little bit to understand what what organizations are really what they're really driving and why automation is so important So the next piece Where I see IT right now is we're kind of you know, obviously there's applications That still are you know the Homer Simpsons There's assembly line and there's some that are you know in this industrialization phase So I think it's a mix and I think if you look at most companies you can't just cut over You know, it's you you build new applications and you do things using new ways and you try to move the old application slowly To these new processes or migrate them. So we're kind of in the middle on the way in this journey So if you talk about the technologies, obviously for doing open stack we want to use open stack heat We don't want to circumvent heat heat is the brains of open stack without heat We have a bunch of Independent API endpoints that are not connected together. There's no value. I can I Get value first when I orchestrate everything when I can create a stack. So heat basically conceptualizes a Deployment as a stack because it involves Obviously an application needs more than just a VM. It needs many VMs It needs storage it needs networking it needs all kinds of things and services And so we can build that very nicely in heat and and do that and we've been doing that and that's sort of Where we're at the point of today, you know, you can either you know, go to Amazon Or you know, go the open stack route and do Amazon in house yourself for a lot cheaper and more more cost effective And and I think more value but Certainly some people have taken heat and said okay. Well, let's use heat to do everything you can orchestrate applications You can do you can do everything in heat But again, I really think there's a value in decoupling the application deployment from the provisioning Deployment and so that's really what this is about What's missing here is language that we can use that is up that can can can go across many Provisioning platforms if need be a simple language that everyone can understand in the company Everyone can contribute so that means people that aren't necessarily techies can read the blueprints for an application and actually understand what's going on there and so That that's where I really see Ansible coming into play and really fitting perfectly with heat together both by the way also use the same language So they're both YAML based which is makes it easy to read So if we look at ansible In ansible basically is the concept of ansible is automation for everyone, right? It started ansible started sort of where you know, we started with CF engine and we have puppet and chef and While those are great automation frameworks one of the things everybody always had had Sort of was disgruntled about is that only a select few people really understood these things and were able to use them And if we think about the industrialization little aspect How of companies having the more people that can be involved in creating these blueprints in Understanding them and sharing the ideas the more value we can we can generate and that's why ansible for me and for a lot of my customers is so powerful and Really really taking over so ansible. I'm not going to go into deep details on ansible But just to give you an idea. There's basically two components. Most of you probably have seen or are familiar with ansible core Which is an open-source project of course and many people are using and so ansible core is basically a runtime for ansible runtime environment it consists of plugins and modules to integrate you know with open stack to you know orchestrate things with networking equipment whatever you can you can conceivably want to automate there's plugins for that in a community for for that and It basically every what you automate at the end of the day is called a playbook. So that's your basically instruction set or blueprint for what you want to achieve and Then sort of what's missing from that picture from core is tower So what ansible tower provides is an api endpoint for ansible? It provides management an ability to centralize your blueprints your playbooks in a location Supporting you know some concepts of role-based access if we think about building cars or building, you know complex Applications there's lots of teams involved and lots of different pieces that teams want to do or want to be Want to control want to drive and it's about bringing those all together So you really need a centralized platform for doing that and that's basically what ansible tower Brings brings to the table as far as that goes So there's many ways to integrate heat and ansible. I'm going to talk about two Again, I'm talking about the premise of we want to from an open stack perspective use heat We don't want to circumvent heat And I think there's a lot of value in using heat heat is grows with with open stack It's it's engineered with open stack and really understands the open stack landscape better than anything else is going to So we want to take advantage of that leverage that so there's two ways basically There's the approach without tower so using ansible core and then there's approach with tower So the ansible the ansible core approach using heat you can In heat inject essentially ansible playbooks You can basically put your playbooks in in heat templates and derive that from there so you end up with still a decoupling in that you have a provisioning Instruction set and you have a application deployment instruction set So you can pull that out and and say you want to roll roll your application on Amazon or something You can you can do that if you structure things that way However, it is a little bit tight too tightly coupled personally for me So I think you know this this ends up if you if you do things in heat here You're limiting who who can really take part in the automation. We all know in complex There's lots of teams and complex applications and deployments that and they really the experts that understand the different components Should be able to actually should be best able to say how things are going to be deployed So I feel that that's a little bit too Too tightly coupled so what I really what I really like is the ability to do this with ansible tower and ansible tower provides basically you have a playbook and ansible and you can create a API endpoint for each playbook so anything you automate picture has an API endpoint And so now that I have an API endpoint right I can decouple these things very nicely And that's essentially what we want to do in cloud. We want everything to be loosely coupled That's the idea. We don't want to tightly mesh things together That's what we learned in the past and how to do things, you know How we shouldn't do things so ansible provides that ansible tower provides that really provisioning piece So that's the main thing that I the biggest thing I think is really great about tower It also provides a central management as mentioned and other capabilities around allowing multiple teams and users to work So what I'm going to show to you today is essentially a demo Using this second option with with ansible tower So if we look at basically the workflow, I'm going to show How this works is you start off you start off in heat so you do it you deploy your heat stack The heat stack will then configure the infrastructure that you need to support your application It then triggers ansible tower through provisioning callback Ansible tower then will do a se a software content management update So if you're using github or whatever you're using SVN Optionally obviously, but you probably want to pull the latest version of your your instruction set or your blueprint So to do that if we think about open stack, we don't know what IPs and host names We're going to have right that's the whole point of open stack is to be dynamic and flexible So we need something that it discovers can discover dynamically instances that are starting up So ansible tower can actually talk to them so we have that and Then finally when when it when the system's discovered ansible tower can execute the playbook So basically deploy the application and Finally, we don't want to just you know to have the heat stack complete and then ansible do its thing We want actually the heat stack to wait For ansible to complete and then ansible to tell heat Hey, the job is actually done because he doesn't know anything about the application And that's the beauty of this is that we can keep our heat templates and our heat stacks Really simplified and really reusable and the same thing on the ansible side. That's that's really the power of this this cut this concept So for the demo, I'm going to basically show a WordPress Example that's what that's what I have. That's what I prepared And so we're going to follow these steps and we're going to go through basically the workflow that I just Illustrated to you to you all so Let's see here This could go terribly wrong. This is so give you an idea. I'm gonna exit out of the presentation at this point This is all running on my laptop my laptop has is is a data center in this case with 12 gigabytes of memory so rather rather minimal and You can see here. This is I'm running running redhead enterprise Linux So I have KVM basically a hypervisor built into my OS It's the nice thing about using Linux in this case instead of instead of a MacBook or something But I have a VM running called OSP eight. So it's basically red hats open stack eight Which is so that's running and in there I have if we go to look at the open stack environment. Oops bring this over here and look at the instances in there I have a Tower dot lab calm so basically I'm using the admin tenant here So I haven't done any user tenants so you wouldn't quite do that But you get the idea you have a tenant and the idea here is in each tenant You can have a tower system for deploying applications and blueprints and all that stuff for for each tenant You can also if you want to bring tower outside of the tenant and have it service multiple tenants in this case I've brought it inside the tenant and you can see it has a public and a private IP So ansible tower needs to obviously talk to Systems that open stack boots using the private the private IP the private network because that's that's the only thing It's known public is for Accessing outside of the open stack environment in this case my laptop So for me to access the application or for me to access, you know ansible tower itself. I need a public IP So that's the I that's the basically the setup I also have a another WordPress 01 instance that I already deployed in case my demo goes terribly wrong And I can then at least show something So let's let's let's hope it let's hope it works So I'm gonna basically start a heat heat stack deployment So I I'm gonna basically my my heat stack. I'll go through the heat stack briefly We'll see how much time we have I can go into more details and show you guys more stuff. We'll see how it goes but So basically my heat stack is gonna take a name I'm gonna call this WordPress 03 actually and it takes a template So it's called central as WordPress dash heat Then I have a name for my instance So I've parameterized my my heat configuration So I'm gonna make that WordPress 03 and then I'm giving it the private IP of the tower system So I could obviously hard code this in the template, but it's better to parameterize those things So I'm basically gonna gonna kick this off And what we should see Is that the if we do a heat stack list we should see that this thing is running It's create in progress. And so if we go to our orchestration here and our stacks We should be able to see that it's yep create in progress And if we go over to the resource overview will notice Over here or over there. You see this blinking our glass, right? This is basically I've instructed heat to to whatever it does It just waits until somebody else tells it that it's done and remember what going back to my workflow We want at the end where Ansible is done to tell heat or complete so the the stack is running And so if we go and we look and see what it's done in this case I'm just deploying one instance because I don't really have enough memory. So it's an all-in-one WordPress So I've deployed here. It's deployed back basically the instance I can follow the log messages here and see what's going on. It's basically booting up And you see basically it made this curl call to already to to heat I'll cover or to Ansible. So I'll cover that what it's basically happening It's booted up the instance and I'm going to log into Ansible tower and we should see an Ansible tower now that actually something's occurring So it's it's doing a deployment. So if you look at kind of the jobs here And we can see right now if we look at the time it did an inventory sync. So remember we wanted to first Have Ansible download latest versions of our blueprints that may exist for the application deployment in this case WordPress And then we wanted to do we wanted to do an inventory sync as well So remember we Ansible doesn't know what the instance is what the IP is it has to gather all that information So that has to be part of the process and then finally it's going to run the playbook. So if we look and see I'll just show in the inventories in In Ansible there's basically have an inventory here for cloud. This is just basically a grouping You can see right here. It it detected It ran and found WordPress O3 and if I click on that it's gonna have the IP information It's gonna have the tenant information all kinds of information from OpenStack by the way I can parameterize I can use all of those parameters in my Ansible tower playbooks in this case I'm doing something very simple with WordPress But let's look at let's look at the job real quick and see what's going on So if I bring up this job, this is actually my job is running you can see on the on the right side It's it's it's basically the the sort of the standard out what's happening. It's deploying WordPress It's deploying a Maria DB. So WordPress needs a database. It's got application this WordPress example uses engine X So it needs obviously a web server, you know, your standard kind of three-tier architecture and then you notice here on on on the left side extra variables On the Lex left side, hopefully you can see that so what he'd actually did when we told it to wait We created a wait condition and it actually created a restful API endpoint in a token that we can use to tell heat when we're done And I've passed this in to Ansible and Ansible tower here So basically Ansible tower can when it's done trigger heat again to say hey, it's actually done We've finished our complete software deployment And so we can follow in tower we can follow all of these things While this is running what I'd like to do is show you switch over to actually the heat stack template and kind of go through that to give You an idea on what I've done So I'm just going to open up the heat the template in in a Vi session here Okay, so this is standard I'm not sure all of you have have worked with heat templates So I'll try to explain some of the basic stuff But basically in the beginning you want to parameterize you saw I passed in an IP address and I passed in some parameters So basically this is the parameterization aspects of heat. So we define our parameters I've hard-coded some things like well, I haven't hard-coded by given a default string So I didn't need to put the network ID and those things I I basically use the default and he's gonna use the default if you don't pass anything in And then you get into the resources right so the first thing we see here on the resources is the weight condition That's actually how you set up the weight condition in heat to actually do this And that's actually what you want to use a weight condition for is if you're integrating with with other things like like like ansible in this case So we create a weight condition. We create a web server So we we tell Nova we want a web server We basically get our parameters and the only thing it does is it and I've just used curl here This is a very basic. So, you know, I'm a I'm a solution architect at Red Hat I'm not a developer or product guy So I've just done something very simple here to give an idea you could do something more sophisticated But basically using a curl call is how I'm triggering ansible tower and what you can see is I'm it's basically Jason So I'm giving it the extra vars the heat endpoint and I'm giving it a weight endpoint is the basically the variable that Heat heat will assign basically the URL to that variable and as well as the token and then basically I just call The the tower IP job templates and the callback URL and I passed in obviously for the callback a host can host key That identifies me So That's some other things that's going on here. We need a port. So we need we need some stuff from neutron We need an IP address for a system. Otherwise nothing's gonna work So we also need a floating IP This is all basic heat stuff that most of you probably already seen and then basically there's the outputs So in the outputs is where we want to communicate from heat to our end user what we you know information So in this case, I'm communicating out, you know, the the the endpoints the tokens things like that So this is where sort of heat heat guys guys developing heat stacks We'll we'll want to create and and produce outputs here that that might be interesting for their users so if we go now is there if we go now from from Heat and shift over to ansible. So that's basically all heat is right all we're doing in heat. It's very simple just provisioning Infrastructure and calling ansible. So give you an idea a little bit more of ansible tower So if I move over to tower here Basically the concepts of tower is you have essentially projects. These are basically github URLs where your blueprints are located So in this case, I have an examples and if we open that up This is basically where the URL is for it's my on my github Where my where my heat template or sorry my my playbook is to deploy WordPress So again, the same playbook can be used not just an open stack on Amazon on anything the mware whatever, you know rev KVM Whatever you guys have bare metal Anything so that's the concept there you see here. There's the update options underneath the source URL So I'm telling it so when a job is triggered for this For this project, it's going to do a clean which means it's going to basically wipe the the local contents And it's going to pull pull things fresh so I could say no I never want to do that I only want I you know, I I want to manually, you know ensure I pull versions of the blueprint when I've tested them or something like this so you have options here to do that The next thing which I I basically showed already was the the inventories so in order to do something with open stack we need to know about What what we're working with what IP's what host names all those things and I already showed you guys I created a group so an ansible you these groups. That's just called cloud It could be called whatever it wants, but basically these groups are just groupings of hosts So we can make the groups however our imagination or whatever makes sense for the organization It could be by application. It could be by you know Geographical stuff location and there's all kinds of there's no sort of right or wrong way This is where we really just need to understand and think of a concept that works best For what we're trying to achieve And then there's basically the job template. So we have a project where our blueprints are we have an inventory that defines the the machines that we're working on and then we have a job template and the job template basically brings everything together so I have one called WordPress here and Essentially here we define What we're doing so once this updates you you'll see here It I give it a name. I give it an inventory. I give it a project I get from the project. I can select then my blueprints my playbooks So here I've got this the blueprints WordPress engine X real seven, but I've got lots of other ones I can select so a job template basically associates to a something you're you're you're going to automate and that is exactly where This provisioning callbacks comes into play so you can kind of see on the on the left side There's provisioning callback URL and host config. This is basically where you can enable that so you can decide okay should my My my ansible playbook should it be exposed? Externally via API in this case. We've obviously chosen to done that to do that you can see here extra variables You can you can put stuff in there. You can you can you can define your own in this case We're doing them dynamically. We're getting them from from open stack so you've you've you've seen that where they show up and then finally there's there's the jobs and Basically, if we look at our job now, it's completed So we can go through here on the right side and we can see all the steps that it's done This is basically what you define in your playbook you expose this you decide what the steps are what you want to communicate what's going on how it's going going to work and Ansible has a really cool concept of roles So you can have basically inside of a playbook different roles If we look inside of this playbook actually there's several roles like for example Maria DB as a role The idea of roles is to create reusability So you may have many applications that use Maria DB, right? And you can basically use the same role that someone's already defined in in other playbooks and Reused sort of what someone else is built and this is kind of goes back to the idea of why ansible Is so awesome is that it's so simple and easy that it allows so many people to To basically come up with ideas share ideas exchange things put things together in different ways. It's like, you know Building blocks with with Legos, you know, and what what can we build and how fast can we build it? And that's something you just don't see with the other automation frameworks you're gonna be Either you love it and you're gonna be the few guys at the company that understand it or you know You're gonna be pulling your hair out and looking for other solutions. So Basically if we go through here to the bottom, we get a recap of what we did So and ansible basically highlights. It's kind of yellow orange. So basically that means that color means something changed So ansible that the playbook actually changed something The green color means nothing changed So every every something whatever it tried to do was already there So it didn't do anything and then there's a red color which thankfully we didn't see that would mean of course we have an error You can also go through here Each individual tasks. These are all the tasks now that are exposed. You can go in there You can look and see what exactly happened. What did it do? What commands did it run everything else? And at the end of the day, basically I've done one click, right? I just I just Started heat stack. I haven't done anything else, but but show you guys stuff So now if I go back to my my my instance over here I can look at the IP. So I've got this IP of 168 so I can basically Should be able to hit that with a browser 22.168 and it should load my WordPress application, right? So there's my WordPress application and so this is the concept or the industrialization, right? Now I've automated everything right and I've got different reasonable components other people can feed into this and I don't have to sit Here and take a request and build something or talk to this person or talk to that person or anything else My business gets a solid process that deploys their applications the same way Many teams can work together in different levels of the provisioning infrastructure level and open stack At the blueprint application deployment level in Ansible and this is really the best of both worlds It allows the most flexibility and the most speed really to to to to do things So let me With that That's basically all I had to show for the session. So I think we have another Seven minutes or so if there's if there's questions or or or anything I could open it up any Yep, could you grab they think there's a microphone. Yeah, otherwise Yeah regarding your Placement of the work of the Ansible tower instance How will it work in the multi-tenant and especially in the single tenon but multiple networks? Yeah, would it require that every single VM it spawns the hidden plate having a floating AP? No, so so so basically Having the same this you need this tower instance inside each network So this is basically an implementation Discussion right there's multiple ways to do it So the way I've done it here where I'm saying okay put tower in the tenant, right? So obviously it has access to every everybody that gets provisioned on that tenant network If you want to make that same tower that's on say tenant a Available to also do stuff on tenant B Then you have to basically allow that networking to to flow through which you can do it in an open stack With with very easily right, but that's something you would have to do and enable The other concept is you and this is just using tenant IPs, right? So we're talking about tenant IPs only we're not talking about public IPs If you wanted to you can move tower outside of open stack And use public IPs But this would of course require that then every buddy that is going interface with towers is using a public IP Which is the main reason why I've done it the way the way I have and the way my customers are doing this is Exactly this way and they're either doing it per tenant or in often They're they're having sort of a management tenant that's sort of one thing that or for a certain subset of the cloud exactly shard network and Putting one tower and then and then allowing the access through ACLs and things like that And if you're using provider networks, right? Then you don't have floating IPs anyway So then then it's basically you can do you know, it's just a networking Networking access discussion Thanks Good question any Any other questions guess how yeah, can can you Or or if you don't want to then I'll repeat your question if I can hear it. Yeah Yeah, so I mean so the question is basically around updates So to the to the playbook, especially like say I have different versions I don't want to stay with whatever version of Maria. I have forever I want to be able to roll out new versions or there could be other components That that need to get updated So this is really the the nice aspect of this kind of design or idea is that you've segregated the provisioning from the application deployment, right? So you can have those those people that are you know responsible in your operations team for the application and testing the versions and all of this stuff They can make it available. They can basically have the only ones that have access for example to check in To to your blueprints and things like that. You can do different processes So you could have an automatic process which I have here. I just download the latest version So this means from an open stack perspective Somebody gives me a URL so they give me a URL to what's the what's the blueprint for my deployment? I consume that in open stack and have a standard process I don't need to care about what that application does they're responsible then for you know what versions how you know when they upgrade pieces and things like that Then since it's a heat stack, you know, what do we do with with open stack or the idea with cloud is you know not to be You know Let's tinker around with a running system and constantly editing around that we throw it away and create a new a new stack That's built off the the appropriate version You could certainly do an upgrade process if you wanted to do that for you know Let's say non cloudy type applications that require this sort of constant maintenance process but Basically, you have all the options open the nice things here You see the nice thing of the segregation because if they were coupled You're provisioning logic. You're provisioning blueprint with your application blueprint. This would be much more difficult to to do Yep, I think it's there's only one yet but As you know open open stack answerable project doesn't support right heart distribution of influence and so do you think Ansible doesn't fit to deploy open stock itself Okay, so Yeah, I mean this is one of the when I talked about the different ways you can combine Ansible and heat so I'm not going to sit up here and say that this way that I showed is the only way This is the way that I've seen a lot of success with the customers that I'm working with In the field and that they really enjoy you could certainly say you know what I don't want to deal with heat at all I just want to have one blueprint language for my infrastructure and for my applications And I want to use ansible to do that and in this case you could deploy and ansible will talk to then Different services Nova and whatever else it needs to talk to and do that I Don't have the details on you know what kind of gaps there may be there I would suspect though that heat is going to be a lot more powerful in the context of open stack because It's engineered and designed with open stack and ansible then basically after the fact has a module that talks to open stack And they then go oh, there's a new feature here for this and that and they start you know building around The automation pieces for that but again that that's a decision, you know that there's no I'm not I wouldn't say that that's that's a wrong approach or A worse approach. That's also definitely an approach. I've seen in customers are taking. Okay. Thank you Any other yeah Yeah, okay, I can show you guys that real well we have 30 30 seconds. I see what I can do I didn't show this but in ansible we have essentially credentials So we basically I have two credentials here. I've one for sent us and one for osp8 So the sent us is you know when we access open stack we need a ssh key We don't have username and passwords right so ansible provides a store for these kind of credentials to do this The other thing it provides is is access to open stack itself. So when it's running these inventories It's not actually looking at at the heat template. It's going to when I configure and look at the inventory I'll just bring this up right right here. And if I edit this osp8 inventory Basically, I I decide here what my credentials are when I give it a credential. I give it a URL So basically I give it at what tenant I'm I'm going to be talking to in this case the admin tenant So it's going to get basically everything but at this case I would put that and so it's going to be talking to Nova and getting the information directly From Nova. So this is a part of tower tower actually has the ability to do dynamic discoveries and all of these things So you can do this without tower, but it involves creating lots of Inventory scripts and lots of basically you at the end of the day You need an inventory of what the hosts are that you have how you do that and create that is up to you If you're using tower though It already has the intelligence to talk to open stack and get that information and so it's going to be able to get IP Information host name information basically everything when you do when you do a Nova show on your instance all of that information is sucked into Ansible tower that Okay, and don't think anyone's coming in the room. So is there any one last question maybe or Okay, well, I thank you very much and wish you a wonderful open stack summit