 How's everybody? Good, good. Everybody's good. Excellent. I know that last night's party was awesome. I saw it from my hotel window. I didn't get to experience it personally because I had to be an adult. But thank you for being here. I know it's Wednesday is hump day, so hopefully I won't be too tough on you. I'll leave the quizzes to the last five minutes, so if you're not good with quizzes, leave before the end. I'm joking. We're going to get started here in about a minute or two, okay guys? One thing before we get started, I'm going to ask that you guys work as a team. This lab is a bit complex in the sense that you are actually going to be handed three separate open-stack cloud regions. So I'm going to ask you guys to try and pair up to at least teams of three or more so that each person can own a region and kind of maybe handle the activities needed for that particular open-stack region. So before we get started, think about teaming up three or more. It would be great. I do have limited resources, so that would be a further encouragement that I need you guys to kind of team up together a little bit, okay? So thank you. In about two minutes we'll get started. I don't see anybody moving. I need movement. I need movement. I need people together. Same row next to each other. Thank you. Yeah, I'm being a bit of a tyrant, but that's how it is. The clicker here. You go left, you go right, so it works pretty well. But I'll hand it over to you once we get there. And I'll let you of course introduce yourself because that's the only right thing to do. Okay? Chris was like, I didn't think I was going to say anything. I was like, no, you're going to say something. You want some water? No, you got some more. Okay, we're going to go ahead and get started. Again, I know it is Wednesday. It's hump day. Everybody's tired. You've been in a lot of sessions and learned a lot of stuff, so hopefully this is just one more thing you can take home with you. So today we're going to talk about how to set up active, active cloud regions. And my dear friend here in Melbourne, my name is Walter Bentley. I'm a cloud solution architect. I'll go a little bit more about me. This is my dear partner in crime here in Melbourne. And we're going to just here to present through what we mean by active, active cloud regions and show you how to go ahead and do that. Okay? Get started. I won't spend too much time on me. If you've been in any of my workshops, you know that I like to hacker people. So make sure to let me know who you are so that I can make fun of you a little bit. But I've been in IT for about 17 years. Production support is my background. I call myself a New Yexen now because I am actually a native New Yorker who just moved to Texas five months ago. So I coined the term New Yexen. Copyright it. You can't use it unless you pay me. Love cloud. Love sharing knowledge. Motorcyclists. And yeah, so that's a little bit about me and some of the companies I work for. Feel free to follow me on Twitter. And my blog, I always put out stuff about OpenStack and Ansible. So if you want to check out some stuff about that, check out my blog. There's some interesting stuff on there about that. Okay? So without further ado, I'd love to let my dear partner crime here in Melbourne introduce himself. All right. Thanks for watching. Appreciate it. So Melbourne Hillsman. Been in Rackspace for a couple of years. Been in love with OpenStack for about four years. IT Professional 47, as you can all pretty much see right there. Right. I'm just reading. I currently just moved over to OSIC OPS team. So if any of you guys are gals for that matter or in operations, I'd love to talk with you afterwards and get your card and you can get mine. You don't have to take mine. Interesting thing about me, I'm afraid of heights, but I like skydiving. I guess that's kind of a problem. Mr. Hillsman is my Twitter handle, but I'm Mr. Hillsman. Mr. Hillsman.com, Mr. Hillsman at Gmail, IRC is Mr. Hillsman. So there's Mr. Hillsman is probably me. Cool. Thank you, Melbourne. We're going to start out with some ground rules. We're all adults here, so I don't know. I figure I don't have to say this, but I'm going to say it just because it's one of those things. So I'm going to ask you to turn off your cell phones or at least put them on silent. And if I catch it rings, I'm taking it. Right now, I'm supporting the iPhone 5. It's kind of old, so if you got those six S's, make sure you put them out and make sure you leave them on ring, okay? Requirement, ask questions. You must ask questions. Do not be afraid to ask questions. Do not be afraid to raise your hand. I'm here to help you. That's our purpose is to be here to help you. So make sure you ask questions. Any side conversations, please take them outside. I don't want to hear myself talk more than anything else, so I don't want to hear everybody else. I'm just kidding. But please, if you have a side conversation, I know you guys are all IT professionals. If you've got to take care of something, just please step outside. So, as I mentioned before, I'm going to need you guys to work in groups, all right? I know it's a bit of a struggle for us, introverts to work with other people, but I need you to work with other people. Primarily because we have limited resources and I want as many people to take advantage of the lab as possible, okay? So we're going to need to work at least teams of two or three. Again, of course, the materials are at this link here. Google's URL shortness are case sensitive, so keep in note of the case. That URL will be posted again, so don't worry. You don't have to rush to copy it down now. But that's where the materials for the lab will be. Before we get started, again, can emphasize more teams of three or more. So when I go to hand out the course material, I'm going to need to at least see three people sitting together, willing to work together before I hand it out. The student ID and instructions to connect to your OpenStack regions are on the handout that I'm going to give you. And what you're getting... So each team will get three OpenStack Liberty Release Cloud regions. And it's going to be two regions called Alpha and Beta that are full OpenStack core services deployments that are going to be your separate regions. And then you'll have one third region which I call your admin region. And it's a slimmed down version of the other regions instead of running all the OpenStack services, it only will be running Horizon and Keystone. And you'll learn why you only need that for an admin region. Okay? And like I said, if you take each member of your team to be responsible for one region, it kind of cuts down the amount of work one guy has to do. There are a lot of commands you're going to have to cut and paste or retype however you wish to do. So I just want to try and divvy up that work as much as possible. Okay, guys? So without further ado, I'm going to turn it over to Melvin to get us started, okay? All right, thank you. All right, so an active-active cloud approach. All right, so here you look at the slide, right? Basically, you see there's a number of different regions. Okay, again, your region A will be your Alpha, region B is your Beta. And then you've got Keystone and Horizon which is sitting in between both of them. So basically, region is essentially, right? Multiple data centers. So you have resources that span across what these AZs are called availability zones. Okay, you can also have it. Move quicker than it was supposed to. That's all right. Sorry, see that's what happened. I made slides and I made him present them and then I booby-trapped them. Got him. That's how he does it. No. All right, so basically the whole idea essentially, right, is I'm not going to try to, I'm going to do it the way I would do it, right? Please. Essentially what you have is you have resources that are in multiple data centers across, in different regions of the world it could be, could be in the same data center for that matter, right? Basically the idea is I've got a full open-stack cloud in one place and another open-stack cloud in another place, right? And I want to be able to share, utilize both those regions together. He can't believe me for that one. There we go. All right, here we go. Okay, so again, this is the architecture, right? You've got Nova Glass Heat Neutron Sender in each region, okay? And if you've got Keystone and Horizon, it's being shared between those. It's your admin. That's basically your admin region. In this cloud workshop, it's the admin region. So you're sharing identity, right roles, projects, okay? Those are all in one place. You can utilize the command line interface or Horizon in the admin region to manage both of those region A and B together. All right, so here's the lab overview, okay? Each region's end point, so you want to inventory each region's end points and take note of the URLs, okay? So you want to make sure you're doing that. You create user service accounts in the admin region and create service on the admin region. Does that make sense? Right, everybody's okay with that? You create user service accounts in the admin region, create service on the admin region, okay? You register each region's end points to the admin region. All right? And you configure each of the region's services to authenticate against the admin region identity service instead of the local region's identity service, right? So region A and region B, right? What you want to do is you want to make sure that they're utilizing the admin region, okay, for these services. And then use your clouds. Hopefully everything goes straight and you don't have to troubleshoot anything and then you'll have fun. Otherwise, you won't have fun. All right? So please go to the URL right there and connect to the lab environment. Cool. Thank you, Melvin. I appreciate it. Yes, sir. So what we're going to do now is... Yeah, we can actually... Thank you. Pull up that URL. What I'm going to do is I actually have handouts and I'm going to give five to Mr. Melvin here and I'm going to take five, right? So again, limited resources, right? We need to at least see people with at least three or more and you will work as a team. So, for example, these three gentlemen here, I'm assuming, will be team one. All right? These are your credentials and these are the IP addresses that you'll need to connect to your region. The username and password that you'll need, okay? To connect to your region. Please only connect to your region and not anyone else's. Don't get fancy, all right? Yes. Yes, I will. So you guys are going to work together, correct? Remember, I have limited resources. So if you're in the back and you want to do the lab, you're going to have to come on down because by the time I get to you, I might not have any more pieces of paper, all right? Not to say I'm trying to do bait and switch. I'm just saying. So I see a loose grouping of people here. Do you guys want to work together? Interested? All right. If you can get together, that would be great. You guys will be team three. I'm going to put that there, okay? We're running out. You want to work together? You guys want to work together, I assume? All right. You guys will be team four. All right. I see a lot of people in this row here. You guys are, and again, you're willing to work with more than teams of three. It can be more than teams of three. I just said three is the minimum, okay? You're good? Okay. I only got one. I only got one, man. All right. So what I can do is, for anyone who did not get LA handout, come to me at the end and I will give you your own resources so that you can do your lab at home, okay? You can absolutely get that option. Just come see me at the end, all right? Okay. So you guys still want to work on it now? You want to do it at home? You work on it? All right. There you go. All right. But you can still come see me at the end, all right? So for you guys, come to me at the end. You get your own servers for a week to do this lab, all right? Okay. Just come see me, all right? So you should be pulling up this URL on your machines. There are a set of instructions at this URL on GitHub. There are steps one, two, three, three A, three B, and four. Obviously, I want you guys to start at step one. Step one, just basically getting you connected to your open stack regions. Okay. Use the IP addresses on your handout. And the credentials there. Let me know when everybody is good and going on step one, all right? I was, like, going to say something to you. All right. So everybody should be looking at the instructions that are up on the screen here right now. Let's see if I can make that a little bit bigger. There we go. That's better. Okay. So the first thing is you're going to connect to your alpha region using your credentials. You're going to execute an LXC command. And you may say to yourself, why do I have to execute an LXC command? And what the heck is LXC? So LXC is the original container format, right, way before Docker even thought about being Docker. That's part of the Mboot 2 operating system. And with OpenStack Ansible, if you're not familiar with that way of deploying OpenStack, the OpenStack services are deployed in containers called LXC containers. Yes. No, no. That's what should work. It has to work. Oh, no, no, for Horizon. Okay. So on the piece of paper that I handed out, there's two sets of credentials. There's credentials just to gain access to your OpenStack regions, and then there's credentials to gain access to Horizon. They are separate credentials, but they're both on your piece of paper. Okay. So OpenStack services are deployed in containers. So in order to see what containers are running on your OpenStack control plane, you need to execute that LXC-LS-Fancy command. And it will list out all the containers that are running on your control plane. This exercise is to get you familiar with your OpenStack environment. It's not doing anything very special. It's more of an inventory as Melvin highlighted. Let me know how you guys progress when you're ready to move on, okay? So if you... Yeah, you have to join a group, dude. I'm sorry. I ran out of... You know, it's very hard to build... So basically what I've built is 30 OpenStack regions on our public cloud. So to go past that, I would have to ask for special permission to increase my resources. So I didn't want to do that. Rackspace is a bit of Nazi around the whole open public cloud in our consumption of it, so... Yes. Okay. So yeah, so each container that runs is given two IP addresses. It's given a 10-dot something address, which is a local address that you can use to get to a container locally, and then it's given a second address, the 172 address. The 172 address is an address that you can actually connect to from outside of that control plane, all right? So it's just given two addresses, but you always want to use that 172 address to connect to containers. The 10 dots are just local for the LXC to manage the containers. Okay? Everybody's doing okay in step one. So a URL suffix is anything when you see the colon and then after it, right? So that's the suffix to that URL, right? So the prefix of a URL would be HTTPS or HTTP colon slash slash. You got your IP address and then the suffix is anything after the colon. So colon 35 357 slash which is Keystone or I don't remember the other ports. I'm sorry, it's Wednesday. I used to memorize that stuff. Everybody's doing good in step one. It gets harder on step two, okay? Just preparing you. All right. We're going to start talking about what step two is. So go open up step two that's on that GitHub repository, that link that you guys went to. And let's talk about that for a second here, all right? So the process of step two. The reason why you have to do step two is what we're trying to accomplish in step two is we're going to connect to that admin region. So what was unique about the admin region that we brought out is the admin region is only running Horizon and Keystone as a service. So it does not know about Nova, Neutron, Glance, Cinder. It doesn't know about any of those other services. It doesn't know that way, right? But in order to use those services that are on the other regions, that are separate regions, you have to tell your admin region about those services, right? So step two steps you through the process of creating the service account because you know when you create an open stack service on a region you have to create the service account, then you actually have to create the service by recreating the endpoint. So these instructions are going to do that. They're going to help you create your service, create your service account, and that's what we're going from there, okay? So one thing to keep in mind, be careful with copy and pasting from GitHub. Sometimes with single quotes it'll give you this wacky-looking quote that's not a single quote. So just keep conscious of that when you copy and paste those commands. I've seen that personally over and over again. It's extremely annoying, but such is life. So you should be connecting to your admin region to execute the command in step two, the admin region. And keep in mind when you get down to line 43 where you're going to register your endpoints you need to insert the Alpha region's IP address or the Beta region's IP address on certain lines, right? So don't just paste the command as it is. You have to substitute and put in and replace those IP addresses. You see how I have it highlighted there? That's an example, right? Everybody doing good? All right, I won't keep disturbing you. Yes, yes. The regions are running in their own containers? Yes. What is the bigger picture that you're trying to show? Yeah, so the bigger picture that we're trying to demonstrate here is to show that you can manage multiple open stack regions from one centralized place, right? Because right now you can manage Alpha and manage Beta separately and there's no issues there. They can't cross talk to each other, right? So by having a middleman or admin region, you can actually talk to that admin region and choose which region you want to talk to, Alpha or Beta. It's very similar to, like, I hate to use the example because, you know, they're our competitor, but Amazon, right? When you log into Amazon, there's a feature opportunity to pick which region you want to deploy your instances in, right? This is the same thing. I basically made you an Amazon, right? I made you a centralized admin region that you can manage multiple regions' resources, but an instance here or there from one Horizon dashboard, right? The only thing you specify in your request is what region you want to spin it up in. Same thing as Amazon. Yes, absolutely. Yep, creating admin to know about those separate regions' endpoints and when you go to ask Admin Cloud to do something, it will go talk to those regions, right? It's not talking to itself because, remember, the admin region only runs Keystone and Horizon. Yes, absolutely. Absolutely. Absolutely. You can do that with 10 regions, 20 regions, 30 regions. It's no limit how many endpoints you can register. But what if an admin goes there? Well, again, and that's the point where if you lose an admin region, you still have your regions available and you can still make direct API calls to it, right? The only difference is that if you centralized Keystone, it may have a problem there, but it will still keep functioning, but your authentication may become to have a problem, right? Admin region is probably the most secure place as possible and maybe even have multiple admin regions because you can do that, too. You don't have to have one admin region. You can have multiple. Sit them behind a load balancer. Boom. Right? So that's an option. Again, we didn't go that route because that was just another layer of complexity, but that would be what you would do there. Yeah, I know. It's not easy to take 100 OpenStack commands and copy and paste them. I apologize, but this is the only way it can be done. And I will make my slides available. These lab materials will be there and anyone who wants to try it again at home, I will give them three OpenStack servers to try at home as well, okay? Yes. What? The admin region? The IP address for the admin region is right there. Yeah. You need to put in the alpha IP address, which is on the handout, and the beta IP address, which is on the handout. There's a prerequisite. What's that? Yes, they are. No. To be clear, you're not putting in a 172 address to register an endpoint, right? Those are internal addresses. Alpha and beta's IP address that's on the piece of paper. Okay? So if you did that, you may have to delete some endpoints and re-register some. Yeah. Yeah. 172 addresses are just internal. You can't use that as over a win, right? So, well, you can if you register here. So OpenStack endpoint space delete and the UUID of the endpoint you created. And to get the UUID, you do OpenStack space endpoint space list and the UUID of those endpoints will be listed there and you can just take that ID and delete them. Okay? OpenStack endpoint space delete. I know it's complicated. I'm sorry again, but this is how it is. You have a question? No? You okay? So imagine that these alpha and beta regions are in two data centers separate from each other, right? So IP space, you've got to use public IP spaces to do it, right? So just imagine that to be the approach here. Well, again, this is meant to be like a centralized dashboard. So if you're trying to do that or trying to share a region and you would probably not have them share that or if that's a concern of yours, right? If you're looking for isolation or security, you wouldn't do this. This is to keep you from having to maintain two sets of users, right? And also give you the ability to take users to use both regions. Because without doing that, you've got to do backend database. You've got to copy the UUIDs of users from one region to another one to maintain, right? It's like LDAP, right? That's what admin is playing. Yes. So right now, they're stepping through so the approach that we're taking here is we're setting up an admin region which runs just minimal services to control two separate open stack regions. So what they're doing right now is they're logging into the admin region and registering the services that are in the separate regions. Because remember, the admin region is on the ground and horizon. So it doesn't have Nova, Neutron, Glance, any of those other services. But you want to register those services from the separate regions on the admin on the admin server, right? So the IP addresses and endpoints are going to go out to separate places, right? So that's what they're doing right now, okay? If you want to do the lab at home, just come talk to me. I can give you some resources to do that. Okay, no problem. Yes. No, no, no, I didn't. So you'll find that you're registering endpoints for internal, you're registering endpoints for public. So the way it works is open stack itself when talking service to service, likes to use internal, right? And then you need to register public if you want it to allow access to come from outside the open stack cloud. Right, correct. So the admin region needs to be able to talk to those separate regions. Those separate regions over internally published endpoints. Or internal endpoints. It can't talk to it. So you have to register it with the external endpoints so that it can talk to it externally. Open stack is weird like that. I'm not going to even try and diagnose that for you guys, but certain calls, it'll use an internal endpoint. Certain calls, it'll use an external, a public endpoint. Certain calls, it'll use an admin endpoint. You can't tell it which endpoint to use. Open stack just does what it feels it should do. So internal as public is what you have to register. That's just the way it is. I try to do one or the other and things that weren't working. So this covers you. Again, you lose nothing by registering both those endpoints. There's nothing lost. It doesn't mess up any functionality. It doesn't take up anything. So you're good. Everybody still working on step two? Where is everybody? I'll give you a few more minutes before I go to step three A because I've got to keep you moving. I think I lost my partner in crime. So don't start step three until you let me know you guys are ready. Okay. If you're still working on step two, I don't judge you. It is hard. I am going to kind of talk through what we're going to do with steps three and A a little bit here. If you can pause for a second. And like I said, if you're not able to finish it or if you ran into problems, just come talk to me at the end and I will give you some resources that you can go in and do this at home. Probably a lot more comfortable to do it at home. So the thing I just want to talk about with you guys before we start step three A is the fact that configuring OpenStack services. So if you have not had the pleasure of configuring OpenStack services, meaning actually going in and changing the config files, you'll personally know that it is painful and tedious. And the reason for that is that you have to have a super level of attention to detail because any typo that you make in your config file can cause your service to stop running. And you will literally chase after it for hours. So the first thing you learn is mess with these config files is tedious. So you want to try your best to not make any mistakes in your config files. In step three A and three B we're going to use a helping aid to kind of help us do that. We're going to use Ansible. And the reason why I'm going to have you guys use the Ansible commands instead of going in and configuring the files directly, of which you can if you're feeling saucy. I'm not going to stop you from doing it. But using Ansible allows you to focus exactly on the command that goes actually after the exact edit you need to make it makes the edit and you don't have to worry about any other typos. That's the reason for that. So we're going to use Ansible. So in step three A when you go and look at that you're going to see that you're going to be executing multiple Ansible commands and that's the reason for that. So just wanted to kind of show in some more detail as to what the Ansible command you're going to be executing why you're executing it so that when you see it you kind of understand what we're doing here. So at the very top here this is the basic Ansible command that you're going to be executing. Ansible you're going to tell what host to run against what module to use which will be shell and you're going to give it an ad hoc Linux command to do or CLI command to do. So for example at the host is a glance container we're going to use shell as your module and the ad hoc command is using said. So if you have never used said before you need to remove yourself from underneath that rock and explore what set is. Set is the greatest tool ever in my opinion because it can go in and do stuff to files quick replace find and replace it can do so many things. So we're going to use the said command and the said command that we're going to be doing is we're basically going to say go to this com file glance.hpi look for this thing and then replace it with this thing which is the blue line and of course remember we've got to substitute those alpha region IP addresses that are on your handout for every one of those commands right and what it does and if you're looking at it what's in the red box that's a sample command of what exactly we're doing so we're going to go in and make a whole bunch of changes to a config file you may say why the heck do we have to do that so keep in mind each of your regions when created are pointing to keystone and each of their separate regions which is cool if you're planning to work out the way it's supposed to you need to tell your services that they need to authenticate back to the admin region right so when you're in keystone as a user in the admin region and you make a request your token that's valid on the admin region can then be filtered out and it's not really filtered out but what happens is when the service gets evoked it can then authenticate it can then authorize your token that you already have from your admin region right so you have to go into both your alpha and beta regions open stack services to talk to your alpha regions keystone right so that's why we have to go in and make these config file changes right everybody follow me I'm going to present it one more time today where everybody is clear you have to configure your alpha and beta region to talk to your admin region's keystone and the way you do that is by going in and making services config file updates alright let's get started 3A if you're ready for 3A if you're still working on 2 no problem keep working on 2 let me know how I can help okay sure so if you remove the endpoint it uh so if you remove the end so let's do just do a quick open stack space endpoint space list for me I just want to see where we are oh it's just list not dash dash list yeah I know it's like some things are dash some things are not okay so yeah so it's gone um which region did you do this and is this alpha or admin uh history this public I'm trying to follow what would happen here can you pull up the instructions for me because no alpha region did you register these endpoints yeah I did so I registered and then this point says list all identity and remove disable it yeah 172 that's what I was doing right after that point I cannot authentic it anymore and you did this you created these with the public IP address of the admin region yeah uh public step is you have to create you're creating an endpoint on the admin region but just using the admin region's IP address right but I use this internal IP not the you use that IP that address should be working stuff is it because when I go into that admin file it still shows me the public IP list not the 104 that should be fine because the instructions should not have told you to disable the internal address just go back to the instructions for me one more time I should be telling you to disable the public and the admin but not the internal so if you disabled the internal no I was just at step you're just there I was just looking for anything 172 I'm trying to remove I only removed two alright so let me see which ones what happened 178 which is the admin that's perfect you disabled the public it's perfect internal should still be there and enabled I mean after that immediately it started chewing me errors well you can try and force it to use this public one so you can try and change it to that and that add keystone and restart keystone keystone yes you can't get to it from there so how are you just showing me the keystone config oh no I was not chewing it oh you were showing it open okay so yes so why don't you try and yeah go in there and change that yeah change that address and see and then do source again and see if yeah smart man just do any open stack command open stack space user list it's thinking about it probably will go up yeah because it's trying to route back out to itself yeah it's because that's why the internal one stays so that it doesn't do this but I don't know why it's users behaving this way am I example then no it's not fine really pointed all of mine at you have you yeah yeah alright yeah just oh yeah let me see something do you want to take a chair? no no no I don't want you to see it's still trying to curl against against an address that is not probably even active anymore 172 this must be something in config right 331 why is it pointing to 31 so you've registered those other endpoints already yeah and see that's right because it's 31 is your beta region is trying to authenticate why is it alright I have to take a second why is it making authentication crests against itself it's 172 what the heck is 172 all as well on that world I have no idea where it's getting that 172 from because 31 is your beta domain so why is it trying to send a request there I can't even see the endpoints to even know where the problem is okay let's do something else the public the public which is probably disabled it doesn't show it though but no they're all enabled yeah they're not I think only this one is disabled yeah only that one is disabled which is your admin so let's let's fix that now we're getting real hacky right now I don't remember I don't remember all the commands testing my sequel knowledge right now I'm gonna use my other brain update table name yeah it was volume it was tempus it didn't blow up and it wouldn't clean up so we kind of leased out in the state so yeah we did a search on this in England we were using fiber channels as a background that's why we had issues yes just give me one second because I'm afraid I'm going to tell you the wrong answer and then I'm going to screw you up so just give me I think I'm at a point now where I'm okay they broke it too badly and that's it and I'm just kidding it's actually probably my instructions that broke it for them so where column name equals value I thought I gave it to them where ID equals update set enabled one okay where ID that's the same problem where ID equals where is the space it's getting angry about table name point or no I'm on the keystone table yes I think update and table name table name is end point why is it saying under no column when the column name is asking for some column and then a value ID is the column you read my mind there we go it happens so let's test this and see if this is fixed I don't know I feel I'm in between on it should be oh do it so yeah if anyone else had the same problem they messed with the end point and disabled it by mistake we can do a database hack so the world is not over so I just went in there and changed the IP address that you had in there that 172.30.238.2 with just a holder for the service name am I correct in assuming that what we want to do is go and look at the NOVA for instance server that's running here and yeah so for the specific one right so NOVA API container I would go and replace that with the IP address for the NOVA API container no what you want to make sure to do you need to use the public IP address that's on the handout for this place holder so that's done so the address that's used so to get to where you were going is the address is not that address it's actually the low balance address and I can't go into the detail of that but it's 172 so it is that IP address it is literally in the file it is literally that IP address you need to replace was your admin or the alpha beta to paint on what you need to do but yeah that is the case sorry that IP address that's in the file for the 172 whether it be 30 or 31 is the address so you don't have to change that address you only have to change what's in that the brackets you rock stars don't tell me it doesn't work so if you did 3a and 3b you go to horizon it doesn't work it generally means that the admin region when it can't communicate to one of the region's services specifically NOVA it won't let you log in why I don't know talk to the open stack developers about that so oh horizon is not up at all that's a big problem so that's your admin region alright well get into that CLI connect to it through SSH and let's see what's going on with it yeah yeah well you know you don't have it but you get an error message when you don't have it that's the fun thing but NOVA if you register a NOVA endpoint on the admin region and that endpoint is registered incorrectly and you go to log into horizon it won't let you log in that's a feature you don't know about that feature that was mentioned in the last open stack summary you must have missed it no I'm joking did you do sudo su? oh but I didn't no you didn't put in that open stack password man you'd be good so you guys horizon you working your magic not yet okay yeah you gotta do LIC-LAC remember it's a container no worries not to rush you well you can do SSH to it as well okay does it enter? that's the whole thing that's it just do a psgrep for Apache to see if that guy is around yeah restart that guy if you can yeah it's over HTTPS and it's a self-signed certificate so it's going to give you an error message but it should still it would say that only if there's something responding oh it comes back there was a restart needed somewhere in there again running open stack on a cloud instance is not cool it's meant for testing development is not stable you can find all kinds of weird things that happen yeah I got a whole bunch of bare metal just kind of hanging out in my office there sure yes yeah so yeah you have to use the .2 yeah you have to use the .2 because yeah so for internal it still has to use .2 because what happens is running on each of your OSA clouds is HA proxy in front of your services so you have to use the .2 you can't use the container address it gets angry no problem but hopefully you should be you okay or okay yeah okay yes what's going on man password right authorization is not working so can you show me your end points that you have listed on your admin region okay so what I'm just kind of verbally what I'm looking for I'm looking for region one right because that's the only one that's running Keystone so you have an internal address beautiful you turn and disable the other public you created a new public and that's the public address for your admin region yes it is okay you created an admin and you disabled the internal admin region and then you created another admin region with the right address 103 23206 so I'm not sure exactly why it's happening so just show me your open RC file because you can see the internal one is still running so that's the address that should be hitting that's yeah thank you you read my mind 238 238 238 238 238 238 238 238 238 238 238 238 238 238 properly set up, Horizon won't let you log in. Why? I don't know. Specifically, the one to look for is NOVA. So check all your NOVA, right? So make sure your NOVA endpoints you register here, your alpha and your beta are perfect, and then make sure that on the region, the NOVA is configured properly to talk back to here. Okay? And make sure you restart those services when you make those config changes. Real important, services won't change, you don't restart them. Side note, while we're talking, we have book signings this afternoon at the Rackspace Contina. If you don't know what the Rackspace Contina is, unfortunately my book did my signing for today, so I don't actually have any more to offer up. But at 3 o'clock, 3.30 and 4 o'clock, there are gonna be additional book signings at the Rackspace Contina. They are free. The Rackspace Contina is on second and Trinity, it's literally right outside the OpenStack conference, right on second and Trinity. Please stop by, free alcohol, free wifi, and complimentary free DJ. I will be DJing at the Contina at 3.30 today, so please stop by. Yes, that does explain my Twitter handle of DJ, State Fly Pro, I am a DJ as well. So please stop by after 3.30, to the Contina. Sorry, my commercial programming is over now. That's why it keeps bouncing up and down. Yeah, go to CD. Right, were you able to log in? No, no. Actually, he said that when I look earlier, he said something like, log in. What team number are you? Two? I didn't have a problem with your cloud last night. I had a problem with five's cloud and eight's cloud, but not yours. I'm just being honest with you. So. Actually, I saw that, we get it authenticated at some point, oh. The Nova endpoint? Yeah, you're falling into the same trick as everybody else. If you register Nova endpoint in your admin region and your region that you registered there is not configured properly, Horizon will not let you log in. And it'll do weird things. So check your alpha and your beta regions, make sure that the Nova service is pointing to the admin keystone, and make sure that on the admin region, your endpoints for Nova are created properly. So make sure they point to the right address, the right ports, really important. Yeah. All right. You're a smart guy, I'ma leave you to it, man. You're a smart guy, I'ma leave you to it. Again, it's lab is not exactly the easiest thing to do, but once you get it done, you don't have to do it again. That's the key. Yes. I think we are having the same. Yep, yep. So the suggestion, whoever's managing the alpha and the beta region, go in, make sure that your Nova services are configured to point back to the admin region's keystone, right? So remember those many said commands that you had to execute make sure that everything is right. Because if it's not right and make sure the service was restarted. Because if they weren't restarted and it's not right, then the admin region won't let you log in. It's a bug. It's a bug with Horizon and Nova. I don't know what it is. And I could give you a proof point. If you're having that problem, go into your admin region and disable the Nova endpoints that you created, and I'll bet you you'll be able to log in no problem. So it has to do with the Nova service. Generally, the Nova service. Yes, yes, yes, yes, right. Well, he asked the same question. So what I would do is you would set up multiple admin regions, right? You can set, so they say, for example, you can have an admin region in site A, admin region in site B, put a global load balancer in front of them. So if you lose one of your sites, you still have an admin, yes, yes, absolutely, yeah. This example is not a production ready example per se, right? You would never have one of anything, right? So this is just an example, but you wanna make sure you have redundancy. You can have redundant admin regions. There's nothing stopping that. Sorry, you think you're leveled off? We're good, we're gonna be wrapping up soon, so. Okay, okay. So you said you concurred the alpha and betas, Nova API, you get back a response, no issues. Endpoints, they all look really good. I'm just gonna focus on Nova, because Nova is the one that's very mean. Good range. Yeah, that is pretty good, Bluetooth range. That's my boss talking, so give me one second. Yeah, this is where I got, I ran into the same exact problems you guys ran into, and it all came down to the fact that there was one endpoint that was registered, and on that region, it wasn't configured right, and that was a mistake on my part. And unfortunately, as you already know, you can spend hours trying to figure this out. Yes. For the sake of time, has anyone able to step through and actually get it to work? Really? Excellent. So it works. Thank you everybody, have a good night. All I need is one. Yes, so as you can see, it is complex, all right? It's not an easiest thing, but you can automate it. You can, once you understand the concept, you understand how to troubleshoot it. In this setting, I know it's not exactly the easiest, specifically if you don't deal with the CLI or registering endpoints and things like that, but it does get easier, I promise you, and it does work. And gentlemen, can you tell me what you had to do? All right, he restarted the Nova-API servers in each of the regions, all right? And now he's good to go. So maybe try that. But we're at nearing the end. I think we're nearing the end, right? Yeah. I think it's supposed to be over soon. As I said before, if you're interested in continuing the lab at home, I want to try it again with a fresh, new, non-tainted environment. Just come talk to me, and I'll give you my business card. Just send me an email, and I'll set it up for you, okay? So you're welcome to stay and keep hacking at it, and I'm here to answer your questions. Sure. What's that? What's not working? Yeah, you got it, you got it, you, yeah. So, I don't know. You can't guarantee me that. So what is it doing right now? It's not doing anything. You hit enter, right? Yeah. Well, you got too many T's, no. HTT, HTTPS, no, that's all right. So now you get the lovely pleasure, so it's connecting, but it may not work. So you're gonna have to probably go in and check out your horizon service in your admin region, make sure that's running. So you just jump into the horizon container on the admin server, and just see if Apache 2 is running, because it runs, horizon runs under Apache, and just check that out. You may have to restart it, right? As well as you may have to look at restarting Nova API on the regions, because remember that was a problem that someone else had, right? Again, low welcome to OpenStack. Yes, yes, great. So yes, yes, but what I prefer for you to do is not have one admin region, but have multiple admin regions and have them sit behind a low balancer. And by doing that, you don't have one admin region. You have multiple admin regions. But when either, if Alpha goes down, both admin regions are in alpha. Well, that's not true. No, no, that's not true. So I understand what you're saying, and if you lose a region, your best option is to disable the endpoints for that region, right? But if you do it in a way where you're pointing, so let's say you have a region that's sitting behind a, right, okay, got it. You're concerned about it. Right, right, right. But again, horizon is always your back. Horizon should always be your backup to OpenStack, right? So, right. I understand, I understand. But if they lose access to horizon, it won't be a total end to the world. The environment is still up. That's the key. Yeah, but there's a 20-person conference call. Ah, yeah, just tell them horizon's down, it's all right. But you can prove your cloud is up. That's all that matters. It's cloud is still up. So, but yeah, yes, I do understand your point. And there really is no cute way of getting around it, unfortunately. All right, guys. We're gonna start wrapping it up. What's that? Just come, get my card. I have a DJ appointment. That my boss just called me about, AKA the runner of all OpenStack things for the OpenStack Conference for Rackspace, so. I've handed out so many business cards this conference, which is pretty cool. Thank you, this is my card. Just email me that you want to continue the lab. Tell me which, the name of the lab, and I will set up resources for you, okay? Just give me a few days, be kind to me. I have to recover from the 200 emails sitting in my inbox right now, so it may take me a few days to get to you, but I will set it up for you, okay? And these, like I said, the information to do the lab will be on GitHub, I will not take it away, okay? But thank you. Thank you, sir. Absolutely. Thank you so much. No problem, thank you. Thank you for coming today, I appreciate it. Yes. Oh, sure, sure, sure. Hey. Thank you very much. No problem, thank you. Come on. Thank you, sir. You said you wanted to use it. Yes, absolutely, absolutely. But the key is that I need you to email, well, you know what, I have your information, you don't have to email me. I will talk to you. But email me if you don't hear from me in like two days, just remind me, okay? Yes, you could, yeah, absolutely. The key is, is having three instances that are three instances that can talk to each other over public IP, it doesn't have to be internal, right? The only thing you use an Ansible for is the use set. Yes, yes, yes. But I use open stack Ansibles to deploy the clouds, but you don't have to, you can use whatever. No, I've been there for this week. It is, it's, yeah, no, open stack Ansible is the way to go. I have two questions for you, sir. Sure. First of all, is a URI or a URI?