 All right, so let's go ahead and get started. So at this point you should have, I think everyone has reset their machines or I reset your machines for you. You should be able to log in to that machine as student with a password of student. We do have a couple of instructors roaming around the room so if you have an issue, take a look around for some of them. Chen in the back, Adolfo over here on the side, Rob should be around here somewhere as well. But feel free to talk to them if you have issues with something not quite working. If you do have a question or something you need me to comment on, feel free to get my attention as well. So before we get started with introductions and I want to take a look at and show you guys what the network looks like and what our machines look like so you have a better understanding if you haven't been to one of our courses today. I'm going to put that off for just a second because we want to get pack stack started and that will take five or six minutes or so and so we'll start that process first and then I'll go back and do introductions and show you what the classroom looks like and things like that. The question is what session is this? This is the L3 agent, the neutron L3 agent. It is hands on, absolutely. So this is actually part of our new course which is the neutron networking with Red Hat Enterprise Linux OpenStack platform. So it's all designed as part of our larger two-day course that we just released. So it is definitely hands on. I want you all to, if you're not sitting in front of a computer that we have set up, please sit in front of one. The thing of which I'd like to thank Dell for giving us these laptops to use during these sessions that we've been doing. They've been very generous to help us with that. They actually had some t-shirt coupons so if you are interested in a t-shirt from Dell or just would like to thank them for having us here, they have the little, it looks like this, stop by the Dell booth. So there are some back there. If you're interested, come by and see me. I have a few more that you could take as well. Anyway, so let's get started. I want you to log in at this point. On the desktop you should see a PDF, something like 6-L3 agent or something. I can't remember the exact name but it should bring up this practice implementing the neutron L3 agent. This is designed as a workshop so I want you guys to do the same stuff that you see me do so we're going to do it together. We want to walk through this and get things working. Again, like I said, before we go into too much detail, I want to get you actually started. So open up your PDF, open up a terminal. So if you haven't opened up a terminal in a while, go to applications, system tools, and down to terminal. Now, step two in your PDF says open up a terminal and look at the host number. You all have a host number. It should be less than 25. And we tried to make it organized, at least a little bit. But you will have student at host X. Now in our documents we describe that as an X. It's a little italicized X. Wherever you see that you need to put in your number. So remember what your number is. And we should, at least on this side, I think we start at 1, 2, 3, 4, 5, 6. We do the same thing over here. We have split the networks. So we have 1, 2, 3, 4, 5, 6, 7, 8, 9, etc. So that host number where it says host 1, host 2, host 3, that number is really important. I want you to remember it. Or write it down. Excuse me. So the first thing we want to do is SSH over to this server X-A. So open up a terminal. SSH root at server X-A.example.com. Now mine is 0. Yours will be something different. So don't put 0 in there. I want mine to be mine. It will ask you for a password. You're going to type in red hat. All lower case, all on word. Then you should be on this virtual machine. Again I'll describe the scenario in just a second. But let's edit the answers.txt file. Because we want to get that pack stack running while we're talking. So open up the file. We are going to find the L3 agent. So I'll just search for L3. And that's a capital L3. So config neutron L3 hosts. We're going to type in an IP address. 172.25.x plus 100. So you're going to have to do some math. I know it's hard. It's late in the day. But take your X, add 100 to it. And you'll get the number. I'm zero. So I have 100. If you're one, you're going to have 101. If you're 23, you're going to be 123. Right. So in previous sessions, there were some things that were already set. This is one of the sessions where this has not been set. L3 agent has not been configured. That's good. L3 agent has not been configured. So you have to add this manually. Unlike you may have seen before. Now, the benefit that I see here, if you will notice, if you scroll up or down your screen, you'll probably see an IP address that already has something like you should put in there. All right. So I see a 172.25.100.10. You can copy that down here. But notice instead of 10, it's 11. The local machine is 10. We want to deploy it on dot 11 on a different machine. We're going to call it server x-b. So we're deploying most of the open stack stuff on server x-a. We want to deploy the neutron L3 agent over on server x-b. OK. Makes sense? All right. So now that I've got that IP address, let's just double check that we have a bridge called br-ex. So that's the external bridge. We need to make sure it's there, which I do already have that part. And let's see. We'll just make sure we're using GRE. If I look for... Let's see. Where is it? There we go. OBS. In fact, if I search for OBS underscore T, all capital, it shows me my OBS tenant network type is GRE. OBS tunnel ranges one through 1,000. And the interface is eth1. Oops. So that looks good. So really, I just had to change that one IP address and everything else was done for me. All right. So at this point, save the file. And we're going to run packstack. And then I'll go back and describe some of the other useful things. OK. So packstack dash dash answer dash file slash root slash answers dot txt. And let it run. And we have had a couple of instances where packstack did not complete successfully. If that's the case, if you get an error and it shows up in red, we run it and see if that helps. All right. So sometimes there's a timeout issue. We're running on some relatively small VMs. And if that doesn't work, switch to another laptop. That may fix the problem, too. That has worked in the past, too. All right. So anyway, while this is going, let me go back and describe a couple of things. First of all, my name is Forrest Taylor. This is my first open stack summit, actually. So I'm excited to be here. It's been really great. I am the cloud curriculum team lead for Red Hat. So I work on a lot of the cloud curriculum stuff. Rob in the back is one of my cohorts. He's been working at Red Hat for quite some time. He also is working on our curriculum. And Adolfo and Chen are back there. Adolfo is new to our team. Chen is also new to a different team within our Red Hat training. He doesn't know much open stack. So you can talk to him and explain all sorts of good stuff to him. He'll need some help. All right, anyway. So the machine that you see in front of you is a physical machine. We have two virtual machines that are running on there, server x-a and server x-b. And those machines, well, so server x-a has been configured with open stack already. We ran pack stack. We saved it in an image. And then we can reset between sessions. So that's what we've been doing. So it's already configured at least minimally as an open stack server. It's all in one. What we're doing here is we want to take the L3 service and start it over on a different machine. We want to use that as a server that's running somewhere else. Now, it actually wasn't running yet, as was mentioned here. We're not actually running an L3 agent at this point. But we are going to deploy one now. It's going to be on server b. And so the idea here is we're trying to show you that, first of all, it can be done on a different machine. You don't have to have everything in one. You can run it on multiple machines. You can run it on a different machine. You can have your L2 agent somewhere else, your L3 agent, your DHCP server. Those can be run on different machines if you want to. And that's what we're trying to show here. So let's see here. We're still going through pack stack. And again, if you have issues, try re-running pack stack or ask for some help from one of the guys around. Yeah, go ahead. Right. So the question is L3 agents, do they need to be running on your compute nodes? Yes. Normally you would have your L3 agents running on all your compute nodes, and that makes life simpler. Yes, absolutely. So on your neutron server, you don't have to have the L3 agent. It can be on a different machine entirely if you want to. You don't have to have the L3 agent running on the machine that's running the neutron server. It can be off on somewhere else. But it still needs to communicate with the neutron server. It has to communicate with it, but it can be running somewhere else as long as it can communicate back and forth. Okay, okay, yeah, yeah, yeah. So the question is, if you're using ML2 plug-in, do you still need those agents? Yes, you still need the L3 agent running somewhere or you ought to. You don't have to necessarily have it, but if you want networking to work, you still should have your L3 agents running somewhere. And the L3 agents can... This is where you can tell it which plug-in to use, whether you want the ML2 or the neutron server anyway. You tell it where... So if you're using open daylight with OpenFlow and OBS, do you still need the L3 agents? I think... That's a complicated scenario. We may have to talk about that offline a little bit. We'll take it offline. Let's do some more discussion on that. Anyway, so at this point, my L3 agent looks to be good. I get the green light done here under finalizing, and I see a little bit of output from my pack stack. And the pack stack, if you haven't used it before, is just a simple command-line way to deploy OpenStack. And you can use it to redeploy stuff like we just did. We said, let's do an L3 agent. We hadn't done it before. Let's put it off on another server, and it did it for us. Okay, so once we're done with the pack stack command, I'm going to open up another terminal, and we want to get to server x-b. So we're going to do a lot of this over there because we want to make sure it actually worked. Did it really deploy on server x-b? I don't know. Let's see. So ssh root at server x-b. Remember, your x is different than zero. Put in your number, but root at server x-b. So x is your number. Correct, exactly. Yeah, so the... Exactly. Right, right. We had .11 as the place that we're going to deploy the L3 agent. It was, in the other places, it was .10, which is server x-a's IP address. Yeah, that's exactly right. So in pack stack, we told it install L3 over on the server x-b. All right, so let's make sure. First of all, well, what have we got here? Let's just do an IPA. Make sure that we got the right IP address. So I see in if zero, I have 172.25x.11. That's the IP address. Well, actually, that's not the IP address we put in. That's our public IP address. But notice ETH1 is the x plus 100.11. That should have been the IP address that you put into pack stack right there. All right, now we have two networks in each of these virtual machines. Two NICs, anyway. We have a public network that we use, right? ETH0, 172.25x.0. And we have the 172.25x plus 100.0 network as the ETH1, which is our private network, right? And that's kind of the back channel. That's the back end of the management layer. All right, so that's redeployed open stack already. It's listing on the ETH1 interface. All right, so that looks like the right IP address. Yeah, so normally the question is, do you have your controller node going out through the internet or out through a public network of some sort? Yeah, normally, well, assuming you want actual public access out, at least outbound, yeah, you would have the controller node or at least one of the nodes have access out to the internet and that's where the floating IP address is coming in and you push it out to that path as well. Yeah, so that's normally how things are set up. Okay, so on server x-b, let's run open stack-status. Now open stack-status checks to see if there are any open stack type services running or installed and then it shows me what is actually up and what is down. So for instance, you'll see that the neutron server is inactive and disabled on boot. What does that mean? What's the difference between an active and disabled on boot? You don't have any thoughts on that? What is the difference? Yeah, inactive means that it is not currently running, the service command, if you've done Red Hat stuff before, and disabled on boot means if you reboot, it's not going to come up. Check and fig on a Red Hat system will show you that as well. So that tells you, is it running now and is it supposed to run when I boot up? So if you've done things manually, this is a good way to check to make sure that the service is both running and it will run when you boot up the machine. So neutron server is not running. That was actually expected. We did not tell it to install the neutron server over here. We told it to run the L3 agent. That one looks good. Now there's no enabled on boot there. If there's nothing there, it means it was enabled on boot. It just tells you if it's not enabled on boot. It also installed, notice down here, the neutron open vSwitch agent. We're using OBS as part of this, and so it installed the agent for us. We'll take a look at the configuration file here in just a second. And it also installed open vSwitch and the message bus. The neutron server is running on server x-a. So it's running on the first one. If any other services are running there, we just did the L3 service over on server x-b. We need a server running. Yes, you have to have a server running. Yeah, exactly. Otherwise the agent is of no use. All right, so that looks good. So we've got the L3 agent over here. And in fact, you could have run the open stack status command over on a, and you would see all the services that are running there. Yeah, question. You could. Absolutely. In fact, in an earlier session, that's exactly what we did. We moved the neutron server over on server b. Correct. And you could. Absolutely. But you don't have to. All right, so this is just really to show you that you can have it on a different machine. Now it may be better if you, so it really depends on your policies. Is your company more about having it on a different, dedicated machine because it may be loaded down or other things, you might want to split it out. We will get there. We'll get to what is L3 agent? What does it actually do? We'll get to that in just a second. All right, so let's see here. So from server b, what I want to do is to copy the keystone rc file. And that's a file that has some credentials for us so that we can run some commands from the command line. So we'll copy the, let's see, server from server x dash a slash root slash keystone rc underscore admin. We'll copy that to slash root. And the password is red hat. So once I have the keystone rc file here, I need to source it. And now I have my credentials. So now I can run some commands that will be able to let me into the open stack stuff. All right, so we'll get to that in just a second. But check config, pipe it into grep neutron. Let's see what services we have and which ones are actually turned on. So you'll see the L3 agent is the important one at this point. All right, that one is actually turned on. We have the open v switch agent as well. And then there's something called the OBS cleanup, which does some cleanup when we first boot up and go down. So that, again, is what we expected to see. Let's see if the neutron L3 dash agent is actually up. So service neutron dash L3 dash agent status shows it is running. Now that we have the PID, the process ID number, we can do a net stack command to see what it's doing. So net set dash tupln, see if it's listening for anything. Put in that PID number. Oops, we need to grep for it, of course. All right, PID number is going to be different on your machine than mine, so don't put in 3362 unless yours is exactly the same. But from the neutron L3 agent status, I see the process ID number. Grep for that when you do your net stack. Now if I drop the L, and right now there are a couple of connections for that service, but it's actually not listening early. It's not one of those RPC services. It's waiting for something to connect to it before it starts up and starts transferring data. All right, and let's see. Next, we'll just take a quick look at the log, so var log neutron L3 dash agent dot log. Now when you're troubleshooting, it's really important to know where the log files are. Well, the L3 agent has its own separate log, so we can take a look at some of the things that it has here. I see a bunch of info messages. I do see one error here, which is a known error. It's looking for this group key called firewall underscore driver. It couldn't find it. It just continued on, though. All right, so there's my log. So if I have an issue with the L3 agent, this is where the first place I want to check is look in my logs and make sure that I don't see something there that's obviously missing or something that's going wrong. And finally, let's take a look at and see neutron L3 agent dot any. Let me scroll up a bit. In fact, let me just do less. That will be easier. So this is the agent file. There is notice a debug option, so if you're really having troubles and you think it's the L3 agent, you can enable debugging. Debugging can produce lots of logs as normal, so that's something you may want to flip on for a while and then turn off once you're done with it. A question. From the horizon interface, you're talking about deploying it or checking the status or... Oh, yeah. So the activities of creating a network and subnet and all that stuff, you can definitely do on horizon. Absolutely. Yeah. Well, check the status, for instance, or run some commands. So the standard admin things. But most of the... If you're dealing with the open stack networking stuff, yeah, you can do it all, at least almost everything you can do from the command line right from horizon. And it's designed that way. We're supposed to be able to do everything there, too. Yeah. If you have an issue, Rob will help you or Adolfo will come and help you or one of the guys in the back. So let's go down a bit further here. We will, yeah. We will talk about L3 agent here in a second. Let's see. What do we have? Oh, yes. Namespaces. Someone mentioned namespaces before, and I don't think we ever got to play with it. Yeah. Yeah. So we, by default, use namespaces. And if you haven't played with namespaces, that's something you ought to try. It's actually pretty cool to see how namespaces work. The reason that we use namespaces is that we want to be able to have IP addresses that are the same, exactly the same IP address for two different virtual machines without getting crossed over, without having issues. Network namespaces allow us to isolate all of our network stuff into a namespace, into an isolated bucket where we can't get out, where things don't get out from there. And so we are using, by default, network namespaces. And in fact, if you want to take a look at some of the network namespace stuff, the router... Can you route between two different namespaces? You can route as long as you go out and then back in. You can do that, but normally what happens is OBS or Neutron stuff will take care of all of the routing stuff for you. But you absolutely can go out from one virtual machine and into a different one using namespaces, yeah. Which seems strange, but... No, right. You absolutely need to route it to do that. You need something to handle all of those requests. Correct. You need to have some sort of floating IP or something like that that allows you to get out entirely and then come back in through a different one. Correct, yeah. And then that's when Neutron will take care of it and OpenVswitch will take care of all of that. It's not going to get across. Yeah, exactly. Okay. Let's see. Let me go back to my... any file here. We're using the bridge, the external network bridge called BR-EX. So again, that's where the floating IP addresses come in. They go out through this bridge. We're using a certain port, 9672 for the metadata port. We have all sorts of other options that are listed here. And let's see. We already did the logs. All right. So coming back here, and someone asked, can we do this Neutron stuff from Horizon? You absolutely can. Neutron net-create. So we're going to go through and create a network and create a subnet and get a router and get all of that cool stuff that you can do with software-defined networks with Neutron. So Neutron net-create. And we're going to use dash-tenant-id. Let's see if that's going to work. There we go. And you'll notice that I can actually tab-complete. This is one of those cool things that you can do. Once you install Neutron, it will create some bash scripts for you that allow you to do tab-completes on options and sub-commands and other things like that. And so you'll notice that if I do a dash-tenant or T, hit the tab key twice, it shows, well, there are several that start with T. If I add an E, hit tab, I get tenant. If I add a dash, hit tab, I get the whole thing. So tab-complete can be really convenient when you're typing on the command line. So use the services, tenant. We're going to call it external one. And we're going to make this an external router. Now the interesting thing here is we need a double dash with a space around it. Actually, I think it will work with dash-dash router external. It probably works either way at this point. Let me see if that will work for us. Yes, it did. So you can do a dash-dash face. It used to be you have to do it that way. Now you can do a dash-dash router colon external equals true. And I knew that worked because I see this router colon external in that output that shows true. So this will create a network that is allowed to go out to the real world, or out as far as I let it go out. That's the real trick to get it to have a network that is external. We call that an external network. And normally when we have an external network we want to assign that network to a router. And then the router will use that external network as a way to get out. Basically it provides a route to get out through the internet or whatever thing we're allowing it to get out to. Alright, so we have an external network. We want to create a subnet. So neutron subnet. Let me see if they were going to bring a new cable here. Alright, subnet-create. Let's see. We're going to put it in the services tenant. Gateway is going to be 172.25.x.254. Alright, so that's actually my machine up here. So you guys can get out through it. And if we actually had some sort of internet connection service you could actually get out to the internet. We don't because we're isolated here. But that allows me to tell where to get out to. Alright, and again, use x instead of 0 like you see me do. So we've got the gateway. Let's see. We want to disable DHCP. Because we don't actually have the DHCP server running at this point. That was one of the things that I was going to present. Disable DHCP. And we're going to have our allocation pool start at 172.25x.25. And end at 172.25x.34. I'm going to give it a name, external1 underscore sub1. I'm going to add it to make that part of the external1 network. And give it a 172.25x.0 slash 24 subnet. Again, when you see a 0. something for mine that's going to be x.something for yours. So 172.25x.254, 172.25x.25x.34, and x.0 at the end. Make sure you're not using mine because it won't work. Routing will not work for you if you put in my IP address. All right. So let's see what that does. So it gave me an allocation pool from 25 to 34. We're using the 172.25x.0.0 slash 24 cider. We have an IP address there, .2254. That's my machine. That's going to do all of our routing. That's our gateway. All right. So we were setting this up as a subnet at this point. But we do need to set up a router. The gateway should actually be x as well. It should not be 0. So yours should be x.254. So for each of those, you shouldn't have a 0. something. It should be an x.something. And we'll see if this actually works. We need to do a router add. I must have missed something. Neutron router create. So we need to add that there. Neutron router create to create a router named router1. And then that command there about the Neutron router gateway set should work. All right. So we created a router. We created a network and subnet. And then I told it to set the router1 gateway as this external1 network. That's the external network that I created before. All right. So have a router. Have a network. Set it up as a gateway. Should be an external network in order for this to work at all. The router is now connected to the network. Yes. Absolutely. As soon as I created the network, it didn't attach to the router. Right? But as soon as I did... Exactly. The router wasn't there. So I created the router here. And then this step, I did a router gateway set, router1, external1. So that's the command that said for router1, add this network, external... What do I call it? External1. And make it the gateway. Correct. As a network to this router. But make it the gateway. So things will pass through this if we're doing an external request. Yep. That's exactly right. You got it. All right. Let's see. So neutron net-show external1. All right. So I see that... Let's see. External1. Actually using GRE. That's something I haven't mentioned yet in this class. But we set up GRE tunnels. Anytime you do communication between more than one node in OpenStack, you're going to need some sort of protocol like GRE or VLAN or VXLAN or something like that. You can't just use local because it's not going to get out. So we chose GRE just to make life simple. The reason that we chose GRE in our classes is that having something like VLANs to work on physical machines. And it means we have to have a physical router that supports VLAN. We have to have it set up properly. In the classroom, that's a little hard to guarantee to work properly. So we just used GRE to make it simple. All right. We don't care about the underlying network infrastructure. It can be however we want it. As long as we can communicate back and forth, we don't care how else it's set up. That's exactly right. Could be no router at all. Absolutely. Yeah. Could be an isolated network. Yep. And it's still work because of the OpenStack stuff. All right. Yeah. So we get to the point, really what this is all about, floating IPs and all. And this is really where the L3 service comes in. This is the ability to provide floating IP addresses and attach them to instances and have them then get out of our gateway. All right. So all of this is part of the L3 service. I want to get that floating IP address. And someone asked me, you know, what is a floating IP address? I don't understand why. Well, a floating IP address is meant as kind of a static IP address, something that's there and available. It's floating because it's not really assigned with an instance unless I want it to be. All right. So if that instance goes down, another instance comes up. That floating IP address is associated with the instance. On the instance itself, you're never going to see that floating IP address. If you use an IPA or IF config, it's not going to be there. It's just not there because the reason we do this floating IP address or the way it works is that we're doing some NAT rules and some other things like that. Let's say, you know, if I get a request in for this IP address, I want it to route to this virtual machine. All right. And that's where the L3 service comes in. Do that routing and the NAT tables and stuff. The NAT table we set up on the server side. Yep. Yep. And question over here. Can one instance have floating IP addresses from different subnets? Absolutely. Yep. Yep. Absolutely. So normally what happens when you have multiple subnets on an instance is that you have multiple NICs. You have multiple network cards in the instance. The compute node doesn't have to have multiple NICs. No. It doesn't have to. The instance itself, the virtual machine, will be assigned a new NIC because you're doing multiple IP addresses. So if you're doing a floating IP address, it's not really assigned to a NIC at all. It doesn't matter where the NIC is. You have to set up the subnet information in OpenStack. And then you create a floating IP address, like I'm about to create a floating IP address. Create a floating IP address, and then you can assign it to a virtual machine. The virtual machine has to have some sort of network device for it to attach to, right? The virtual machine itself, yes, that has to have the network card. And that can be correct. That can be on the Neutron server, yeah. Yeah, so the question is where is Natting actually done, right? And it is not in the network namespace. It is outside of that, so it's on the Neutron server. That handles all of those requests. So on the server itself, yeah. Yeah, so ML2 plugin, what is it? So it's the newer version of the plugin that allows us to, for instance, it's a much more pluggable architecture. So right now in Neutron, well, with Havana, for instance, the older version with Havana, without ML2, we only had the ability to do one type of networking. I could do GRE or VLAN or VXLAN. I could not do all three of those. With ML2 plugin, I can do VLANs. I can do GREs. I can do VXLANs. I can do whatever the next new thing is. So ML2 lets us kind of abstract a layer above where we are now and allow us to have plugins that do VLANs, that do GRE tunnels, that do other things. The defaults in Havana, ML2, at least in Red Hat's version of OpenStack, it is not the default, it's really simple to add and change. In Ice House, I believe it is going to be the default for generic OpenStack. For VLAN, for VXLAN, yep, yep, exactly. Yep, same sort of thing. With VLANs, right, with VLANs, you normally have to have a physical router that does VLANs and you have to have them set up properly and then you tell OpenVSwitch about the VLAN tags that you have configured so that they can basically send the same tags out through your physical network layer. Yep, yep, exactly. Depends on your physical switch. All right, so, well, we're probably out of time already, but let me just go through quickly here and hit a couple more commands because we're almost finished, actually. Neutron? Correct, I have not created a floating IP yet. I'm going to do that now. I was interrupted. All right, so, create a floating IP address. There it is. It was that simple. All right, I did a neutron floating IP create, tell which network to use. External one is our external network, so I have a floating IP address. It was assigned .26. All right, let's do it again. Oh, I have another one. I now have two floating IP addresses. It is .27. It picks up from the pool for the same. Yep, exactly. Yep, mm-hmm. Yeah, so if I look back at my command here, it's .25 to .34. Now it started at .26. Where did 25 go? Where is .25? So what happened was... Let me see if I've got... I'm going to network namespace because that's where it's at. IPnetns shows me there's a network namespace, qrouter. q stands for quantum. That's the old name of neutron. So there's a router that is now available in this namespace. And if I do an exec on that namespace, and I can do... Let me paste that, there we go. I can do all sorts of sub-commands within that network namespace. If I do an IPA, show me the IP address in this network space, I see a .25. That's where it went. It was assigned to the router. It says, add this thing to the router, add it as a gateway. It said, I've got to have an IP address for the gateway. So it took this first one, .25. And then the floating IP address, 26 was the next one, 27 was the following. So .25 is the router IP address. That is exactly right. So it needs that IP address for the router itself. It is the gateway. That's exactly right. Here out is through this one, and then it hops through the other gateway that I have. Yeah, yeah. Yep, exactly. Okay, let me continue on here quickly. Let's see. Neutron floating IP list shows me the list of IP addresses. Neutron agent list shows me the list of agents that are available in Neutron. And you'll notice that L3 agent, that's the one I just created. That's on server B. But I also have open V switch as agents on server X-A and server X-B, on both of them. I see that. And finally, a couple more commands here. Neutron agent show. And then if I look at the L3 agent ID, to go back to my previous command, L3 agent ID will show it, and it shows me all sorts of cool and interesting things. Let's see. Is it using namespaces? Does it have a router? Does it have an external port? Where is it running? All of that good stuff. And finally, Neutron L3 agent list hosting router, router 1, shows me that server B is the host that's running it. And Neutron router list on L3 agent. And then the ID that I got here shows me router 1. It's got SNAT enabled. And so there's some more information there. All right, so we're well past time. My apologies for keeping you late, but I hope you got something from this. So again, this is part of some of our new courses that we're putting out about OpenStack. This is the Neutron networking on Red Hat Enterprise Linux OpenStack platform that we have recently released. So I'm going to stick around for questions, although we're going to have to kick you out because we've got to clean up and put things away. But thank you so much for coming. I hope you have enjoyed our training today. All right.