 Okay, hello everyone and welcome to this session. It's been a long week so far, so it's impressive to see so many of you still hugging around to listen to this talk. However, I think after the talk, you might agree that it's really time to go home. Don't worry, I'm only joking. So I'll try to make it interesting for you. So my name is Damien Igbe and I'm a technical instructor with Mirantis. Most of you here know Mirantis. So as a matter of fact, what I'm going to be doing now is what I normally do on my job. Every other week I get to explain OpenStack to a couple of people who come for training. And I like it so much, so if you like, call me OpenStack Evangelist. So for today's talk, Neutron Network Namespaces and IP tables. So it's going to be a deep dive, a technical deep dive on Neutron with emphasis on Namespaces. You can't really do Namespaces without Neutron and you can't really talk about Neutron without Namespaces. So it's going to be a combination of these two. My outline is very, very simple. Just going to introduce Neutron and Namespaces. Then we're going to deep inside. We go deeper like you're swimming. Now you keep diving and then eventually you start coming up. And by the time you come to the surface, I should be rounding up. So by a show of hands, how many of you here have heard of Neutron? Thank you. Nice and easy one. So how many of you have really played with it? Quite a lot of you as well. For those of you who haven't played with it at all, for the benefit of all of you, I will try to go through the introduction to Neutron in the way that is going to be useful to my talk. And then after that, I will go into the deep dive. So on my screen, I'm going to be switching between a whole lot of screens because it's not a theoretical talk at all. I'll try to do a lot of writing and doing some commands on the CLI. So for the introduction, for those of you who doesn't know, Neutron is network as a service for OpenStack. That essentially means that you give the tenant the ability to go and provision their own networks. So all that is good. There are about four components to Neutron. The first component, as you may know, is the L2 agent, the DHCP agent, and the quantum server. So four main components for Neutron or Quantum. The quantum server is the one that is responsible for taking care of all the APIs. And once it takes the API, the request, then it has to pass it over to the appropriate agent, be it L2 agent or DHCP server. The way you structure it depends on you. Typically, people will have a control node, a network node, and the compute node. So the controller node will typically have the quantum server along with the plugin that you're using. And then the network node will have the L2 agent and the DHCP agent. And then the compute node, typically, we have the agent, that's the plugin agent, which helps you to configure the network interface, which means that at any point in time, when you try to provision a VM, there's going to be a lot of coordination between the compute node and the network node. So we're going to understand how all this takes together, because that is very important for you to troubleshoot issues with Neutron. So there are many use cases that you can have with Neutron. One of the new use cases, the cloud provider can decide to give very few features to the tenant. In other words, you might not be able to create your own routers, so the cloud provider can have the routers already provided for you, then you as a tenant, you can now provision your own networks. There is another use case in which the cloud provider can give you all access to the APIs, which means you can create your own routers, you can create your own networks, and then you can attach your instances to them. There is a cache, however, if you want to use this feature, and the cache is that your Linux kernel must support namespaces. As cool as the feature is, if your kernel doesn't support namespaces, or if you disable namespaces, then you won't be able to enjoy this feature, which is the reason why namespaces is very, very important as far as Neutron is concerned. So, when you enable the namespaces, that gives you all the ability to do what you want to do. Most of the Linux kernels at the moment support namespaces. From kernel 2.6.59 or thereabout, there is the provision for you to do that. So, typically, Ubuntu 12.04 LTS or Fedora 17.18 and above is going to support it. However, there are some Red Hat releases that doesn't support it. So, if you are using Red Hat, you have to first of all check to ensure that the kernel supports it, otherwise you will not be able to use all the features inherent with it. You mean the kernel? You mean... Oh, well, for... Yeah, I don't know there. I've had a lot of discussions about versions which doesn't support and those that support. So, the best thing is to contact... Yeah, at 6.4, I think supports, but it's not a concrete conclusion. I read some blogs that say it doesn't support, so other blogs say it doesn't support. So, please check. Which one is that? Okay. Okay. We can... Yeah. Yeah, the best thing is it's always very easy to check your kernel. So, when you want to use namespaces, just make sure you use the command to check. I can show you the command to check. Yeah. That's why I think it's better you check. But I know for Fedora 17 upwards and Ubuntu 12.04 is fully supported. Okay. So then, what are namespaces? Any of you here familiar with VLANs? VLANs, okay. Yeah, thank you. Most of you are familiar with VLANs. So, if you understand VLAN in layer 2, which is a situation whereby you have a big broadcast domain and you segment it into smaller broadcast domain, the same principle applies with namespaces. In the routing world, it's used to be called virtual routing and forwarding, which means that you have your global routing space and then you have the opportunity to also have smaller routing spaces. So, most of the routers in the market at the moment from Cisco, Junip and the rest of them, they all have these kind of features. So, when you come to Linux, doing routing in Linux, the name is called namespaces. So, essentially, in Linux namespaces, you have the ability to have different routing tables, you know, example-pay tenants. So, each tenant can have a different namespace, which means that tenant A doesn't have access to tenant B's network. Okay. This is a cool feature because before the time of foursome, there wasn't that feature. Not only is it only ugly because if you do if config on your compute node, for example, you're going to see a bunch of interfaces all lumped together. But now that we have namespaces, when you do your if config, commands, and the rest of them, everything looks neat because it's all separated into different, you know, namespaces. So, that is a diagram I try to use to illustrate it. You have your default routing table, then you can now segment those routing tables into different segments for your clients. Okay. So, we're not going to really treat namespaces at the Linux kernel. We're just going to see the application in OpenStack. Okay. If you've heard of LXC before, LXC, which is used, which is a way of running multiple operating systems on the host, they're all using the concept of namespaces. If your kernel doesn't support namespaces, you still can do LXC. So, it's a very useful feature coming into kernel and a whole lot of people are finding the use for it. In OpenStack, however, the two main advantages are one, overlapping IP addresses. Okay. So, in your cloud, you know that you're going to have a lot of tenants from perhaps all over the world if it's a public cloud. So, what namespaces allows you to do is a fact that different tenants can all create their own networks without you having to worry if they are going to overlap or not. Okay. Because namespaces create all these different segments so they can use whichever IP addresses they want without any overlap. So, that's a very good use of namespaces in OpenStack. Another very useful case is in the case of routing, which I explained earlier with VRF. So, you have your L3 agent which creates all these different routers. So, with that, each tenant can have a separate router. So, if namespaces is not allowed, for example, or if you don't enable it, you can only create one router, pay network node, and not only that, you have to make sure that you put the ID of that router in your configuration file. So, you can see that without namespaces, for the network administrator or the cloud administrator, the configuration is not even, you know, flexible. You have to do a lot more work than when it's enabled. So, if namespaces are not supported, these are some of the things that happens. I've already explained most of them. Except to mention again that if your kernel doesn't support namespaces, then you are advised not to put the L3 agent and the L2 agent on the same node. You are advised to put them on separate nodes, otherwise your system will not be able to handle them. Okay, so, now you understand what namespaces are. How do you see them? How do you recognize them in your cloud installation? So, let me say here that every L2 agent is going to have an associated namespace, which means if you create a network, every network you create in Neutron is going to have an associated namespace. Same goes with routers. Every router you create, we also have an associated namespace. So, for example, if I have three networks and one router, that will give me four namespaces, right? Same goes for all the other tenants. So, the case is this. You have a cloud running, let's say you have about 2,000 tenants, or 1,000 to make it easy. Each of them having at least three networks and one router. So, that will be four namespaces times 1,000. That gives you at least 4,000 namespaces. So, you see that as your open-source cloud continues to grow with a lot of customers, then you have to learn also to handle a lot of namespaces. This is very important to understand because if one of those tenants were to call you and report a problem with the networking aspect, then you have to understand which of the namespaces belong to that particular tenant. If you don't understand this, then you can't even start a troubleshoot. And this is quite different from the traditional way of looking at troubleshooting in the usual either virtualized center or bare metal data center. That is why it's very important for you to understand this. So, we're going to go a little bit deeper. On my laptop here, I have a multi-node installation. Most of you will be familiar with this diagram. It's in the OpenStack Network Admin manual. And this is the topology I have on my laptop. So, I have the controller node here, which has all these components running. Then I have my network node with all these components. I've already mentioned most of these components before. So, for those of you who are not familiar with neutron or quantum, you can have a look at these components the way they are distributed. So, here I have my quantum server. Here I have my plug-in agent, which is a layer 2 agent. Then here I have my DHCP agent and LTRE agent with some of the plug-in agents as well. So, I have this set up on my laptop, and I'm going to show you some of the things that you might need to know. So, on this configuration as well, it's based on levvit slash QMU and the VIV driver is as indicated. It's using quantum security groups and it's using GRE tunnels. And the DNS process we're using is a DNS mask. And of course, IP namespace is enabled. So, with that, let me show you how many namespaces we can have for architecture here. So, I have here my horizon. This is for ten and two. So, I can come down to my network topology. So, that's my network topology I have here. This is my router connecting my SNR network. Then I have network one, network two, and each of them is having one VM connected. So, here you can suspect how many namespaces we're going to have. How many? Three. Thank you. We're going to have three namespaces. One for the router and one for the network-ish. So, that'll be three. So, I have another one. This is the admin network. I can log out. I can sign out and sign in again as user two. And I can come down to my network topology as well. So, it's the way it's configured. I can see it very well. Anyway, let's make use of this one so we don't waste so much time. So, what's happening is that my settings on my laptop is not very good, and I'm not able to see the display here. But I wanted to show you the second network topology to see how it looks like. So, I have three namespaces here and also three from the other network. So, I'll go to my terminal and I'll show you how many of those namespaces we have. Okay? By the way, the command you use to manipulate your namespaces is called the IP command. Most of you at Ministry Linux, you know the IP command, is a command that has replaced the if config. Okay? So, if I want to see how many namespaces I have, I can use the IP command, IP net ns. You can put lists as well, but even if you don't put lists, it shows you the display anywhere. So, here you can see you have one... there's one, two, three, four, five, six. If you count them, so these are all the namespaces we have. Based on the network, I've created for two tenants. Okay? So, you will notice some of them started with Q-Router and the other one started with DHCP-. So, the first command, Q-Router, you can use it to identify which of them is a router namespace and they won't start with DHCP, you can use it to identify the DHCP namespaces. Okay? So, the question you can ask is this. Assuming you have 1,000 tenants, like I said earlier, and you have maybe 4,000 namespaces all displayed on your screen. So, how do I know which of the namespaces belong to which tenant? Okay? At this time, my research, I haven't been able to identify any command that can do this easily for you, apart from the fact that this second, the ID you see here is either the ID of the router or the ID of the network. So, for example, if I say quantum netlist, so I will see my network listed there and if you look very well, this one starting with E, this is the ID. So, if I look for my namespaces, I will see this one starting with this one, EO. So, you see the ID there is same as the ID here. So, with that, you can easily identify which of the namespaces belong to which tenant. Okay? Another way you can do it, if you know the ID and the password of the tenant, you can source the file. You can source the credential file and when you source the credential file, for example, I have one here, if I source cred-prog and I do the same command that I did earlier, so it's going to, okay, it's doing the same thing because it's the same user. So, when you know the user you are talking about, you put the credentials in the file and you source it, when you do your network list, it's going to show you only the network belonging to that user. And in that way, you can check for the ID and use it to match it against the namespace you are looking for. Okay? So, once you've been able to identify the namespace for the user, the next thing you have to do is to try to troubleshoot around that. Okay? So, for example, I can do ip-net-ns-exec, which is a command you use to run commands in your namespaces. Let me bring them up again, okay? Then I run that, I say exec, and then I'm going to copy any of them, the router, for example, and then I'm going to put it there and I can run the commands now, if conflict. Yeah, okay, yeah, thanks. I didn't copy well, thank you. So, okay. So, now I'm seeing some interesting IP addresses being listed. I can also do my route minus n here. I can run any of the networking commands that, you know, normally you run on your global or default namespace. I can even ping one of those interfaces there, 10.0.1.1. I can use this to ping it, but I'll not do that now because we're still going to move on swiftly. I want you to understand something more before we do a lot more of that, okay? So, this is just kind of getting you to understand this. So, now you know how to identify your namespaces and you know how to go into those namespaces and run the commands that you need to run, okay? So, let me go back to my presentation. Okay, I actually have them listed here. So, this is where the networks I was trying to show you but from the installation itself, okay? So, we've already seen the namespaces. Now I want you to understand something else, and that is the topology that I have here. I told you earlier that whenever you create an instance, there has to be a lot of, you know, coordination between your compute node and your network node to set up the networking for that instance, okay? For example, you're going to have the Nova compute trying to create the interface for it, for the instance, connect it to your bridge and then make a dhcp discover offer that would travel over the network, this data network we have here, come to your network node. The network node maybe gives it an IP address from the DNS Max process running and then once that is allocated to it, you can have your instance running, okay? So, we're going to see how that sticks together. Some of you may have seen the diagram again. This is all from your network and quantum network admin manual, okay? So it's a very useful diagram because it explains to you what is happening with this topology. But you have to understand here that we're using OVS login. So if you are using another login, the way this is structured will be entirely different, okay? So about this, we give you an idea of how your instances are constructed on your compute node and then the next diagram which I will show you, this will explain to you how everything is structured on the network node and then with this, they are able to communicate with each other. So I will explain this to you now. Then I will also go back to my demo to show you how everything is linked up, okay? So here we have the compute node. The compute node is this box you see here. Everything is happening inside the compute node. These are your instances, VM1, 2, and 3, okay? So you have a bridge here. This is a BrInt and another bridge here called BrEth1, okay? So when you're using OVS login, this has to be present. That must be present because all your instances get connected to that bridge. So what happens with OVS is if I have 10 tenants, each of those VMs will be tagged. They'll be tagged with a particular number, maybe starting with 1 if you don't alter it. So what that means is maybe tenant 1 will have tag 1, tenant 2, tag 2, and the rest of them. I will show you how that is also structured, okay? But everything will be on the same bridge there. Then this is optional. Well, for my own installation, which is a multi-node, it's compulsory. But I'm using a tunnel. So I call it BrTone or rather OVS, we call it Br-Tone, which is for the tunneling, okay? So this is a tunnel that connects all your compute nodes together. Based on this diagram, so that interface, that bridge is the one here connecting all these ones together. Without that, they won't be able to communicate, okay? That is the only thing that one is being used for, okay? So something is pressing it happening here. Instead of this VM1 connecting straight to your BrInt, there is an additional bridge here. Now this is a Linux bridge, as you can see from the key we have over here. That is a Linux bridge, which needs to be created at the time because of the way that OVS doesn't recognize how IP tables are applied on the virtual interface. So to come around that issue, to overcome that issue, we have this created, and then each time you have a bridge and you want to connect that bridge to another bridge, you also need a tap interface, okay? And you call it the VETH, that's a virtual Ethernet pairs, which helps you to connect two bridges together. So typically you're going to have something QVB with another number, which I will show you. Then on the OVS bridge side, you're going to have the QVO, okay? So you see that on all of them, they get connected here, and that is how your network is going to be constructed, okay? Then you come here to the network node. On the network node, you see here the DNS match process. So every tenant is going to have a separate DNS match process. This is a process which is going to be allocating the IP addresses to your instances, okay? So every tenant is going to have that, and it's attached also to BR end, okay? Don't forget the BR end on the compute node is different from the BR end on the network node, okay? But together with the diagram I showed you earlier, they're able to coordinate through the network to do all the provisioning that needs to be done, okay? So another interesting thing here is that we have QG. This QG is your gateway, okay? I'll show you a diagram now that helps you to maybe visualize it in a better way. So you have your QG, which is your gateway to the external network. So you might realize we have two interfaces here, ETH1 and ETH0 and ETH1. So ETH1 goes to the outside to your public network. Typically that is going to be going into your router in the data center, and then you're going to have this ETH1 here, which is the one that goes to your compute node, okay? So this is how the structure is, you understand this? This has been used to connect all the compute node and the network node together. This helps you to go to the outside, and then this is where you are going to have your gateways all installed. So you're going to have the QR, which is the interface on the router connected to your network, okay? I don't know if all that makes sense to you, but very soon I'll show you my system, and perhaps it's going to click more. So you have a diagram on the network node, a diagram on the compute node, and this is going to be the name spaces, okay? So here we have two networks and one router. So network one, network two, and then router one. So each of them occupying their own name spaces. And you can see the example here. When I show you the IP net NS, you saw how we had Q router dash, then an ID, an idea of the network or an idea of the router. So that is what is shown here with all these ones, okay? So with that, let me show you a little bit more before we go further, okay? So let me mention here as well, how many of you have used DevStack? Okay, quite a lot. DevStack is usually the point of entry into OpenStack. The only thing with DevStack is that, you know, you run all the commands from your single node, okay? So it doesn't make any difference whether you have to run that command on the controller node or the network node or the compute node. But when you split your network the way I have done here, then you have to know which of the commands to run on which node. Because some commands has to be on the controller, others has to be on the network, and others has to be on the compute node, okay? So in the troubleshooting section, I'm going to try to show you which of the commands you need to run on the individual nodes that you have. So first of all, let me show you on the compute node. This is my compute node, as you can see there from the name, okay? So if I do here if config, you're going to see a whole lot of devices being spawned, okay? So you can see all of them there. Some are starting with QVO, some are starting with QVB, and some are starting with QVR, okay? So the QVR, QVB, I think the display, okay, so, okay, getting better. So the display is getting better now. So I can see that my QVB, the bridges, there are five of them. If you want to see them, I can do again more. So you should be able to see them and count them, okay? So the QVB as you saw in the previous diagram, those are the bridges, the Linux bridges that are being created, okay? What it means then is that for every instance I create, there will be four, for every instance I create, there will be four virtual interfaces being spawned, okay? And in fact, let me show you on this diagram again, for traffic to flow from your VM to the network interface, it has to go through nine individual interfaces, nine individual interfaces, because this is one, two, three, four, five, six, seven, eight, nine. So that is how many devices your traffic has to go through before it comes down, okay? So I've shown you the QVB, which has been spawned for every instance to have four of those are going to be created, okay? If you have DevSec at home, you can also run these commands and see the interfaces that you have spawned of, okay? So now you see the bridges and the QVB and the QVR, all of them lumped together, okay? So let's see the switches, the bridges I was talking about. You can see OVS, VS control, show, and that will give you the bridges. But before that, let's run this one. It's usually better. Bridge control, show, okay? So this will show me all the bridges that are available on my system. Oh, sorry. Okay, so you can see here we have VR int, which is a bridge integration I was showing you earlier. Then we have the bridge tunnel. The bridge tunnel is the one connecting all the network, the compute nodes together, also to the network node. And then we have here, we're going to have the VR external. Okay, now this is on the compute node. You won't get it on the compute node. When we go to the network node, you're also going to see that displayed. Bridge control, show here. So that will show you also bridges tunnel, okay? In the diagram I showed you earlier, on the compute node, there was no bridge external because it's not a router. It doesn't have to go out. So here we see it on the network node because the network node need to have the interface bridge to connect, to go out, okay? So what are these interfaces here connected? Well, if I go back to the diagram I showed you earlier, you will see where all these are connected together. So that is going to be okay. So on this one, for example, so your Q gateway is this one and your Q router is this one. In other words, this is a gateway to this network which has to get connected to the router. Those are those interfaces you saw earlier. Does that make sense or is anybody getting lost? Is it making sense? Okay. So these interfaces, we saw them here. Just showing you how everything gets linked together. So Q gateway, there should be two of them because there are two networks, but I think one of them is from my previous network that I created, okay? And on this one too, we have a router with two interfaces attached, so there should be four QR attached here. So one, two, three, four should be four of them. One of them also from my earlier work, okay? Then we see the tap interfaces, okay? So on my network node, these tap interfaces are my DHCP instances. So each of my DHCP instances is going to have a tap interface on the BR int, as we saw on this diagram here. That should be this one here, okay? Okay, so these are the interfaces, the QR, the QR, which is every interface that is connected to my bridge, okay? Okay. So now let me show you this OVS command, OVS VS Control Show. So I was telling you how different customers or different tenants need to have their need to have their networks tagged with a different VLAN. So you are going to see here tag one, which is for customer one, and then if you go down, you're going to see more tags. Let me do slash more, okay? You see tag three here for tenant three, tag three for tenant three, and you're going to see tag two for tenant three, tag four for tenant four. Now this tag can either be for a separate tenant or it can be for a separate network. So if customer A or tenant A has got two networks, there will be two different tags for it, okay? So in my own case here, I had two tenants, each of them with two networks, so they are going to be at least four tags, okay? That is why we have those tags there. So the summary is that every instance by a customer or a tenant has got a separate tag attached to it, and all of them are connected on the BR end, okay? Where pay network is only one tag. Every VLAN is equivalent to a network, and every VLAN is going to have a target assigned. Yeah, just one target assigned. And this is the interesting thing that happens that there has to be a translation between that tag, which is on the VLAN, which is different from the tag on your fiscal switch. Okay, so on your fiscal switch, you need to create your own VLANs, and so there has to be a translation between this internal tag and the tag on the switch. So if I were to use tunneling, which is the case here, you're going to see the flows, which I will show you now, those translations are being done from internal to the outside, and vice versa. As the traffic is going out, there's going to be a translation from the VLAN tag to the tunnel, and as the traffic is coming in from the outside, it's going to get transferred or translated from... Yeah, it's a VLAN tag. Yeah, yeah, every VLAN, one tag. Yeah. So I'm going to show you how the translation is done with a command, OBS. Okay, so that's going to be... Yeah, that will be on the network node. OBS. Yeah, if you're using your VLAN, but if you're using... No, that is for the entire system. You don't... Yeah, for the entire system, you see, you have that... Yeah. Yeah, I think your question is valid. When you go to more than 40, I'm not sure how OBS handles it. When the internal tags go for more than 4096, then I don't know how the OBS is going to handle it, because that is internal segmentation. That is different from outside. From the outside, you have the option to use either your tunneling, GRE, or VXLAN, or whichever you have, which is going to eliminate... Yeah, those segmentations are only locally significant. Oh, no. You were not timing me, were you? Was someone timing me? Okay, okay. No, I'm not done yet. One second to finish this. At least I can show you this before we go. This is VS control. No, it's VS control. OBS? Hello. This is the one. Yeah, DP. Did I remove the downflows? No, but what I want to show is still different. Let me see. OBS. This works. I don't know why. Okay, so... Okay, this one. OBS. And I'm going to pose a slide by this one I'm looking for. Okay, so yeah. This is the one I want to show you. This will show you all the flows, how they have been translated from inside to outside and from outside to inside, okay? Well, my time is up. I'm going to pose the slide. So if you want more, you can have a look at them. Thank you for your time.