 Okay, it looks like we have most of the line in here, and it looks like we've got all the seats, okay. So we've got all the seats full. So let's go ahead and get started here. That way we won't have to go too long today and we'll try to fit everything in. Welcome to everybody. Who all has had the wonderful opportunity to have been up all night trying to figure out how to get your neutron going? You know, I mean, which one, I was up half last night doing that myself, so I can sympathize with that. You spent these hours of getting your open stack installation up and running, you spent up your first VM, and guess what, no IP, what happened? So let's, we're gonna talk about that. We're gonna show you what to do. We're gonna show you how to troubleshoot. In fact, what we're gonna do is I, this is supposed to be a hands-on seminar, but the problem we've had is the Wi-Fi has been up and down this hour leading up into this. I don't know whether it'll be up and down, so it's gonna probably be impossible to try to have you all log into the instances. So I'll walk through and show you how to do it. And I took the precaution of making sure I had VMs running local on my laptop so I didn't have to rely on Wi-Fi just so we could get around it, which is why I was up half last night. You know, I get installation half done, and then all of a sudden Nova says I can't find an image and Neutron doesn't wanna. So let's look at that. Let's talk about what's going on, but let me introduce myself. My name is Phil Hopkins. I'm a principal engineer at Rackspace, and I contribute a bit to documentation, Neutron. How many of you have all read the OpenStack networking manual? Hopefully everybody's at least seen it. Good. I helped write that. I can, a small part of it. There's another Racker by the name of Matt Casabar that helped write a whole lot of that. And if you get into the documentation IRC channel, you can see him as Sam-I-M, and you can ask him questions, as well as my handle in the IRC is Phil underscore H. I have some of the other folks working with me, John McKenzie over in the, my right, yeah, my right, your left. There's a couple of other Rackers that will be in and out here if we can pull off some hands on towards the end we'll try to do that. Matt Dorn and Byron McCollum, who are also from Rackspace, that we've had the opportunity to go around the world teaching OpenStack to the world, and I go around the world teaching Neutron to the world in a Neutron class. So let's talk about Neutron, because it's one of those fun things that we all get into, and like we said, we've all had the fun of working with. One of the comments I'd like to start out with is that how many folks consider yourself Linux admins or Linux skilled, knowledgeable? Now how many of you consider yourself network engineers? Yeah, a lot fewer hands. And here's what we've got to talk about here, is that you Linux folks, see I used to work, I've also worked as a Cisco admin from time to time. I've taught Cisco networking classes in the past. So I have a fairly strong networking background, along with a very strong Linux background. So it gives me the best of both worlds. The problem we find is that we have lots of folks coming into Neutron, need to make the networking, and they're typically Linux folks. And Linux folks all think we know networking, but if I gave everybody a quiz right now, tell me how many packets occur for a machine to get its IP using DHCP, and what type of packets go out, each of those are, how many you think could tell me successfully? Yeah, you see, that's the problem. Not many hands go up. So this is the point, is that we need to understand the networking. And so I'm gonna take a few minutes to talk about the networking stack, and then we'll look at what happens when Neutron doesn't work. So the point being, as we look at this, is that you need to understand and go back to your networking 101. Get the networking basics down. Do we have those up? We'll make them available here. I think one of our gentlemen has it on a URL that I'll post at the end. If not, we'll make sure that they're up and available. Because there's not many slides, but yeah, we'll make sure you can get a hold of these. Good point, and yeah, I'll figure out how to get it out or you can find them. Bottom line is that the networking stack is important, and in the Neutron world, we're actually focusing at the bottom four layers of that stack. In other words, layers one, two, three, and kind of three and four are kind of interrelated. That's where all our focus is. And it's what you need to know is how this works, and you need to know in detail. This is a problem, is that we kind of know, but then the specifics we don't, and that's what gets us in trouble. How do we figure out what goes on? How many of you think that TCP dump is your friend and ally? If you're a neutron float, you gotta get your hand up high and you need to know how to use it well, because this is your way of finding out what's going on. But this OSI model, you need to know, and you need to know what's going on at the various levels. You need to know what happens for what type of packets is our request. We need to have this type of networking detail, and I urge everybody, if you're getting into neutron, if you haven't done it in a while, it'd probably worth going back, taking a networking 101 class just as a refresher. I don't have time to go through all that detail here, but it's critical that you understand that. Because as we get into neutron, as we get into what happens, one of the things is that we need to segment traffic. We have the issues of scalability. We're trying to put up clouds that may have 50, 100, several hundreds of tenants and networks, and all this needs to be separate. Each tenant gets in there, needs to think that they're the only world in town when they're on our cloud. They can't see everybody else. So we need this traffic isolation. We've been doing things like that for years. We use a concept called network overlays. And really, that's all this is, is a type of network overlay that we used to segment it. And there's a couple of choices we have. We can use VLANs, and we use that quite a bit in neutron, or we can use tunnels, GRE or VXLAN tunnels. We can use those for overlays to segment. The problem with VLANs is, how many VLANs can you do? Well, you have what, 12 bits there? How many bits? Okay, this is that. I have to make sure everybody understands. There's 40, 96. Okay, I think that's a little more than 12 bits, isn't it? But anyway, it's 496 VLANs. And in any reasonable-sized cloud, we can easily exceed that. So then we have to use the technology, such as tunneling, using GRE or VXLAN. So understanding how these technologies work is critical. VLANs are layer two implementation. In fact, in the normal frame, the framing portion of it, not the IP portion, but the framing portion, is a section that we can do VLAN tagging to separate. And we use this all the time in our networks, and in fact, we set our switches up. So when we plug our machines into the switch, we only get the traffic that's on the VLAN that we're on. We don't see everybody else's traffic. We use that for security to separate. It's a whole lot easier to manage a number of smaller broadcast domains than one huge flat domain for an admin. And we set this. So we've used this for years, but we have the 496 limit. The other technologies that we can use are tunnels, and these are fairly new to a lot of us. We don't understand or are we familiar with it. Both of these basically take the previous packet that we had and wrap a header around it, whether they're using GRE headers or VXLAN headers. Guess what? That presents a little problem. We now present to our framing layer potentially a packet that can be bigger than the MTU that we have. If the bottom level packet coming is 1,500 bytes, and then we wanna wrap, for example, a GRE header of additional 50 bytes, we got a potential packet of 1550. That's a problem. You can't route those packets. So now we gotta do things like setting our MTUs down so we don't get into trouble there. It's things that we have to think about what's going on. And so both VXLANs and GRE have this issue that you have to think about and plan ahead of time. You've gotta make sure that your VMs have MTUs small enough. The other thing is there's something that should take care of it called PMTUD, which is Path MTU Discovery, which allows machines that have smaller MTUs to talk to ones that have bigger ones, the ones with bigger ones will send a bigger packet. And if everything's set up right, then that packet gets dropped, but the place where it gets dropped to the smaller MTU will send a ICMP packet of a type that sells a originator, the packet it receives is too big. It should dial down and resend a new packet until it finds and gets it through. Of course, then you get the admin that's gonna keep his gateway real secure by dropping all ICMP traffic and guess what? They just broke the net for you. And so there's things like that you have to understand that can break things very subtly and in some ways that cause all sorts of difficulty here. So both of these have some different technologies. VXLand, like this talks about, uses something that actually is Mac and user diagram protocol. So it actually talks to a VTEP, what we call virtual VXLand terminal endpoint, if I can get the terminology out here, which listens on a particular port. And here's another wonderful thing that gets out is when Linux implemented its VXLand plug-in, there was no standard. So it uses a port by default of about 8,000. Since that happened, IANA has set a port in the 4,000 range for VXLand. So guess what? If you load the Linux kernel VXLand tunnel without changing the port when you load it, VXLand's not gonna work. So now if I try to use mixed OVS, open V-switch and Linux bridge technologies, and I don't fix that on my Linux bridge side, guess what? Things don't work. It's these type of things and knowing how to find them out is what you have to learn. This is why I talk about networking 101 is critical to understanding how things work. And knowing how to read a TCP dump trail is gonna be critical to finding out what happens here, as we'll see in a few minutes here. So Linux networking. We need to understand all the tools, all you network folks that raised your hand, how many of you are skilled at Linux command line? Because you need to be that. The issue here is to troubleshoot Neutron, you gotta be skilled in both worlds. Not just the networking world or Linux world, you've gotta be able to marry the both. And the things you have to understand, for example, is like the next one. How does the network traffic go through the kernel? We have things like IP table rules. In fact, if you look at open V-switch environment on a compute node, you see that the traffic, the tap interface goes out into Linux bridge and goes into another V-parent open V-switch. You wonder who came up with this? It looks like Rube Goldberg. For you folks that are English speakers or you understand who I'm talking about, you folks from other countries may not understand Rube Goldberg. But he was an artist and other things from back in, I think, 1930s type. And he'd come up with these machines that were, you wanna go light a candle over here, instead of just walking over the match, he'd have a contraption with things falling over and you'd just wonder how it ever happened. Eventually the candle got lit. And you look at the Neutron layout, sometimes it feels like that. But there's a specific reason. It's because we need to do security group rules and we need to filter traffic immediately as it comes in and goes out of a VM. You do not want a VM out there that somebody's broken into it, now spoofing MAC addresses to make bad things happen in your network environment. And so we apply security group rules on the traffic immediately as it comes out of a VM. If you spoof the MAC address, you know what's gonna happen next on that packet? This is gonna go to the bit bucket. We're not gonna let it go. We're not gonna let you change your IP on an interface or we assigned it, you don't get to change it from open stack viewpoint. And that's why we have that Linux bridge right here is because of the flow table. We have to move from one user space program to another user space program and we have to get the packets through IP tables so we can filter them. And so we use the Linux bridge so that they will go through the forward table and we can filter the packets. So it's understanding also how the IP table rules and how the network traffic goes to the kernel is critical also to understanding where you can lose packets and where things can go wrong. If we look at TAP interfaces, we build a TAP interface. A TAP interface is what we plug into a VM and the traffic from our NIC and our VM comes out this TAP interface into our, out of our user LAN program. And so it's basically, as this says, a layer two virtual internet, ethernet device that we use. The VM will view that as a normal NIC card. It thinks it has a NIC card and the system does that. And we use that to send data to other user space programs. In this case, like I mentioned, in the case of Neutron, we plug it into a Linux bridge so we can filter traffic through IP table rules right on that Linux bridge. Another terminology you need to understand the Linux world is bridges. Now, from a networking viewpoint, we all understand what a bridge is. We plug things into bridges like a hub and traffic goes in, gets reflected out to everybody and there's no filtering. Well, a Linux bridge is not quite like that. It's really, I put it in connect with more like when we have an unmanaged switch and does MAC address learning. So it only sends packets out the interface based on MAC addresses. It's not sending it everywhere. It's a smarter device than a simple bridge. And so we use bridges all over the place inside of Neutron. And it's important to spend time studying, understand that. So understand how they work. And as it points out here, you really need in a Neutron on your compute node and on your network node, wherever that sits or your network functionality, you need bridge utilities installed. So if you're running Ubuntu, it's get install bridge utilities. It's yum install and the red hat version of bridge utilities. I think that package is the same name in both environments. You need that installed. It should install by dependencies, but sometimes it doesn't get there. And if you don't get bridge utilities, you can't build your bridges. And so your Neutron gets broken. So it's a package you need to know and it's important to understand because the other thing is we'll do is use something we call VEath pairs, virtual ethernet pairs. We create those. These are like having two layer two knit cards, if you please, connected with a cable. If I put a packet in on one VEath pair, it comes out on the other end. And so it just basically connects two. And we use this as a virtual path cable, a patch cable like it says. That way I can connect to my Linux bridge and open V-switch. I can do, for example, that way. So I can connect between things between kernel space and user space or different functionality that goes on. So understanding how VEath pairs work. They're actually quite easy to build. And when I teach a class, we actually have a lab where we go in and manually create and connect, wire up some systems using VEath pairs and VXLAN tunnels and let you do it, see how to do it manually. It's something to learn how to do because if you know how to do it, you can then know how this troubleshoot and see what's going on inside your machine here. The other terminology we all need to know is network namespaces. Something that's fairly new, it's come about in the last several years. Network namespaces are very important. They're critical to making neutron work because inside a network namespace, it's like being in its own little world. From a network stack viewpoint, it only sees the stuff inside its namespace. And it's how, for example, who wants to have to go give each tenant tell them what IP ranges they can use? I don't wanna do that. I don't think anyone's wanna do it. Allow us to use overlapping IP so tenants can have the same IP range and keep up separate. We put endpoints inside the namespaces so that things say isolated. This idea, like it says, is similar to a CH route environment for a network stack. And inside each of it has its own route tables, has its own IP addresses, has network devices. It's totally segmented and separate from anything else. So I can put one end of a vEath pair inside a network namespace and terminate to have things happen for router or DHCP service running inside that namespace. And it won't see the other ones that are running inside other network namespaces. So we don't have time to go more deeply into this but these are important things to run and to know about. I have a question for you Linux folks. What command do you use when you go try to find the IP address of your ethernet card? Who said I have config? Raise your hands, come on, I heard it out there. Okay, all you that did that, which if you're right handed, you just take that right hand and say, don't do that anymore. Hopefully what we should know by this is that those commands have been deprecated. I have config could go away at any time. There is a new set of commands that came with the IP route to stack that came out. And these are the IP commands and they have the ability to allow us to do lots of other important powerful things. Now I'll tell you a little aside, the IP route command, it's output. I don't like the format. I like the route command and I still use it. So anyway, cause I don't like it. But for example, IP neighbor replaces ARP and IP neighbor gives you information you'll never get with the ARP command. Like the status of whether it's reachable or whether it's gone static, stale, what type. By the way, how long does something stay in an active state versus a stale state? And how long is it kept in the ARP table? These are things that we should know. It's active for a minute, it's in the ARP table for five minutes. And so that four minute thing that's called stale, we treat it like it was recently used, we treat it like it's an active one. However, if we treat it like an active one and we don't get a response in a certain time period, then we can send an ARP request. And so those are handled a little different than an active. And the system handles it. So understanding how these are used and how we can use them, creating the IP neighbor to look at our IP tunnel to create tunnels and oh, by the way, there's one command, I'll give you a tidbit that you try to find the documentation in this because it doesn't exist. There is a dash D flag with IP. The only time you ever see it documented, if you do an IP help or IP dash H, you'll see that says there's a D flag that says details. Well, what does that mean? Well, I'll go to the man page. No, it doesn't, it's the same thing about D flag. You don't find documentation. If I have a interface that was built is like a sub interface for VXLAN or VLAN or GRE tunnel, how do you know what the tunnel key is or are the VXLAN ID or the VLAN ID is? Well, unless I made the interface with that built into the name, you won't know. But if you run an IP space dash D, space link, space show, and then the interface name, and I'll show you this when we get to it, you'll actually be able to see that. You'll see that information. It's the only way I know how to find that. And oh, by the way, it's not documented anywhere that it does that, so that's one of these joys of all the Linux world here. The other command that you have to understand how to use in troubleshooting Neutron is the TCP dump command. You've got to be able to use, this is your magic tool. This is where you find out where things go and don't go, where packets happen and don't happen. It's a packet analyzer tool. And the one nice thing is that I don't understand where all the bits in a packet, I know where to go look it up if I have to, but that's painful. I'd rather do a save as a PCAP and let this wonderful tool called Wireshark do it for me. Because I know when Wireshark does it, it gets it right every time. And if I gotta go figure all those bits and bytes out, my math is bad enough, I get it wrong half the time. So using the automated tools is wonderful and use them. You can use Wireshark to help you, if you're having trouble finding out understanding. And if the person that wrote this tool is in the room, I'd like to hug them or at least shake their hand or do something. If you're not familiar with it, easy OVS, because if you've twisted your way around OVS flow table and try to figure those out every once in a while, you'll love this tool. Now it has a little quirks and it could be maintained a little better and you install it, you gotta watch out, I can do some funny things to your environment. But if you get it out there running, we'll use it here and we'll look at it in a few minutes. This one is a lifesaver because it puts the flow tables in a much more readable format. And oh, by the way, if you were done to dump flow tables of the open V-switch, you know as it says, out port one. What? Port one. Oh, I gotta go over here and look in this thing to figure out and look through what port one is. This one tells you that, oh, port one is a patch cable and it puts it in there so it fixes all that for you and does that very nicely. So it's a handy tool. It's really powerful and very useful. Let's talk about neutron for a few minutes and then we're gonna look at a real life scenario of what can happen and we'll troubleshoot and look through that. First off, understand the key neutron pieces. You have a neutron API server. That's a server that runs on your controller nodes or node. We also, the server uses a database for persistency and it talks to all of its components through our messaging queue. Typically we're all using RabbitMQ but you could be, there's several zero MQ and there's several other ones, QPID and others that people use from time to time. And then we'll have a plugin agent, Linux bridge or open V-switch or some others out there we'll be using. And then we'll typically run a DHCP agent and L3 agent and they might be more than one of those out there that we have running that they all talk through this messaging queue to each other. We need to understand those. We also need to understand the magic terminology that neutron uses. If you look up in the networking manual and the neutron book there and the open stack documentation, it tells us that a network is an isolated segment analogous to VLAN. I say that's baloney. Basically when I create a network in open stack it's a place holder into which I can plug subnets. So I create something I call a network, it's a name and in that then I'll create subnets that I plug into my network. And then to route between subnets I have to create routers and I'll have to have those type things. Every time I plug a VM, a DHCP server or router into a subnet they use something we call a port and this is like it says it's a connection point. And so we can have these connection points. You ever get to where you try to delete a network and it says it won't do it because there's still ports. Somehow something didn't get deleted. So I have to go manually do a neutron port list see the ports on that network and maybe start having to make sure I deleted all the VMs if something didn't clean up properly I have to manually delete the ports before I can delete the network. And so we have the ports out there these are connection blocks. Now, if you're trying to learn neutron or troubleshoot let me give you a suggestion and make your life easier. Start out with a Linux bridge version of it and get that working and understand that before you go try to do an open V switch. In fact from experience in years past we had our ranks based private cloud group in production we still only use Linux bridge versions because back up to about a year ago the versions of open V switch that were being used in Ubuntu and Red Hat were the version one and they had serious reliability problems. And rather than have our customers have problems like kernel panics because when the open V switch kernel module plugged into causing kernel panics that doesn't make anybody happy and their cloud goes down because of that we started using Linux bridge. At some point in time we'll probably go back to using open V switch. It's finally at version two getting what I call relatively stable and seems to be much better. But if you're trying to learn it learn how the Linux bridge so you don't have to deal with flow tables it's much easier and more sensible because here's the problem that leads to the next slide this eye chart that I have up here is the open V switch version of it. And you can see over there the in the top far right hand side of that or my right hand your left hand will get this right. The instance that has to go through the Linux bridge goes to open V switch it then goes into the bridge that's the integration bridge it goes gets hooked to the patch cable to the tunnel bridge and then it will be put out in the work to wait either going to another VM on a different node or into the DHCP or a router namespace that would be on the network node and in the network node you have a similar thing a tunnel bridge, integration bridge and V pairs that hook this all together and tracing through that can be a bit of a pain. The other choice is what I call a bit simpler is the Linux bridge version. In this case you see two compute nodes basically the instance is plugged straight into a Linux bridge no open V switch no flow tables to deal with and VXLAN or VLAN or GERY tunnel implementations these drawings are taken straight out of the networking book so if you want to use that to refer to the open step networking guide that is your handy dandy decoder ring for a lot of this and I recommend you spend get close to it and familiar with it. So with that said, these are the flows and I know everybody's got these two diagrams etched in their memory right now so we can go through and start troubleshooting. Fortunately I do know them so I can talk you through it so we'll look at this. I'm gonna bring up here in just a second so let's look at some nightmare scenarios. I'm gonna go online here, I've got some local VMs and we'll look at at the end of the class if there's some time we'll get some hands on but I don't know how the wifi's still doing in here so let's look at what happens. Let me bring up and I have right here let me bring over the screen. Unfortunately I can't make this any bigger. I've got a VM here so let me log into it. This is, I just use a simple cirrus. Trust me when I get into the command line it'll be where you can read it but this one, things going here. Let's go there and let's log into it. I always make the comment. I'd like to meet the person that wrote has done the cirrus image because somehow they apparently must be a baseball fan and it's come true again. The password in there is Cubswin and anybody that's a Cubs fan is doomed to a life of agony and pain I think for those of you who know American baseball there. Also, so we're in there so let's do an IPA and let's see our, uh oh, we spun for our VM and how many of you been up all night trying to get this and you get your VM up and you see exactly that. What are we missing? We don't have an IP address on our interface. In fact, if I try to do if up, guess what's gonna happen? DHCP, I bet it's gonna fail. So let's try this. Let's do a pseudo. I'm gonna cheat here and become root and let's just run the UDHCP command if I can type here and that will go off and start sending packets and sure enough it says to send discover packets out there and so what kind of packets is it sending out? Well, DHCP is a broadcast. It's a Layer 2 broadcast packet. So it's going out to everybody and it communicates on also on the IP ports 67 and 68. So it's doing this, but this is a Layer 2 broadcast. It's looking for a server. Okay, we clearly aren't getting IP and we've all seen this, we've been up. So let me go and let's get my window here and I'll go over to the compute node here. Let's bring up the compute node and let's do an IPA. Let's look at that and I actually have one working here. Okay, so I do an IPA on my compute node and I can see down there interface 24 is a tap interface. Well, I know that's gonna be my interface for my VM. So let's just go make sure, as we troubleshoot this thing, let's go grab this tap interface number and let's run TCP dump on it. See if we see the packets come out of our VM. So we're a good place to start, right? So I'm gonna do a TCP dump. I always like to do a dash E, which gives me the layer and yes, I know you folks in the back can't see so let me pull this up a little higher here and I'll do a dash I and then I'll paste that in there, that tap and let's let it run here. Okay, and then when I pulled that up, I got this, let's get this down here so we can watch what's happening and we'll see this thing here in just a second. It will start sending some more DHCP packets out there and we should see them come in here in just a second. And if I wanna make it happen quicker, I can control see that and just up arrow and start it. And sure enough, I see packets coming out and those look like DHCP request packets, don't they? We see from a one Mac address to a broadcast of FFFF address and we can see it went from port 68 to a broadcast and port 67, but we're missing something. We see them going out, but nothing's coming back. We've been there before. So let's try something here. We see these going out. Well, let's take a look at open V-switch and let's see what's happening inside open V-switch and I know we've all done this. You run this, we run that and we see all this and what is all that garbage there? We all spent hours trying to figure out what all that means. Let's try something else here. Let's look at it another way here. So I'm going to easy OVS here and I got a nice green screen here. So let's try dump. Let's look at the tunnel bridge here just for a thing here and this gives a nice blue here and if I go slide up to the top of the output there we can talk down through it and see what happened here and you can see the columns. You can see in green here that says the on my left or your right most column is an ID number, so rule number ID and then we get a packet count number and then we get the next column is a table number and then we start seeing the rules in here. Next actually is the priority level and then we finally see the rule and you see the various first rules as come in part, the GRE part as resubmitted into table three and you can see the second rule says if it came in the patch INT so it came from the patch cable from the integration bridge that get resubmit to two and you can see here that the first rule has had 40 packets match. The second rule has had 142. So we can see the packets and we can go down through this and see the different packet numbers as they've matched the different rules here and different flow table rules. So let me get over here and we've got some packets going through there. I see from the bottom so we can go through this. So let's go notice some rules that learn has a 38 and the bottom rule has a 65 packets. So if we go, let's just run this again and see if we get some different numbers. Sure enough, the very bottom rule right now it had 60 some before now it's got 74. So we see packets in that very bottom rule is saying that that flow table rule is going to be output on the GRE tunnel endpoint. So it tells me that at least open V switch flow table thinks it's got packet that are matching flow table rules to point of going out of open V switch. So wherever we're broke it's not broke right here on my compute node. Packets are going out and sure enough, if I go into I can also do an exit out of here. And if I do a TCP dump do a dash N, I'll do a dash E and dash I and my V ETH five on that is my data path in this particular environment. So if I sit and look at that and we can look inside of the very bottom here and wait for just a second for a we see the VXLAM packets. We see the DHCP requests packs coming out of the compute what they, this is mean is coming out of the compute node coming out open V switch is now should be going over to the network node. So let's go over to the network node and let's go to the node that's running my other end of this and I'm gonna just stop that there. Let's get my other term law just so happens I have a connection open there. Let's go over to my controller network node and let's do an IP net and let's just look at my network namespace and if I grab my network namespace let's look inside my network namespace here see what's going on here and let's do just an IPA to see what we have inside that one and we see a loopback, a tap and a tap interface so it's a tap interface on which this is a spot onto which my DHCP server is gonna be listening so let's do a TCP dump there let's just see if it's making it into the DHCP server let's start there so if I do a TCP dump I'm gonna do a dash E, a dash N I'm gonna also give you a little magic here if you're listening inside network namespaces use the dash L option because otherwise it buffers the data and you'll sit there for five minutes thinking there's nothing going on and then you go control C to kill it and you see all the data come out the dash L will allow you to see the data in real time so then I'll do a dash I and let's paste in that tap interface and let's let it sit there let's move this up here and then while that's sitting there we can see that over there I'm gonna kill this process on my VM and start it up again let's see if we can packets go through there Oh, that's what happened we're in the namespace we can see where it's sending discover packets out on my VM but do we see anything coming in TCP dump? Okay, so we know that the packet came off the controller board or came off the compute node we saw it going out there but we don't see it inside the namespace so let's go and see what let's go back on to our controller node let's look an open V switch so if I do a bring up easy OVS let's bring that up here and let's do a dump let's look at our tunnel bridge right here and notice we've got a few packets that have made it through to the end of this thing and some of these have hit other places in here so let's look at these last set of numbers let's kill this and let's restart so we get some packets go through this and send discover so if this packet was making it through open V switch at this we should see the flow table counters increase, right? the packet counts so now we've seen some packets we at least have seen a couple three packets go through there let's go look into open V switch and run our command again and look if I come over here if I get my mouse pointer over here and it's always fun working on two screens because I lose my mouse between the two screens here but if you look at my packet count right here packet count right here none of those this is my way out of it at one way and then table three would be my other way out or table up here if we go up here you can go see that we would see that the packet counts never change between the two okay so packet counts that means that we've got a problem some are between open V switch and our compute node packets aren't getting into it so let's go from there we know that or at least I know because I built this vEath5 vEath5 should be the interface of which they came in on my controller node should be coming into this so let's run a TCP dump there E dash N dash I V E T H 5 oh well I guess we just found our problem what's that tell us TCP dump can't connect there why? because the interface is down okay so let's that helps out so I'm gonna do an IP link set V E T H 5 up so we'll set it let's do an IPA and let's scroll back let's scroll back up here and look at vEath4 and 5 or vEath4 we have we can see vEath vEath5 is now showing up vEath4 is showing up so we're good now let's go to our VM here and oh look what happened it got an IP it's already gotten his IP magic so instead of being all up all night in about half an hour we figured out what was going on okay let that's kind of neat so let's go ahead and let's make sure we can ping our device it's got an IP address and then let's see if we can get SSH into it that all should work so I'm gonna come back onto my compute node where I have my network namespace here and we can come up here at our network namespace and let's go in here and I don't remember what IP that machine got so let's go look on the screen right here it said it got a 10.2.0.3 so let's go ahead and see if we can ping it I think that's what I said that's the right IP address and guess what host unreachable I can't get through to it what's that so from my inside my namespace it says it's not working right I can again look at the IP address because it's why it's it's saying from 10.0.2. and that's the interface of the tap interface is sitting at 10.0.2 so let's look at here so we were trying to ping this device here let's go on this one let's try to ping 10.2 did I get the right address 0.2 let's see if it can ping the other end and it's saying this network is unreachable so let's see what happened here IPA well let's try I know what happened that UDHCP didn't really assign it so let's do it if up to E0 and there now we get the route said let's try ping it again and sure enough so we up the interface it got an IP and we get our ping through life's good okay so let's go ahead and try SSH into our service image here and see what happens here so if you remember we can come back here let's SSH into it we know what the username is there anybody ever had that happen SSH is just hanging it's not dropping it's just hanging so what's our first way of troubleshooting it what's going on what okay we're all trying to come up with solutions rather than let's figure out what let's take a look and troubleshoot let's gather some more data before we know what's going on yeah I do know what's going on but uh... let's gather some data all good suggestions well there's SSH provides us a nice little flag called dash V that will show us what's going on the SSH protocol now notice we got all these things the last thing it says it sent an SSH uh... key there exchange is a key exchange is expecting one back now if I go look at it I will go actually see I'm not gonna take the time right now because we're gonna run out of time today so we don't run out of time if I do TCP dump on these things I would see packets on port 22 going back and forth in fact we can verify that if things don't work really one of the first things we should do is to do a neutron and how many times have you ever done this you try to ping something and it's because uh... because you forgot to uh... you forgot to open a security group rule so let's see what we got our security group let's see if we have all our security group rules uh... yeah thank you you're you're trying to tell me I can't type and that's right okay let's try that I confess to that and if we look at this we can look in here you can see that there's an ICMP rule in there that's ingress and you can also see that there should be in at the very bottom rule is an ingress rule for port 22 so what was going on in SSH is not a security group rule well it turns out that we're using uh... what I'm doing something that I'm connecting the two boxes with a VXLAN interface but my OVS are talking to each other with GRE so I'm running GRE inside VXLAN talking to two so now we talk about encapsulation and packet header sizes one of the things that SSH doesn't like is that when it starts its key exchange it will not allow you to fragment packets on that key exchange bad things happen in fact it does exactly this it sits there and hangs and let me show you something here's another troubleshooting technique that you can use let's go back up into this ping that we ran here and there's another flag here let's do a dash c of two so we do a count there's a dash s and let me set the size at fourteen fifty let's set the packet size what that says is now ping is going to pad the normal ping packet with no extra data in there so it's a packet at fourteen fifty we've got a problem those aren't getting through but we already saw that if I take that size off and bring the size down to just a normal ping packet that goes through just fine I think we found the problem we have an MTU problem this is a classic MTU problem that you can run into in uh... neutron so let's go back up the size let's cut the packet size down to let's try let's just try fourteen hundred remember I got GRE inside OVS or fourteen hundred is still too big we need to figure out what size we can get through there so let's try thirteen eighty let's see if that goes through and sure enough thirteen eighty let's just go through and it turns out and why we have that going on is that we have fifty bytes of VXLan header and several bytes of GRE header and I know if you look at it people say it ranges anything from twenty four to bigger uh... if you figure it really if I look at it as about sixty four bytes of GRE plus fifty bytes that brings me down to about thirteen eighty six so I bet if I go to like thirteen eighty four will go through and thirteen eighty six I bet doesn't let's just experiment with that thirteen eighty four those don't go so it's somewhere in there but we know thirteen eighty did thirteen eighty two might go through may or may not they don't go through but we see thirteen eighty does so all these header packets wrap it make it too big so we have our SSH problem here because we're not setting the MTU down on our VM so open V-switch go ahead I just just just because I was I couldn't remember exactly where it worked and didn't I've done this before but I couldn't remember I knew it was somewhere around there so I was just picking some numbers just to play with it I could have done one I mean it'll take odd numbers I mean it I probably should have gone down a little further and gotten there quicker okay it turns out that we can set something in the DHCP if I do it if I look at my Etsy Neutron my DHCP config file I want to put DHCP agent if I page down into this thing I can set something in here and let's see here if I can find where I set it up by here see this line right here it says DNS mask config file we're using DNS mask as our DHCP server and neutrons typically there are some others you can use but the default one is to use that in this case I set a parameter so that the DNS mask config file is specified now if I go look at that file let's go look at that you can see as a DNS option force with 26 is the first and the second number is my MTU well it was set to give my VM of MTU of 1500 bytes and we already shown that 1500 ain't going to cut it is it so let's go ahead and edit that and I'm going to come up in here and we already saw that 1380 was the magic number so let's go 1380 let's that set that now to fix this we need to do a restart a neutron DHCP agent so now the DHCP agent will reset and configure that so now let's go back to my VM and do an IF down E0 because I need to go get a DHCP again so that I get my MTU set properly because you can you can see what my MTU was before was at 1500 so let's do an IF up and then do an IPA and notice what that now tells me that my MTU is just 1380 okay so let's come back in here and get out of where I did my SSH here I'll take out that DHCP and sure enough there I get my key so I can say yes and I can do a Cubs win, smiley face what did I do wrong, typed it wrong I can't type some days and sure enough I'm into my VM now my day and my boss will be much happier with me today right yeah I spent all last night troubleshooting some of these type problems so I've been through this this is a type of problem hey I've seen the comments and the things we've all been there we've all spent hours trying to get our OpenStack installed our VMs didn't get IPs or our networks didn't hopefully right here we've seen some ideas and some things with some tools that you can use to go troubleshoot your environments any questions on this thing these are things that you know basically you've got to understand we have plumbing and we have tools that allow us to look at the plumbing some allow us if you please like the thing the camera that go down inside the plumbing I'll get you just a second here some of us allow us to go around it and we have to think about our tools along with that not jump onto a conclusion fast enough until we've got enough evidence to find out what was going on I've also been culprit of that many many times where I see something and I think I know what the problem is and start tweaking things and oh that didn't fix it and I tweaked another thing and in about 30 seconds about 30 minutes I've tweaked so many knobs I can't ever get it back and I've now I thoroughly broke it so what could have been an hour or two job becomes a three-day job to get things back up and running I mean we all been there go ahead I'm sorry yeah if you're doing just VXLan in fact Neutron will typically do that for you if you're using VXLan it'll set the MTUs to 1450 but in this case I'm playing I played a little games in the way I built these machines normally I would have separate interfaces this is built just like if we could run these in a cloud they don't like the cloud doesn't like us run some of the types of packets so I hide everything inside of VXLan and I do my Neutron inside that so the cloud doesn't get all upset and start blocking things that I try doing and so that's why this is constructed that funny way but it also allows me to do a little more work in showing what MTU issues can happen inside it because if you use VXLan tunnels if you use GRE tunnels I'll guarantee you you're going to run into MTU problems in your world and you've got to know how to troubleshoot what to look for and then how to get it there's another question go ahead why was it down? I put it down I did that I did that ahead of time that was a setup in this case but I've had situations like that happen where the interface has been broke somehow I thought everything was working and something went down in my data path and it was just to simulate a real-life scenario of stuff I've run into in the past oh it does happen oh yeah oh yeah go ahead if you're doing load balancer as a service yeah you can and the same it's the same same technology in this case if you're doing load balancer service realize that now things are stuck inside a load balancer namespace so I may have to troubleshoot inside there but again TCP dump is your friend because it's if you know what packets need to be going on that's why I say understanding what actually is going on in your network what type of package you should be seeing that's also why having your laptop with a couple VMs and a working little open stack environment means then I can go simulate and see what type of traffic I should see then I can go into the broke case and see what kind of traffic I'm seeing yeah so so I really you know I strongly recommend that you keep a couple VMs around uh... allow you to do at least a simple open stack and you can bring up because if you get into problem things aren't working if you now can compare and see what works what should like when it works what should look like when it works you have a yes and no case you can compare so now you can have a better chance of troubleshooting so really that's a thing go ahead and I'll get you next other other tips really uh... in this case that the tips I gave you the techniques I typically use are exactly what I've done here as I start using TCP dump and looking at what's happening because I need to see what's going on the packets what packets are going where I had since I've done this enough I know what I should see so but and that's where some experience will help you but it just in the end I don't know of any short cut way of other than to just follow the packet you know it's the guy with the little the uh... piece under the walnut shells and mixing them around we're trying to look under all the walnut shells and figure out which one has got the packet and you just have to you have to file the packet around so filing it around until you it's it disappears something you can see it coming out we can see it coming out we're just filing around now typically I would have gone about this a different way I wouldn't have since I broke that and knew where it was broke I didn't jump straight on that typically I would have gone and look at the packet as immediately came into my network node to make sure it's getting on my network node I went inside the network namespace because I didn't want to jump on the answer too quick but but I would have gone that way typically typically I'll just go step by step as that packet goes to the network at some point I'll lose the packet and then I know okay what's what's broke in that area uh... in general I haven't had too many problems with IP table rules but yes they're there uh... and by the way I've got a couple of questions here I'll get to them one thing too there's two wonderful flags uh... that are useful in IP table rules that we can use and let me show you real quick here the difference in output because I think this uh... let me go over here and show you let's go back on that control the compute node if I do an IP IP tables dash l if you look at that you see all these wonderful rules that jump up there if I do a dash dash v option knows what v gives me you see I now have two things called bytes and packets those are real helpful and there's one other one dash dash line numbers can be helpful but the dash v option is really important because that would give me uh... I can go through there if you look at this one this rule right here said that I had two hundred sixty five packets or sixty seven hundred bytes I can see which rules are being hit you know the packets are being going through which rules so using that dash v option is really helpful it's one of the things we forget and I I forget and then all of a sudden about the time I get into it I go why didn't I do that and then I go back and do it but it'll help so question yeah I I in the case of Neutron you've got to set it up to use network namespaces don't try you don't try it the thing is if you try using Neutron without network namespaces then you have to make sure that your client addresses are all separate from anything else and you can all you're gonna have to have your client so any exact address ranges they can use and they can't be addresses around any of your machines or anyplace else and so you really you don't want to run Neutron without using network namespaces so okay okay I'd have to let's talk about afterwards we can get a little more deeply what's going on there and talk about that Vxlan or GRE I don't have a preference although I've seen that of late I see some of the networking companies at least in the Neutron world push Vxlan over GRE and so I see them wanting to do you know right now inside the software defined network space now it's one thing out into the as we go into the backplane world there those folks are pushing GRE in that world but in the SDN space I see that Vxlan is being pushed more heavily but but I from everything I see network performance wise there's probably a little difference the bigger issue is that you need to make sure your NIC cards support the technology so you can offload that on your NIC card as opposed to your host CPU doing that so and some of the newer NIC there's a number of newer NIC cards that support that so that you can you don't have to do it into CPU space you can do it in your NIC card space and that's what you'd really prefer I'm not saying there's not any differences but I'm saying that from what I can see they both they both work adequately the big difference is I think GRE has the ability to do maybe up to four gig worth of networks whereas Vxlan only has 16 million but I haven't seen anybody with a environment that 16 million is a problem so I think the bigger thing is what is your backbone people care about and the NIC cards that you're going to use on your compute nodes what do they support that would be the deciding factors well yeah Vxlan uses UDP that's correct but you see since your packets are all in typically TCP if you care about it your actual packet using TCP so it will resend it if it needs to so you don't need the inherent technology and I guess that could mean that in theory Vxlan might be a little faster since it's going through UDP portion of the stack instead of TCP portion of the stack I in practice I haven't seen any difference between the two but yeah yeah so but you know again I see that the networking folks of the world the Cisco's the brocades and some of those seem to be preferring Vxlan over GRE tunnels I haven't had anybody say why but I just see that just going on I can tell you why okay okay cool good and you get into another thing that people don't think about enough when they're building clouds just making sure that they have sufficient entropy for their VMs because when you run out of entropy for your VMs you create another problem in that now your packet sequence numbers can become predictable and can cause other issues so when you design your cloud you need to think about it the OpenStack security manual talks a little bit about that but making sure you have sufficient entropy on each of your compute nodes is it yeah yeah one last question then we're going to have to call it quits for the day well you know you could go to jumble frames there's nothing wrong with that I run I find the fact in our data centers our backbone people say no jumble frames you go ask them please I want no the answer is always no so in my world is I have to dial down now now if the world of your data center your backbone people allow you to do jumble frames that would be a perfectly acceptable and actually some preference to do that because if I can run my inner size packet at 1500 byte mtu obviously fewer packets and a little more efficiency so yeah okay I appreciate everybody's time here if you have any other questions I'll be hanging around a little bit afterwards I'll try to answer questions out of that appreciate your time and sorry we didn't get time to do hands-on but the network was up and down earlier and we've had some real serious problems and you'll never get this done