 Hello, hello everyone. I think we'll we'll get started My name is a when you javid I work for Ericsson I'm part of the Ericsson development unit and Part of the team that does the development and then product development introduction to different markets So I go around get the customer needs see what they're doing and then based on that bring back that feedback Try not only to improve the product, but the process as well My friend Constantine He couldn't make it the problem with being really good is that you're in demand and when you want to go to sessions like these And we wanted to drive him along Well, he had important more better things to do. So he's the one he's a he's also part of the development team and He's the one who knows almost everything about everything. I know it's a cliche to say that But if you talk about go programming if you talk about shell scripting it will talk about Linux You talk about you you go and talk to the guy and it can be ours before you actually leave the conversation and Then so here. Yeah, my name is so hail. I'm actually I'm a solution cloud solution architect in cloud platform Team in Ericsson. Yeah, so we have like I doing all the cloud presales and postal activities within Ericsson so Haley is responsible for not the region North America, so he works with the local customers and He'll be able to give you the local perspective and I work with the global customers So you'll be able to hear what the customers globally are doing and what specifically is being done here in here in North America So let's get right into it Our expectation is that people who are here. They have basic knowledge about open stack I mean we are all part of the community and we are learning as we go so you have basic knowledge of open stack, you know a bit about networking and Understand the architectural choices. I mean that's the most important part of it Open stack gives you a framework of how you want to or how you can deploy your system to be but it doesn't tell you exactly What you need to do so you have to figure out what you want how you want and put the pieces together So those are the choices that we are not going to go into the details We've picked up a scenario We picked up a couple of scenarios that we found were very common with our customers and we've tried to do the troubleshooting techniques around that And Then neutron details we put that there in the slide for the people who are new to open stack It's it's not what we are going into the details. So somebody there's I've been to Vancouver I've been to Paris every time there's a lot of people who are just beginning to look into open stack So anyone who's new or is getting his hands dirty now with open stack these slides are for for them basically So after the presentation they have had a chance to look at the details and So this is where this is where we want to get started When we talk about cloud troubleshooting and it's not specifically neutron troubleshooting It's cloud troubleshooting when we talk about that kind of troubleshooting There's certain set of tools that that you absolutely need not only you need to know these tools You need to know a little bit of details about these tools. So we put them together and This could be a good slide. I mean, I'm pretty sure most of us here know all these tools We'll give a few examples about ARP for example Everybody knows ARP entry, right? But if you're doing testing with with a VM That's not able to talk to a namespace and you want to see the ARP cache and you want to clear the ARP cache Then you do dash the option and clears it out things like that ping with a count option or ping with the Different size of the packet so if you're doing troubleshooting and you see that the certain packets are going through and others are not And you see that oh my ping is working just fine But when I start the TCP session, I realized that certain Packets in the beginning of the TCP handshake work, but when the payload starts flowing it doesn't work So how do you troubleshoot that so pink would be a pretty good tool there So you can select a packet size and see if the empty of size in all the segments where your packet is flowing Could could actually be tested through that Arping that that's a very interesting tool when we were putting this tool kit together We we install the tool because we Our approach is to start from the ground up if the layer 2 is fine layer 1 is fine And that's a given we expect that if layer 1 is fine Then the second step is layer 2 to test layer 2 I didn't know about this tool or ping and Constantine it reduced me to that So we install that on our controller on our network node as soon as we install that we are seeing the packages that are being installed And we get this funny message L3 agent will be deleted like what? So that that was interesting so something to watch out for make sure that you don't install it where you're running the L3 agent TCP dump I think that's by far one of the one of the most useful tools that we have come across There are limitations with TCP dump. So for example, if you're using DPDK or if you're using SRV Then TCP dump doesn't really cut or doesn't really do what you what you would want it to do So there there are ways around it and maybe we can talk a bit about that as well IPA bridges OVS That's that and IP tables. So let's let's go a bit into details so bridges We we've seen deployments where You have an option of bypassing the Linux bridge provided by the OpenV switch So let me go a little bit here. So yeah, maybe I can talk about this and then I'll connect the Connect the bridges. So if you look at this particular diagram, you see this part right there This is the bridge, right? This is the Linux bridge and when I started looking into open stack I I was asking people what why the hell do you need that particular bridge there? The more legs that you add to a flow the slower the flow becomes, you know I mean if you want to go to Paris, you're not going to go to New York and Montreal and you want to go to Paris you go to Paris, right? so the reason behind this bridge is the security and That's why when we go back, this is so you need to know a bit about Linux bridges as well And then how the Linux bridges will connect to Connect to the IP tables OBS tools so OBS as far as I know is most widely deployed in commercial And if you want to know about open stack networking OBS is your best friend And when we talk about the obvious flows there are the data path and Open control flows if you know about these two I I think you'll be okay, but if you've not started looking into that. I think it's a good time to do that So we'll we'll talk about how the flow actually goes We'll we'll we have the we have a diagram that that will show the packet going from one place to another to another And then we'll discuss how you can actually trace that packet And in the end we have a surprise for you because we'll tell you how the packet flows But that's not how the pros do it. There's a different way to go about it and we'll leave it to the end Debug step. Yeah And okay, so let's let's go right into the common architecture and That's where I was saying that we will not go into the details of how you have deployed your open stack You know, we are assuming that this is the most commonly deployed architecture And we'll we'll keep that as a reference And we'll discuss how you can troubleshoot this There's there's about nine steps in this When a packet flows from the VM to the namespace or when it goes externally more or less line If you can troubleshoot this or if you can find out about these points You can apply the same scenario or you can apply the same concepts to almost anything else so we'll go through this diagram and Here what we are showing is that you've got two compute nodes Compute compute one and compute two and then you've got a controller node or you can call it a network node Whatever where we're basically your DHCP agent is running your L3 agent is running and Typically speaking your VM will be connected to a tap port That tap will be connected to a Linux bridge and this is where the security groups will be applied As I was saying earlier, I mean, there's a choice for you to bypass that So if you're a private cloud or if you have trusted applications for one reason or another I mean, there's un-uncountable options You do have the option to bypass this and hook up your tap directly to be our int But this is for the security and this is the typical deployment that we've seen After that you've got to be eth pair and then you've got the integration bridge The integration bridge actually is a smart bridge that will do the internal switching and we'll go more into details about that and Then the integration bridge connects to your physical port the physical port connects to a layer to switch and We are seeing a lot a lot of the customers They they want their switches to be controlled by STN and we actually do provide a solution where You don't actually when you create a network the network the VLAN the corresponding ports on the compute They get pushed directly onto the switches, which is a really nice thing because everything becomes automated you script it The drawback however is if something feels it also adds another thing that you need to troubleshoot So when you talk about the layer to switches Just keep in mind whatever company whatever switch you have port mirroring on the switch could also be a best friend Especially when you're doing troubleshooting with the SROV and you're trying to prove that the packet is leaving compute TCP dumb is not going to help you there OBS is not going to help you there port mirroring could be one option and For particular vendors I've looked into the Cisco's the Juniper's the extremes and they all support port mirroring and So basically that's vendor dependent and we don't go into those details So when we get started most of the deployments they they use DNS mask So let's pick up a let's pick up a very simple scenario in which your DNA DNS mask Process is running on your controller node on your network node and your VM wants to connect to it to get the IP address So as soon as you run your DNS mask process make sure that you go into your name space and Make sure that the process is running. There's a lot of times especially in the beginning I was trying to find out. Oh, wait. Why is why is why is it not listening on port 67? Long behold. Yeah, you have to go in into the name space back in the days That and then each of the process whenever you create a subnet and DNS mask Make sure that you have the corresponding process running in your host So With that now once you've made sure that your DNS process is running Then we go right into the use case where your VM you boot a VM and you want to dynamically allocate the IP to the VM The VM comes up Nova boots it neutron connects the ports everything is so so far so good Then the then the VM sends the boot be request it goes to the it traverses through all these ports The ports that we've talked about before and then it finally makes all its way to the to the process DNS mask DNS mask replies back. Okay. Here's your offer. This is the IP you can get Long behold if everything works fine your your VM will be able to get the IP address But this right here is the crux of it if you can master how to troubleshoot this path It's it's more or less the same for everything else or more options than not The only difference or the slight difference that you can see or another option that you can see perhaps is When your VM tries to go to the external network So then there's one more layer that you need to look at and we have that case covered as well So we'll go into that and we'll tell you what to look for and where the problem can be So let's go right into the into the OBS flows Because as I was trying to say so what all these points the places where we have numbers These are the places where you can run a TCP dump to make sure that your packet is being received or not If your packet is not received then you need to troubleshoot either one layer above it or one layer below it You can run TCP dump to do that and that can be a good starting point But that's not what we suggest so to start with a simple case if your VM is actually not getting the IP that it needs Then you need to troubleshoot instead of running the TCP dump on all these points Maybe start with something simple. Maybe dump the kernel. Maybe don't dump the data path So on the compute where you're running your VM You can see actually what the hell is going on. Is the VM really sending something out If the VM is sending something out, is it going on the right port? Is it using the right VLAN things like that? So if you can decipher this this message right there Then you're pretty good. You're in good shape to find out. Okay Or you have a good clue to know where the next problem could be instead of going back and tracing Nine different points, you know, which can be overwhelming could be waste of time and it's not efficient as well so you run this and Basically, okay, so let let's go through this quickly Um, it would have been nice and I think this is what we'll do for the next time We will take a account of ants to see how many people are familiar with this and then we'll adjust to Adjust the explanation To to see how much depth we need to go in But for now, I assume that most of you know a little bit a little bit about this flow So I'll go through this quickly and if you have more questions, then we will we can take the questions towards the end And we can explain it a bit better So what what you're looking at here right now is all these ports So these are the ports which which exist within your within your host, right? But ovs doesn't match map these ports ovs has a different way of mapping these ports. So For example, eith to that you see This this port right here Which is connected to the external switch This is mapped in ovs as port 4 qvo v2c Which which is right here This is mapped in ovo in ovs as port 8 So when you when you dump the flows what you're actually looking for is Am I receiving packet in port 8? So port 8 will correspond to this If you're receiving packet there, it means that the vm is trying to send something out The next thing you check is the source MAC address So when you create the neutron port or when you when you try to boot the vm This is the MAC address that that the vm will get you can check it from From either DHCP lease and those logs are present in your controller logs And I can point you to where exactly they are You can look at the MAC address there and make sure that the vm is using the correct MAC Because this is what's been established and pushed forward And then we see that okay, so it is trying to send a broadcast Very good so far so good And the next thing you're looking for is that The the flow will enforce to push vlan 2020 So when you do neutron net create, this is the vlan that will be pushed and configured on the external switches So this this path right here is using vlan 2020 So you make sure that yeah, okay the correct vlan will be tagged onto this frame And the frame will be sent to port four Port four as we were talking port three and four my apologies So port three and four they correspond to this part right there So just looking at this line tells you that you know what okay My problem most probably is not within the compute I see that the vm is trying to send the packet out and it's sending it with the right parameters That's the most important thing So Um, and once you've established that then the next step becomes Then okay, so if that's correct then the most probably the problem is on the other side And you need to start troubleshooting on the other side on the other side you do the exact same thing You dump the flows and you see if the packet is received in the correct port With the right source MAC address with the right destination MAC address and the vlands are post correctly We've seen many times that OBS is a pretty stable tool for the most part But we've seen a lot of times that if you have a funky configuration It will show that the packet is being received in the correct port But it will put some some strange vlams on top of that You restart the process and you see if the configuration is as per the as per the guidelines and things start working well But if you don't want to Go into the details of the configuration files you want to find out right away If there could be a problem with the vlan or the mac you look at the flow and you get a pretty good idea of Where the problem could be So this is the screenshot of the same thing that i explained earlier We are making sure that the ports Action means that this will be sent out to this port import means that the packet is received on on this particular port And the rest of the details are the same Now after you've dumped the after after you've dumped the flows You can run tcp dump as well And tcp dump here we are running on qbr. So qbr is right there. This is the bridge And this bridge will tell you okay, so what is the flow of the packets, you know, am I receiving the packets correctly or not as well So that that could be the second thing And then tcp dump options As i was saying tcp dump is your best friend and the more options the more you learn tcp the better because No, that that's okay I was hoping that we we don't look very intimidating, you know, so you can come to the front And i'm pretty sure that we You know, we'll we'll share some of our experiences But i'm pretty sure we'll pick a few things from from everybody here as well So it's it's it's a two-way stream. We give some we take some, you know, so Maybe it's a wake-up call as well, you know for the people who are dozing us Okay, so we're pretty subtle So yeah going back to the tcp, uh tcp dump is or could be your best friend the more options, you know, the better In the beginning when we were running tcp dump we'll do tcp dump on particular interface And then we'll see we'll dump all the packets But when when we started working more in production realize you can't really do that, you know You've got 10 gig links and you've got multiple vms You've got a whole lot of thing going on in that particular mix So you don't want to run just tcp dump on that interface You want to narrow it down to as much detail as you can You can do it on the mac you can do on the source ip destination ip source mac destination mac ports and whatever not So make sure that you if you're troubleshooting cloud, you know tcp dump on on the back of your hand, you know And uh, we'll um towards the end as as i was saying we'll give you some more examples and we'll tell you how we do it Here the important thing is again to to make sure that you you see that the mac addresses are Okay, the mac addresses are correct And we are doing the troubles we did this tcp dump on on dhcp port because that's what we're troubleshooting tcp dump in the namespace So Again this slide is for the people who are relatively new to to open stack I mean, how do you find the namespace the name of the namespace? How do you connect the dots and actually finally go into the namespace? But one thing that i'd we'd like to point out is this ln option tcp dump is awesome But like anything else it's not perfect you put the ln option and it becomes a lot better So it doesn't buffer the packets and you start seeing the packet flow right there You don't have it doesn't overflow the buffer So that's uh, that's one recommendation or that could be one takeaway If if you don't take anything else away from the presentation take the ln option and i'm sure you you'll buy me a coffee next time we meet So for going back again to the to the to the vm not getting the the ip address If your vm is not getting the ip address One option could be that you configure the log into the console of the vm and give it the ip manually Once you give the ip to the vm manually that that opens a bigger box of troubleshooting Because if you don't give it the ip have it you can still log on and then you have to Again and again execute the dhcp request send the dhcp request So if you give it the ip manually now you're in control you can send any packet you want whenever you want So what we found out Is that if you give it the ip then you can start the ping And you can stop the ping whenever you want and you can create the flows and look at the flows Instead of restarting the interface or restarting the dhcp to to get the ip that that could be a good option as well and something to keep in mind vm to vm flow the only thing It's more or less the same Where basically you'll still see the packets on the points that we've highlighted here But the difference is the packet is not going out You know the packet is actually being switched from here to here It doesn't even hit vr end the art will but everything else goes directly between the ports Yeah Okay, so in this scenario actually we have a three use cases so which we will explain you how you can troubleshoot and which part of the port or The obvious which you can troubleshoot the first first scenario is our vm vm to vm flow The both both of both of the vm are on the same compute Actually our scenario is we have a one compute and the one controller and the controller has also compute node So in this scenario you can see like we have a six steps Like the packet flows from when the vm one initiate the ping request so You can see the tap interfaces which connects to the bridge interfaces and then the other one Let me go through the other scenario and then I will explain you each each step with the screenshot like which I did And the second scenario is when our vm is on the other compute node So in this case we cover like if we have a Another physical host machine and then how the packet will flow from one machine to another machine And the third one is like if you want to connect to the internet external interface on the switch side So so let's go all these steps one by one actually We have Okay, can you see now? It's good Okay, so Okay, so the first scenario is vm to vm flow on the same host So first you need to check on the control controller node that your vm ip So nova list if you will know the vm ip So if you see the port if you want to know the port list of that vm So just to neutron port list grab and the ip address the important thing of this command is You have to take that 11 A character from that port list because that 11 character will be used for your tap interfaces Your qbr interfaces and your qbo interfaces just like I showed you here. So if you see here So you you'll see that this Tap interface these 11 characters, right? And the qbr is 11 characters and the qbo and the qbo these all all 11 characters come from that port id which we Did from a neutron port list? So the same thing is for the other vm So now the next step is like you have to see the instance which where that instance is running Of course, we have only one we one computer node our both instances are running on the same computer node When we do worst list you can see your instances are instances 1d and the instance 1e So if you do br ctl command, which is a bridge command you you can also you can see the which that bridge in bridge Connected to which port that connected to qbr and the tap interfaces. Yeah Yeah, just a quick interruptions. Sorry about that. You will take this document with you We'll we'll give you a link. I know we are going a bit fast, but we have a lot of to cover When you go back to your environment, you follow the rules monkey see monkey do kind of thing, you know And you then you start to realize that yeah, this could be a systematic approach how you can go about and collect the information Okay, sorry So now, uh, this is the first scenario vm to vm flow on the same host Let me go. Okay. So now you can also check that 11 character if you go to your computer And if you grab on that 11 character, you can see that all the bridges and port are connected All the bridges have been created. So that's the one step Other one like if you go to the vm, you can see the mac address is just for your reference like Whenever you troubleshoot you you you would know that which mac address you are using here So now let's start from the ping. So I'm pinging I'm trying to ping from vm one which is a 1.4 and to vm Two which is 1.5 so Once your ping is you can do tcp dump on all those hopes like tap interfaces on qbr interfaces qvb and the qvo interfaces You can see the request goes to 1.4 to 1.5 and then we have a eco reply everything. Yeah And then on the obvious side as say you explained like when you do the dump of obvious And you can see the see the port number is of five and three you can see here Ovs like is packet is Going from port number five, which is this qvo bridge and And going towards a six, which is a qvo on the another vm. Yeah, just like this so I mean back when we do ping packet is going from this qvo port to with port to this qvo port because that's the internal so you cannot like means Troubleshoot more on this. Yeah, so only this one this way obvious dps dpctl You can see that the ports are like the your packets are flowing from one port to another port And another another vm you can also see the same tcp dump on all those ports Yeah, so that's the one if such scenario the second one is like if your vm is on another compute node on in our case Our controller is our compute node. So all the steps will be same. Everything is same Only difference is when we do the tc obvious dpctl show here you can see the packet is like Going towards the eith to which is our physical interface of that compute node. Yeah and You can see that here is that push the wheel and of 2020 Because that's a segmentation id which we put in in our like The Network and then the same id we will have in our switch and then if you think we have some problems still it's not reaching You can also troubleshoot port mirroring on our switch. Yeah on some switch any Okay on other side of the compute where the other vm is exist When you do obvious dpctl show you can see that port is coming from the eith to Which is the physical interfaces and then comes to this Qvb qvo port which is which attach to the vm2 interfaces. Yeah And then also it Popped the wheel and because it will pop the wheel and and then send the packet to outside of the That bridge interface This is a scenario you can see here So everything is same here So let me go to the last step last scenario Okay So the last one is a vm2 external flow if you if you want to ping or like troubleshoot your vm like from the from the external side Yeah, so in this scenario a Most mostly mostly is everything is same when we Troubleshoot from the external side like when the your packet reached to the namespace or the router namespace how you can troubleshoot it So here you can see obvious dpctl show it's it's Packet is receiving from the eith to which is the physical interfaces of that vm And then it's sending packet to the qr which is a qr port attached to that namespace router. Yeah So here is qr 59863 So if you want to see more information about that namespace if you do ip net ns on that or of your Second like external nor or controller nor which is the network where the network is running newton is running So you can take this 11 character same 11 character from The neutron port and then you can search this 11 character inside this net ip net nm net ns command So you can see this q you will have a qr Port created inside that newton router. Yeah And then when you do tcp dump on On this qr router You can see here like because we we ping from we ping from vm1 to external interfaces, which is a public ip You can see the translate here like it's translating the ip like mean it's pinging the ip External ip so how the translate works is the net interface in the on the ip table So the net how the netting is work as net so if you See inside this ip net ns ip tables you can see the s net is like transfer like Translating your like public ip a private ip into the public ip. Yeah Okay, now your packet is on the qr now from the if you do again on the ob ctl again on on your like With that mac address you can see that your packet is moving from your qvr qr router to q q g port Yeah, so in this scenario. Let me show you here here So your packet is like here So now it's if you see ove ove sdp ctl here You can see that your packet is from qg to moving is so external interfaces and your bridge or like switch router. Yeah So now let's we have Actually, that's how like everyone does it So now we will show you how the like professional to it Yeah, so yeah We will give it light. So something yeah So yeah, so yeah, so he'll explain different kinds of flows and you know Then we will share this document with you using this document Monkey see monkey do kind of approach you can list everything it will show you the ports It will list the mac addresses and then you can manually connect these things together, right? And if you do it manually then yeah, not only it takes time But it's cumbersome and when you have to do it over and over again, it becomes pretty boring I mean there's no fun in it anymore So instead of doing that we we started collecting scenarios that that actually fail And based on those failing scenarios Be based on those Yeah based on those most common scenarios that that fail what we did was we created A little script for us we we call it the magic genie inside in our in our shop But maybe we we share a bit of it and it's not really clean yet because I mean it's for our internal use So we do plan to clean it and then share it with everyone else You can use it as for your own reference for your own architecture add things to it And you know do whatever needs to be done So let's uh, let's take right into it. So if you do know our list Currently, uh, we've got six VMs running There's two different networks and So because of the lack of time I I would have wanted to create a live network a new VM I I don't think we have time for that. So I'll use an existing VM and then it would have been nicer actually to create A new network in me, but let's Let's run with it and see how far we can go So for example, if if I have vm one or any other VM Could be vm one to vm six and I'm having network troubles My networks are not working fine. I want my script. I want my magic genie To just take the vm and then get everything that's connected that vm whatever network It's connected to whatever tap ports qbr qvo Whatever the ip rules are wherever tags are connected. I want Everything to be done automatically. I don't want to go into the details and then connecting the dots together So let's let's run with it Yep, that's That doesn't look pretty does it Let's try to do it again. See if we can do it better this time Okay, so it asks you for For a vm name you can give it a vm name or a vm u u id if there's multiple it will ask you Okay, please give me a unique id if there You know, I mean nothing can be made stupid free. So You have to be For the lack of a better word So, um, the script goes in takes a vm and then it starts collecting everything around it So it's telling me, okay, so this is my private ip. This is my associated floating ip Aha So it finds out that right now vm is not being able from the namespace It says, okay, I'll try to find the problem Vm is connected to these ports. So it starts the back at capture So remember the ports that we were talking about the the ovs ports Yeah Aha, okay So our request stands successfully, but the our reply is not received problem could be on the top of the rack switch So just just to quickly go over this and then we'll fix the problem and see what will be the outcome So basically I give it a vm name and then everything that so he'll talk to you before How you collect the information the 11 characters of the vm ports the ip addresses the qvr qbo The the script does everything for you if and then it tests the connectivity towards from the namespace towards the vm It can be other way as well If if you own the vm, of course So, uh, yeah, it runs the connectivity towards the vm and it finds an error message Now when it finds the error message it will not just leave it there. It's not going to hang out We want the script to tell us where could be the problem, you know Instead of us doing the tcp dump and all kind of funky software nine different flow We want the script to be smart enough to actually at least point us to the direction where the problem could be So it's it starts the packet capture It runs the packet capture and then it dumps the packets in a file It analyzes the file on what is the expected behavior The vlans the max The ingress port the egress port everything because if you if you think about it all this information is already there All you need to do is Make a program which is smart enough to do what a human will do So that's that's what we've done and it collects that information and makes sure that the flows are okay Now If we can go ahead and Let's see 30 34 So they've enabled that port Yeah, so that basically creates a broadcast storm And they should not be here They are in different vlan The physical if config port was okay. Yeah, just connect to a vm one So we will connect to the vm one Because he's saying that it's would be a tor switch port problem or the compute switch problem might be the port in the computer So so he'll keeps me. He keeps me on my toes Um, I go in and create problems for him and challenge him to fix it and he does the same thing Now he didn't tell me that this will be a surprise because he'll do it live when I'm doing the the demo Thank you. So yeah But he did save me the embarrassment by you know not I could have panicked there. So so okay, you know and I'll yeah, yeah, okay, so Did we put it down here or Okay, I told the guys not to mess up with this what? Yeah, that's a fun part of sharing the environment as well So I know the tap interface and okay Let's run with it again. Let's see if it really is a problem Oh Let's see what we am three We were playing with it last night and I think that's one of the worst things you can do, you know, right right before you are going in and doing the demo Okay So, yeah same story basically it finds out this is the tap. Okay Packet capture successful. It sends the ping and the ping is successful as well Okay, good stuff. So what if we did this? Okay, so now we've we've disconnected the VMs from the namespace Let's let's run the same thing and it should be able to find out that there's well it it needs to add must find out Um, it will find out where the problem is and then we'll be able to go and then fix the problem But not only this but it also works with the With the namespaces it it looks at your natting. It does a whole lot of things. We've stripped. Yeah, so there you go right there It's not pingable. I'll try to find the problem for you. Then it starts dumping. Okay. This is the network So you see 2020. This is the external network which is connected on your top of the rack switch The internal vlan id that's mapped to it is vlan one The the ports Are ports seven and five It starts the packet capture So analysis checking if the our request uh-huh Okay, so the our request was sent out successfully, but it doesn't receive anything back So that's that that's basically it We've got this with a lot of other scenarios covered But unfortunately, we don't have time to go into that. I promise next time if we do this demo We we're going to cover a whole lot scenario not just this but we'll take it one notch up We also wanted to cover dpdk and srv, but we didn't know about the about the level So next time when we do this around in barcelona, hopefully see you all there We will talk a lot about dpdk how to troubleshoot that and how to troubleshoot srv That that's been one of the fun the most fun part that uh, that we've uh, that we've seen It's not very well covered. I think there's a lot of work that needs to go out there Yeah, we have a link to download the troubleshooting document All the steps here. Yeah. Thank you very much for your time. If you have any questions, we'll be Thank you