 Thank you very much and you are now a host again. Thank you. And you can see my screen still, right? Yes. Sorry for the introduction. No, no problem. So, getting back to the presentation. L3 agent, that's another big difference between ML2 OEF or Linux breach and ML2 OEN. Backends, in case of ML2 OEF, there was a neutral and L3 agent run on controller or networker nodes if you have separate networker nodes. And sometimes if you are using DGR, then L3 agent is also run on each compute node to do distributed virtual data. But that's only the case with DGR and not always. Usually when you are using, for example, L3HA, a neutral and L3 agent with some networker nodes only. In case of ML2 OEF, there is no L3 agent. L3 staff is handled by OEF and OEF logical order. FNAT traffic is centralized still on the gateway traffic and the floating AP, so DNAT is distributed by default. So you have DGR by default for all floating APs. And DHCP is another big difference between those two backends. So in case of ML2 OEF, again, there is nodes on DHCP agent. It's typically it's run on networker or controller node. And it has a configuration for many networks and that there is a lot of problems when you have a lot of networks, when you want to, for example, I start this agent, it takes agents to synchronize everything and so on. In case of ML2 OEF, again, there is no any agent for that DHCP service is provided by OVN, like OVN DHCP options feature. So basically that's the draft entry in OVN database and OVN controller will respond to locally. It's also fully distributed like metadata and DHCP request will be applied locally on the compute nodes always for VM which is asking for this. And yeah, mapping between nodes on the source set and OVN because nodes on have got a lot of different the source set which you can create and manage. And in fact, OVN has got the same or almost the same, but it's similar and we are doing some kind of mapping and knowing about this mapping may be very useful when you will try to debug or check something in OVN. So basically port in Neutron is logical switch port or in case of VM port or logical router port in case of ports which are plugged to the router in Neutron. Network in Neutron is something called logical switch in the OVN router is from Neutron is logical router and then subnet as I said is DHCP options. Security group is represented in OVN by port something called port group and it has ACL rules which are basically security group rules. And then floating IP is something called NAT in OVN and NAT can have two, one of the two types. One is D NAT and S NAT type and that's floating IP and the other type of NAT resource in OVN is S NAT and that's the gateway port for the router. If you have external gateway, that will be NAT with S NAT type, basically. But let's try to now to do some basic exercises. You can run your VMs. I will, in this part I will share, I will still show slides with examples and you can run the same comments on your VMs to see how it is for you and your desktop environment. If you have any questions, please just ask. I will try to answer them. As we will go through those exercises. So let's, those exercises are very, very basic and I just want to show you those mappings and how things in Neutron are really represented in OVN and how to find those things and so on. So let's first start with network. Of course, every one of you can do, know how to do, how to list networks in Neutron so you can do OpenStack Network List. And you will have output with network, with some IDs and some names and that's it. And now from the central node, that's maybe from the beginning, there is to interact with Northbound, OVN and Northbound DB, there is OVN and BCTL tool. It's pretty similar to, for example, OVS, VSTL tool, which you are probably familiar with. But this time, this NBCTL tool allows you to query Northbound or interact with OVN and Northbound database. And the first command which you can do there is OVN, Northbound CTL, LSList, which is logical switch list. And if you do that, you will see your network, some IDs, which are IDs of the logical switches in the OVN, but in the parenthesis, you will have main called Neutron dash and some ID. And this ID is actually ID from the Neutron database. So you can easily, if you have Neutron ID, Neutron Network ID, you can easily find it, find logical switch for this network in OVN. And also you can do simply OVN and BCTL show and then this Neutron dash network ID. And you will get a list of the switch, list of the ports, which are connected to this network. Basically, each port represents, like port in Neutron. So VM or router port or things like that. Any questions or did you try to do that, was it okay? Maybe if you will do such exercise, maybe you can click on the yes icon in the participant list. I think that I will see how many of you actually did that so we will be able to move on or give you more minutes. What do you think? Robert, do you have any question? Okay, I think we can move on. So let's go to the next thing. Next one after the network is done, we have subnets in Neutron. And basically the subnet can be found in Norban database by doing command like on the slide. So OVN and BCTL find DHCP on this corruption. And you can set in external IDs, you can specify subnet ID. The subnet ID is the idea of the subnet in the Neutron basically. So again, it should be pretty easy to find it, to find in OVN something from the Neutron. In this DHCP options, I think everything should be pretty easy to understand. So basically you have some external IDs, which is the subnet ID in this case. You have the CIDR set and some options for DHCP like classes, static routes, DNS servers, NTU and other things, which in case of ML2 at the end, of course were configurating the DNS mask configuration. Also, you can easily find all ports, which are plugged to the same, which are using the same subnet, by doing command like the bottom one on this slide. So OVN and BCTL find logical switch port. DHCP is for options equal and you can specify here your ID of the DHCP options from the command. So you can find the DHCP options for a specific subnet from Neutron and then find in OVN all ports, logical switch ports, which have IP addresses from the same, you can try the same on your VMs now. I think that this should be pretty easy to do and I think we can move on. If it's too fast or you have any questions, please just ask on the chat or just here. So next thing is in Neutron is port, of course. So you may have ports, which are used by VMs and routers. First of all, let's check the ports, which are used by VMs. So we have, you can do OpenStack port list for that device ID and then some VM ID. So you will have ID of the port, which is used by one of the VMs in the OpenStack. And if you want to find this port in Neutron, in OVN northbound database, you can do OVN NBCTL show, which basically will show you everything, all logical switches and logical routers and other things. It's kind of like OVS VSTL show command. And you can look for logical switch there or for this port there. You can, if you know the network ID, you can also do OVN NBCTL show to get this one logical switch, as was shown on one of the previous slides related to network. And you will get OVN logical port, which is the port from the Neutron. You will get IP address, MAC address from this port, it's the same like in Neutron database, and you will have the same ID of the port as in Neutron. And in the parenthesis, there will be always name of the port, even also name of the port specified in the Neutron. Actually, so you can again pretty easily find things which are find things from Neutron database in the OVN database to investigate and check what's configured there or how it looks like. And the next thing is router actually in Neutron. And routers in Neutron, as I said, are mapped in OVN database like as a logical router switch. So again, it's very similar to networks, like there was logical switch, now there is logical router. So if you do open stack router list, to list router in Neutron, you will get some router, which is created there, and then you can do OVN northbound CTL, LR dash list, which is logical router list. To list those routers, logical routers in OVN, there will be some entry mapped to the Neutron router with Neutron dash and ID of the router there. So you can easily find it. You can also check if command like OVN and CTL show and then Neutron dash and ID of the router that you work to. So I didn't add this to the slides, you can check that. But basically it should be like the same like for logical switch and for the networks. So that's the logical router, which is router in Neutron. Then if you check ports from this logical, from this router in Neutron, in our case there is three ports in the router because we have external gateways and two ports for two different subnets, two different networks, and you can do OVN northbound and CTL show to get this information about this logical router and to list ports which are in this router. And actually they will be the same like in Neutron. So we have three ports here with names like LRP, dash and some ID. This ID is the same ID as ID of the port in Neutron again. And there is information like MAC address of the port, the same as Neutron and network, the same as the subnet in Neutron. And the IP address is the same as in Neutron actually. In case of gateway port, there is additional information which is gateway chassis. So as you remember from the beginning, chassis in OVN is basically host in the open stack world. I would say so. In our case, in this case, there is only one chassis because we have only one node with external gateway configured in our lab. But if you will have for example, three network nodes, three gateway nodes, you may have three chassis here and you don't really know which one is active in such case. But by doing the command which is given below, OVN and CTL, LRP, get gateway chassis, you may get list of the chassis for that router with some priorities and the highest one is the active one. And also if you have this gateway chassis ID, you can query OVN south band database with OVN SBCTL show command for example. And then you will find this chassis there and you will find hostname. So you can easily identify which, actually which host it is. On which host is this chassis and where to go to do some other investigation in debugging what's going on there. So that's basically this. You can of course try to check those things in your lab so maybe I will wait a minute or two. Okay, let's move on. I hope you will check that and everything works fine for you. So next thing is Neutron router gateway which as I said, we have in Neutron, you know we have this gateway and that represented, I said here, Mark said floating IPs. No gateway. So, okay, so for if we have router with one gateway port and two subnet, two private subnet, private network connected to it, we will have two in the logical router in OVN, we will have two NAT entries with type S NAT, which means that it is only for the gateway so traffic can go out with this. And we will not OVN output from OVN and BCTL show command. For example, you will see the NAT entries with external IP, which is IP address by external gateway in Neutron and the logical IP which is a seeder of the subnet from Neutron. So you can easily check that you have entry for in the router, you have entry to do S NAT from the logical IP from subnet in Neutron to some external IP, which is a gateway in Neutron. And pretty similar thing, so if you have any floating IPs configured in the router, you will see also in the same output additional NAT entries, which will have type D NAT and S NAT. And that means that if floating IP, then logical IP is 6 IP address from the Neutron database, so basically IP address assigned to VM. And external IP is this floating IP from the Neutron world. And basically that's all. So you can find easily any D NAT and S NAT entries, which are basically floating IPs from Neutron. And one last thing about floating IPs and this NAT entries in OVN, you can find for example all floating IPs which you have configured in Neutron by doing OVN and BCTL find the NAT and then specify type D NAT and S NAT. That will list you all of the NAT entries. And one important thing here, you can also check that in your VESPAC hands and Daniel will also talk about it in the troubleshooting part. So there are two attributes here, external MAC and logical port. If both of them are set as like on the slide, then it means that floating IP is distributed and traffic will go directly to the compute node and from the compute node. And if one of, at least one of those fields is empty, then floating IP is not distributed and traffic will go through the gateway traffic. So that's the easy way to quickly check if floating IP, specific floating IP in your environment works as distributed or centralized. Next one are security groups in Neutron and as I said, those are represented by port groups. In the OVN, you can again list all port groups by using OVN and BCTL command like on the slide. Then you will get some groups with some ACLs, so rules from the... So things which represent security group rules from Neutron and then there is external IDs where you can find security group ID from Neutron. So you can know that this port group is representing the specific security group from Neutron and there are ports also. So those are IDs of the logical switch port from the OVN. Logical switch port from the OVN, which are using this port group. So the port which are using specific security group in Neutron. So you can also check that on your desktop. And the next, the last thing which I wanted to show you is security group rule. As I said, it is OVN ACL, which is listed in port group. ID of those ACLs are listed in port group, but you can also list ACLs for specific port group. And the output of this should be pretty easy to understand. There are rules too. And for incoming and outgoing traffic and there is some much action which is exactly the same, actually like something defined in Neutron. So that's easy to find out and to understand what's going on there. And there is also some more kind of complicated, longer command at the bottom of the slide where you can find all ACLs in OVN northbound database just by knowing Neutron security group ID. And that will give you exactly the same output like it's above on the slide. You can try that out later. I will maybe take those commands in the hr path later so you can copy paste that. I should do that before, but actually I forgot. So that's basically everything what I have for this basic introduction to OVN and now I will show my console and Daniel will take over and show you some typical failures and how to investigate and how to feed them. Thank you. Now I can see the screen. Should I make it bigger maybe? I don't know. I can't read it. Okay, but all right. So, yeah, thanks, Lavek. Just a super quick introduction on my side. My name is Daniel Alvarez. For those who don't know me, I've been contributing to networking OVN code for the past two or three years. So I'm going to just show a couple of what we think is typical troubleshooting exercises. Please feel free to interrupt me and mute yourselves, ask any questions. I want it to be as interactive as possible. So if you think that I go too fast or something is not really clear, just stop me whenever and we will go through it. So for those, I'm sorry, first, because we have been sending patches to these background file projects. So for those of you who have deployed the thing before today, you're probably missing these files here with the failure scripts. So if you don't have them, just pull the latest bits from the repo. We can do it here together. So if you don't have them, you should be able to find them here. So let's go with the failure number one. So just super quickly to see that this is what you are supposed to have in your current lab. We have made some changes to make sure that certain machines are put in the worker one and certain others have been put in the central node. If you don't have this particular layout, I will give you some tips to figure out how to follow up, how to follow the exercises. So I'm going to do this first exercise with the blue one machine. So I have loaded the credentials here. So the blue one here, let me figure out. In my case, I have it in the worker one, but let's verify this together. So if you see the port ID is 310 ECB and the 310 ECB is the last one here. So you can see that it's under the chassis worker one. So that means that the blue one machine leaves on the worker one. So again, we can actually verify this by just, like in ML2, yes, we can just check the top interfaces. 310B. And we see the top port here. So this is the same way, like the top interfaces plugged into the OBS grid, and OVN controller will take over of the VR int. So similar way. This is another way of figuring out where our machine is. Of course, you can use the open stack service show and figure it out that way too. But okay, so let's execute the first failure. Big one failure one. Let's read from the central machine. So basically this has disrupted the traffic to the floating AP. Let's not disclose everything. Although you have the files. So let me ping the floating AP. Okay, so it's not disrupted, but it is because we have DVR and the DVR setting in one thing that we probably missed was that in ML2 OVN, the east-west routing is always distributed. So that means that if you have a VM running on one hypervisor and another hypervisor on different networks and you ping each other, that will go directly from the source hypervisor to the destination hypervisor. It will never go to a central node. And that is true regardless of the DVR configuration east-west is always distributed. But for the floating AP, you need to have the current configuration. You have to have a port on the floating AP network on the hypervisors. And then we control this setting with a neutron config option that I can show you here. So in this setup, this setting is true. So if we ping a floating AP from this machine that we have an IP on the... We have configured the 172.2443 IP address in the central node. So I'm going to ping the floating AP of blue one. And it is expected to be distributed because of the setting that we have seen here, right? So ping the 132. So if you happen to have blue one on the central node, I think it's best if you would just use the other floating AP red one. I mean, unless you have something weird, I think that one of both should be on the worker one. If not, just let me know or you can please just follow this term. The team obsession and ask me any questions. So we're pinging here. As I said, we expect this traffic to be distributed. So one thing that we can use to verify is like let's see the OBS configuration in the worker one node. So we have a VR int, the integration bridge, typical like with the multiple OBS. And then we have the external bridge, which has the Ethernet to nick attached to it. So if we put a TCP on the Ethernet 2 for ICMP traffic, we should be able to see the traffic. But because we have executed the failure script, the traffic is not coming through the physical interface because it is working. It has to go through the tunnel. There's no other way. Either it goes to the physical interface and then it's using the tunnel interface, the Geneve tunnel interface, the equivalent to the VXLan one that we are using, that we have in our Montreux. So for that, we can figure out if that is true. And instead of capturing packets on the Ethernet 2, we're going to capture on the tunnel. Now we see that the traffic is coming from the tunnel interface. So I'm going to start the same capture but then the physical interface. So what we need to do now is to restart it like restart the traffic to be distributed so that it doesn't need to go to the central node. In this case, because we have two nodes, it's actually going to the gateway through the tunnel. It's going to the central node, which is hosting the gateway. We can see that here as well. So you can see that the CR LRP, CR stands for TASI-REDIECT and the LRP stands for Logical RouterPort. So we have two gateway ports and those are bound to the central node. So the traffic is going through the tunnel to the central node and from there it's going to go again through the actual, through the hypervisor which hosts the destination VM. In this case, because I'm pinging from the external network, it will just go through the interface which I configured the IP on. So in order for us to restore the distributed bit for the floating IP, if you remember earlier, Slavic commented about these two fields for the node table. So we can check this particular entry and you see that the external MAC and the logical port. So both fields have to be filled and this is something that Neutron does. When the DVR setting is on, it will populate these two fields for the external MAC and the logical port. So now this one is empty. So this is why the traffic is going, sorry, external MAC. This is why it's going centralized. And we want to make it distributed, right? So the MAC that we have for this one, we can check how it does it look for the other floating IP that we have. And the other one is centralized, right? So sorry, it's distributed. So it's populated. We need to do the same. And this MAC actually corresponds to the floating IP MAC address allocated by Neutron. So if we do something like this, we will see that for the 131, we have the one in the F4B6. Whereas for the 132, which is empty, we need to use the MAC address of the floating IP here. So we can write directly into the DV. We use the same tool, OVN and CTL set, because we want to set on a specific column for a particular row in this table. Actually, let's fetch the ID. There's the ID here. This is the row that we want to change on the external MAC. We need to escape the columns here. There's a bit of a pain, but let me, and before I execute this command, I'm going to zoom out so that we can see the TCP. And yet it doesn't work, which is good. Okay, so now the traffic should be distributed, but it's not. Okay, I was changing the wrong one. This one, because it's the 132, which is the one corresponding to BLOOD2. And 132, the one in 89. And I don't have the ping button. Is it what is not working? Okay, so I'm pinging now, and you see the packet flowing from the upper right window. So it's coming from the Ethernet interface, and nothing comes from the tunnel. So we restart the distributed part of it. And that's another way of fixing it. And this is a recent patch that we landed. So long time back, because you may face the situation where you have, I think this is kind of rare, but it can be certainly possible where you have a bunch of floating APs on a non-DBR setup, and then you want to make it distributed. So what happens with existing floating APs? Either you go and do this, or then you let Newton to do it automatically. So we landed a patch not so long time back to owner these chains. So if we run the failure script again, I mean, this is something that if you update your system to make a DVR, then you will just change the setting on the Newton.com from then ideally you restart the server. So now if we do this, the traffic goes again through the tunnel. So we have centralized traffic. But as I said, like Newton has a mechanism to detect this and it runs up on restart. So if we restart Newton server, we can let it restart and ping. So eventually Newton is supposed to fix it because you will see that the configuration option is set to DVR, but there's certain floating APs that are not properly configured to that. So during the startup, it will figure that out. And as you can see, now the traffic is flowing from the internet through interface. So this is another way of fixing it. I mean, the first way is like the OVN way using the DV commands. But if you're having a setup and you want to change the configurations, Newton will take care of these things. As I said, this is something that I think is probably in Nusuri. I don't think it's in earlier releases. For early releases, you may need to go onto the same thing that I just did. Okay, so let's move on. Now that we have the traffic restored here, let's execute the failure to... Is there any questions to hear? All good? Let me check there. Just feel free to... I'm not able to catch up with the chat, but if there's something that you would like to ask, feel free to interrupt. I will move on now to the failure number two. Let's run it again from the central node. So let's do the same thing. Let's try to ping the same floating AP, and it doesn't work. Why it doesn't work? Let's put a TCP down and see what we can see here. So if you see the traffic... Actually, you see the replies are being sent by the virtual machine, but the destination MAC address is this one that looks incorrect. I don't think that you can see what I'm highlighting, but usually that is the dead beef 0101. So that is definitely not right. So this is why the packets are not arriving to the central node. So we see we're pinging from 172.2443, which is located here, and the MAC address is the one that ends in easy work for them. So what has happened is that the failure that we have introduced is poisoning the local OVN... It's not local, but it's the ARP cache that OVN holds. So this is stored in a table that is called MAC binding. So if we see... Let me have a bunch of... Let's see this particular entry that we are interested in. Right? So we see that is the poisoned entry. And this is the source of truth that OVN uses when it needs to... Kind of like similar to the L2POP mechanism driver, where instead of sending an ARP request to every destination, it has this local... Again, not local. It's distributed because all the nodes are going to use this table. So it will have this OVN cache for the ARP entries. So it will not try to ARP the destination if it is known to him. So it will just use this MAC address. It will trust the database, and it will use it. So what happens is that it's not reaching the destination because it is wrong. So why can this happen? It can happen, right? Like if you have this 172.24.43, something that we have is an external host. So maybe this IP is moved. Like you can, you know, reprovision this host, move the IP somewhere else, and the OVN will have the wrong IP address or MAC address for this actual IP address. So this is a problem that we have reported upstream. Like it's, you know, we have discussed it a little bit. Basically a few times, but we never reached to an agreement on how to fix it. Like there was some discussions toward having an agent mechanism. So entries that have not been used just expire them, but it's really, it's kind of hard to figure this out because sometimes, you know, this is distributed. So maybe one hypervisor is using it, but how, what if other hypervisor have never used this. So you need some sort of consensus on where, when a particular magma and the entry has not been used for a certain amount of time. So there have been some discussions, yet nothing has been decided. In Newton, we did some mechanism. We introduced some mechanism protection for the floating IPs that belong to OVN. Those are handled by Newton, but for IPs that are external to OVN, this is hard. So what, what is the solution here? Well, you know, first, the first step is coming to the conclusion that this is the actual issue. Once we have it there, we can actually have a, I mean, we could have like some sort of fallback mechanism to, like even periodically, you can drop this table fully. And the drawback of doing that is like, you are going to repopulate it. So all the, you're going to have a lot of art traffic. If you drop the whole thing. Another option could be if you know that one entry is problematic, like in this case, you can just drop this particular row. And it will be our, I don't know. Okay. Thank you. Yeah. So I'm going to do the second one. So, you know, therefore highlight this. So what is going to happen is that I'm going to drop it. So when the VM is going to reply to the 1722443, OVN is not going to know the MAC address. So OVN controller is going to send an ARP request to the 1722443 in the central node. And we're going to see that coming through the TCP. And traffic should be restored. So we do from here. So let's run the ping from here. It doesn't work. Okay. Let's capture ARPs. And then I'm going to drop the smart binding entry. By the UUID. It's eight, four, three, whatever. And there you go. Like the traffic starts to work. And in the upper window, you see that what that triggered OVN controller to request the ARP for the 1722443. So now the, the new location has been learned. And if you check again, this query up here is not, it's not going to have the dead beef. The poison pack anymore. There you go. It has the right one, which is the one that we configured in the machine. Well, we're ending in E049. Okay. So we restart the traffic here again. Any questions up to here? You destroyed the database rights. Don't you have to keep up with it? Sorry. You destroyed the database rights from the south. You have to repopulate it or it's automatic. Oh yeah, like, yeah, I destroyed this particular entry. So what is going to happen is that when the reply packet is sent by the VM, OVN controller is not going to find any entry in the Mac binding. So it will decide to send an ARP as we saw in the upper right window. And what, upon the receiving of the ARP reply, OVN controller will populate the DV automatically. So if we check it again here, the entry has appeared right. So if I, I can delete it again. So, okay, okay. I got it. I got it. Okay. Okay. Yeah, but every time that OVN sees an IP, it will just populate it. You see the port where it has observed the ARP. Yeah. Yeah, and if you see like what I used here, you see the UUID that I used earlier, 8-8-4-3-D. Now it's a different one is 8-8-8. So OVN controller created this one. Also you can go into the DV and actually see the transaction of who executed, but that would be too much. So we're going to run out of time. But yeah, it was populated again by OVN controller. Thanks for the question. Okay, so let's move on. The next failure we need to execute it in the worker one. So let me do that from here. Failure number three. Okay. And then I'm going to use the same IP and ping it. So it doesn't work. We have disrupted the floating IP traffic. Again, what is happening, we can, we can see nothing is coming. Right. So because it's, we know that it's DVR. The traffic needs to come from the physical interface. Right. So we are, we cannot see anything here is probably because. Even this is not the central machine is not even able to resolve the 132. So the entry has expired in the local, our cash for the central node. So you cannot even resolve the MAC address of the 132. So what happens is that we are probably going to see the apps coming through the physical interface. There you go. So we see the central node. I'm querying the MAC address of the, of the VM of the floating IP, but it doesn't work. So who is responsible for replying to this app. So actually it's OBN control running in worker one. But the art is not coming through a VRM. Because we check here. Sorry. In the worker node. We see the layout in OBS. We see that. Yeah, we have the two breaches. We have the VR int and the VRX, but there's no patch port that in second export. So the packet is arriving to the Ethernet to, to VRX. Yeah, we see the, the packets here. I think that I need to set the open flow 15. Okay. It doesn't matter. So we saw that the actual apps are coming through Ethernet to. So what is happening here is that the packet arrives to VRX, but that's no connection to be ready. So the packets are going to be dropped in VRX because there's no other ports. So what we need to do, like the interconnection between VRX and VR in is. Is down through the configuration of the breach mappings, which is a similar concept that we have to neutralize as well. So those are configured on each hypervisor. In the external IDs. Right. So if you see. Get open. Sorry. So this is actually telling OVN that whenever you see a physical network data center, then you need to create a patch for between VR int and VR and this breach is specified here. So if we see in our desktop setup, our public network is called public. So that's the, that's the name of the business. And in OVN we can, we can figure out like the way that OVN realizes this connection to the external breaches be a portable of type local net. So we will have a logical switch port. We can see it here. We have a logical switch in OVN. Which has. The router ports. Each router port. Represents a subnet and interface connected to one of the red and blue networks. It has a local port. This local port is used for metadata. And it has a local network. This local network is to provide a connectivity. To an external breach. So if we see the configuration in the database. For this port. We can. No, you can switch. Type equals local net. So you see the options column. It says the network name is public. But what we had to configure in the worker one is data center. Another way that we can have to figure out this type of, you know, external connectivity issue is by checking. The OVN controller lock. Usually OVN controller will warn us about this. Sorry. Yeah, so what OVN controller is telling us here is like. There's this local net port. That we haven't found a breach for it. So the traffic. Kind of go out or in. So it's warning us. So definitely we know the issue, right? Like the issue is that we don't. We have the wrong mappings configuration. So all we need to do is fix them. So instead of. Getting this, we just set it. Like. Public. Yeah, yeah. And then if I ping again. It works. And then in the work node, we can. We can see that OVN has created a patch for it. So now you see this patch prop net. This is a patch for between the area. So this is how the traffic is coming from the physical network. To the variant and processed by OVN. So also be aware that if you do this. If you do this command here, you may be able to write another network. So you could have something like. I don't know. I don't know if he's meant to be our provider. Something like this, right? This is the same syntax as in a multi obvious. So I don't think that there's a way to set a pair. A mapping individually. I think that you need to. Write the entire thing. So. Whatever you. You want to configure here. You need like, if you just want to eat a. You need to be careful and not overwrite whatever it was in the DB already. So this configuration is read in real time by OVN trailer. You don't need to restart OVN controller at all. It just. You write it into the local OBS TV and it will be picked up automatically. But OVN controller as we just saw. Okay. So far so good. Good. So let's move on. You have 15 minutes. Hopefully enough. So we're going to run the failure. Number four. Okay. So what I have done is to, you know, basically prevent the HCP. From working in this particular VM in the blue one. So let's verify that this is the case. So what I'm going to do, this is. This is kind of like the equivalent thing that we used to be an amount to OVN when we were to the Q router name space or the QD to P namespace to access a machine that doesn't have a floating AP. We have a floating AP, but we're going to do it locally anyway. So our floating AP. Sorry, our local IP is on the 20 network. The catch with this trick is that we need to go to the actual hypervisor that hosts the VM. So we are going to use the metadata name space. But we need to go to the machine that is actually hosting the VM. In this case, we know it's on worker one. So. So it's on the 20. If you can see here, the top 744 has the 20 network so we can use this name space to access the machine. SSH 02011 go foos go. This is our machine. Right. And then let's less query the HCP. And it doesn't work. So we have a machine it kind of fetch an AP what's going on. So ideally like. I mean, there could be things that we can look at. For example, the way that the ATP works in OBN is we don't have DNS mask in us in a remote hypervisor or the DTP service. I mean, it would, it could work with the Newton DTP agent by the default way of handling the ATP is locally in the hypervisor. So there's going to be a flow in OBS. There's going to be a flow in OBS that will send via controller action to OBN controller and OBN content will process that packet and reply to the DTP request. So if the if OBN controller is not running. Then we don't have the ATP. So we can, you know, it's OBN content running. It is running. So that kind of be, or maybe there's something wrong with you. We can check the logs to see if there's some error or something. So the most, you know, the way that we should start troubleshooting these by looking into DB content is this, of course, we have checked that Newton has the DTP enabled for that sound and so on so forth. We isolate everything to OBN. And then we need to figure out what's wrong in OBN and hopefully fix it. So what happened here is that we are sending the ATP request and there there's no reply coming. So OBN sends everything to contract. We can actually verify that there's check like, you know, these queries is broadcast queries for the ATP for 60 a destination. And replied in contract. So there. The connection like the DTP request is saying here, but there's no reply. What could be the reason could be a reason for some misconfiguration on the OBN side. Let's see. So how we, how neutral configures the ATP. Is if you recall what Slavic said earlier, he, we have a direct mapping between a neutral side and a row in the DTP options table. So if you see that the one that corresponds to this particular summit will be the first one, right, which is the 20 to 00 slash 24. And then what neutron that is for each logical switchport. Let me fetch the blue. Command here. And then we can find the logical switchport whose neutron port name is port blue one. Right. Sorry. So we can see that this, this is actually our, our VM right and it has a 20 to 0, 0, 11, which is our VM. And you will see that there's this column DHP with the four options, which is empty. And this is what is telling OBN that this machine doesn't require any flows to handle the TV. So in the table that handles those flows, there's going to be nothing. So when the traffic arrives, it's going to be dropped because there's not going to be any match in the open business tables. So what we need to do is to configure this column. So we can, you know, we identified which subnet should be and which is the first one here. So I'm going to just use this ID. And then we can set it manually here. So we can OVN and VCTL set. Now you can switch port. This row. And this PV for options equals the nine to the, whatever. Sorry. All right. And it works right so now just when we said that OVN controller installed the flows. And there's it replies. So it actually, I think that we can even see it here. 16, 19, 50. So we can see the DHP offer and the DHP. So we fixed it by adding the particular DHP options row to the logical switchboard. But this, again, like, this is, if you know of a single part that has a configuration, you can go manually to the DB and fix it this way. We have other means of fixing this. We have been, so now I did it again. I introduced the failure again. So we don't have the HPP anymore for this particular VM. And one thing that we can do is we will, we have in the Newton repo we have this script called Newton or VNDB sync, which maps or tries to match the Newton DB contents to OVN content. So if there's something missing, we will try to fix it. It doesn't work on all SNR. So if you're going to go on, you know, I don't know things. Some feel it may not be fixed. But for example, for the DHP, or if you remove a complete object, or even if you drop the whole OVN DB, this script is going to make sure that all the OVN resources are going to get created. So if you're unsure about certain field or something, you can remove the object around the script. So I'm going to do it for you to see. It's very handy. And just as a, you know, just to point out that this script is the one that is used for migrating. So if you are moving to formal to OVN, you're likely going to use this script, which is going to populate the OVN DB from the contents of the Newton DB. So I'm going to run the script. Actually, the parameters are the ones that you use for running Newton's server. It's going to pass the configuration. The same way to have all the settings, DVR and DVR, the connection strings to the DB, so on and so forth. I'm going to put it in repair mode. You can specify luck just to see what is supposed to be doing. Because I know that what it's going to be doing, I'm going to just go ahead and do it in repair mode. And then I'm going to dump this output to the, to the DB single. By the way, I'm going to remove it first because I probably have one. So I'm going to do it on the moment that this tool, and by the way, you don't need to stop Newton server, you can run this in parallel, and it will connect to the DBs and fix whatever it needs to fix. But if you see in the upper right corner, the machine has already been able to fix the ATP. So it fixed the DB. One thing that we can inspect the log actually now to see that this is, this has been fixed. So one thing that we can do is because we know the port. And finally, we can, we can rep for this particular operating in the log. We can see what, what the DB sync script has done to the port in the OB and DB. So if you see it now, well, this happened just now, but basically what it has done is it has detected that the HPP for options were missing. And it fixed it by running, I said, like the same thing that we did earlier manually. So on this particular logical port, it says that the HPP for options to that one of the subject that it belongs to. And it worked. There's a way of fixing it. Some basic troubleshooting how to determine whether, you know, the problem is normally a misconfiguration, it could be, it could be something else, it could be open and try being too busy, or it will be going to have been dead. So these things you need to figure out, but this could be most of the times the issue would be probably misconfigurations, or, well, this is also applicable to any other object. So if you like, if we are unsure about certain, for example, floating AP the same exercise that we did in the failure number one, we can totally drop and not entry and run this script and it will recreate it for us. Okay, and then moving on to the last exercise. Let's do this as well on the worker. So failure number five. And so what we can do here is, let me, let me for example run this command here with a watch. So we are going to query the agent list. So we have central node acting as an OB and contract gateway agent and the OB and content agent in worker one. So what I have done in the worker one is I have, I have modified the connection string to the dbs to the south bond db. So, so in the logs we can see that there's some error in the connection. I just moved the protocol from TCP to SSL so that it fails. So it cannot connect to the south bond db. Like if you see, for example, if I do the DATP request again, it works because OB and content is up and running. It doesn't need to be connected to the southbound in order to fulfill request from BMS like the ATP or DNS. So everything that requires a control action is going to be running regardless of the db availability. So data plan should be fine. But if I could try to put a new VM on this hypervisor, it will not work because OB and content is not aware of the changes via OB STB. So as you can see on the, on the left window, the OB and content agent is now shown as dead and XXX. So OB and content is dead there. We need to fix it. So usually how you if you see something on the left, like, like in the left screen. Usually you need to go, it may indicate a problem with OB and content being able to connect to the southbound db. So there's a mechanism in place in neutron where neutron server writes a timestamp. And it has, it needs to reply to, to that right now, to that timestamp acknowledging that it has seen it. So in this case, it's because it cannot connect to the db. It cannot reply to the db. So eventually after 75 seconds, which is the default agent downtime, it will show up as dead. So what we need to do in order to restart to restart the connectivity is fixing the connection screen right so again the connection everything, all the OB and configuration is in the local of the db. So we can do open OB and remote. And you see it's using SSL. So actually, it could be SSL, but we have not configured this machine with SSL yet. So basically we would, we would, we can actually verify this for example like this. The connection table uses has ptcp. So it's going to use tcp. So all we need to do is fix the connection string. So that it uses tcp. And after I set these OB and I will pick it up automatically and it will register the heart beats. Hopefully and use it's immediate. It's happy again on the left window. Yeah, another scenario that we thought it was a typical troubleshooting scenario like, you know, identifying issues with the connection to the db. And that's pretty much it. We have just two minutes. I'm sorry it was longer than I was expecting but feel free to make any questions that you may have also whatever is pending on the either part will make sure that we answer all the questions that you may have. All right. Thanks everybody for attending. Thank you. Thank you. One more words at the end. Later tonight I will export the slides to the PDF or some other something like that and I will put the link to the slides in the ether part so you can you will be able to download them. Okay, thank you. Thank you for coming. I hope it was interesting for you. It was very interesting. Yes, it was. Thank you for the presentation. Thank you folks for attending. Thank you. Bye.