 Good afternoon, so I think we can start So hello, my name is Emilia Mackey. I work for in events as an open-stack automation engineer Hello everyone. I'm Sylvain of China. I work at in advance as a software engineer. So Today we are going to talk about High availability in neutron and three agents. This is a like a hot topic in neutron because We wonder that a lot of people deploying the else reagents in production By the way, we wonder how many people in the room is deploying neutral and free agent in production today Okay, good And do you feel like this guy a little who feel like this guy in the room me Okay, so what's the problem in the else reagent on current implementation? so if you if you use the L3 agent in production your tenons will create a virtual routers and All the virtual routers will be scheduled on the L3 agent nodes The current problem is that if you lose a L3 agents If you lose one or two the virtual machine will lose a connectivity from external networks and Also, they won't be able to reach the internal networks between Tenant networks, so that's a big issue in production However in ISOs, we had some cool features We are going to talk about two features today The first one is about bringing a new scheduler in the L3 agents It's called the least list routers scheduler The idea is Instead of a randomly scheduler virtual router on the nodes you can know Schedule the router on the L3 agents where you have the least number of routers so you can win in scalability and And also you don't break the existing infrastructure The other nice feature it's not related to HA but just to talk about the new features in the L3 agents The new cool features is about the ability to schedule More than one external network to a single single L3 agents by using the provider networks So previously you could not use more than one external network on the L3 agents and Since high source you can know by using the provider networks having more flexibility By having multiple external networks. So the use case is probably the Public cloud when you have a lot of Floating IP pools you can now have multiple external networks and Connect them to the provider networks However with all these features we still not have HA that means Again, if you lose a L3 agents you lose all the virtual routers and the floating IPs and so on so at in events we developed a tool which is not part of neutron this is a This is a tool that we introduced in grizzly and it works on grizzly Havana and I south its open source You can download and install it So the idea is to install a third-party service running on all the L3 agent nodes So by using RPC it will check the actual L3 agent status and if there is a Failure on the L3 agents it will reschedule all the resources. So in this picture you can see that you have Virtual router a scheduled on the first L3 agents And if you have a failure which could which could be a system or network failure The service will reschedule the resources on the second to the second L3 agents So we think that it's it's still not the solution because first of all it's not introduced in neutron So it's not part of the project officially Also, it's not a full HA because you still have a downtime the service is not stateful So using it in production could be a bit tricky when you have a lot of resources on the L3 agents Because you you can have like a one minute downtime depending of the number of virtual routers that you have However, it works we are going to show you a short demo and it works you can You can install it on existing in platforms. You don't break open stack code. It's just Another service that you plug to the L3 agents The service is quite smart because it it detects when the service is Is down. However, the network will be isolated from the L3 nodes and Also, the service is distributed. So if if you run the ill check on the multiple nodes it will When the when the reschedule process will will will happen There is a lock system that will lock the other ill check to be sure that we don't reschedule reschedule two times the same resources We so at conclusion we think it's not the solution but We are Sylvain is going to show you after the demo Two eventual solutions that will be introduced in open stack Juno. So the next table release So we are going to show you a demo of the ill check on our infrastructure So in this demo we are going to spawn a VM so this is I south as you can see in the dashboard We're going to spawn a VM which is a CROS image We are going to Show you that we can ping the VM Obviously we have two L3 agents to show you the the HAP process Okay, so the VM as a floating IP and This floating IP is connected to a virtual router So at the top left you have the neutron agent status So you can see that They are all working for now At the top right you have the second L2 L3 agents Which currently host the virtual router? We are going to bring it down later to show you the failover At the bottom left we can ping the VM You can see the namespace of the external network in the top in the top right and And so Sylvain just run a command to stop the L3 agents and we can see In the bottom right that the namespace has been rescheduled on the other L3 agents and The again the failover happened and we can now ripping the the virtual machine So here it's it's quite short and the downtime is quite short because we only have one one resource Which is a one virtual router, but in production when you have hundreds of virtual routers we We we could see that the downtime was about two minutes so it depending of your Scheduling process it could belong to to reschedule the resources So this is definitely not a solution adopted for Neutral development and in in ISOs We started some work on Some features and Sylvain is going to continue explaining The next step for bringing the L3 agent in production with H8 Okay, thank you Emilia. No, I want to remove Okay, so I'm going to talk about two solutions to new approach That should be introduced in the next release The first one is a very happy approach The idea of this approach is to use is to to use a very pretty protocol, which is a network standard and the goal of this Development this approach is to to not introduce to too too much changes in the neutron as a current neutron design, so We will not have too many changes in the API no changes in the API Just a few configuration few extra configuration parameters in the neutron config file should works for of course with the external network and internal networks and Doesn't change anything to the current Services are relying on the on the L3 node. So the firewall as a service VPN as a service on the LBS of course Will not change We have an active passive design for the routers and active active design for the L3 agent And the goal to find the main goal is to the main goal is to reduce the downtime of course So we are in this picture. We see what's happened in terms of the scheduling We can see that there is each agent is able to host Either the master version of a router or the slave version of a router. That's why I just said we have a Active passive design from the router point of view and active active design from the Agents point of view since we can have on the same agent Both active or passive Routers so the implementation is quite straightforward We have just we just have a new L3 scheduler just to to schedule a router twice On on two different adjunct One version will be one the master version will be hosted on On an L3 adjunct and the slave version will be seen on another one Although there is a little change in term of the database camera a new attribute for the virtual router Is introduced just to specify the VIP ID for the for the router And if you if we look at the L3 adjunct side Finally all the IP will be converted to to to a VIP and the interface management Will be handled by the keep alive the By people have the Optionally we could use a contract D to support to to to keep the TCP session alive It's an option If we look at this picture, we can see in more detail what's happened exactly we have a keep alive the instance pair network pair, sorry pair pair router running inside the The router namespace We can see also that all the IP is just converted to a VIP and managed by keep alive D And we can see there is a special Interface an H&A interface Where the all the VIP traffic will go In order to just isolate the VIP traffic there is some limitation few limitations finally One is the number of router pair tenant Since we have only one with this design. We have only one H&A to our pertinence. So due to the limitation of the VIP protocol We have a limitation of 255 virtual routers We could remove this this limitation just by allowing the to add the only allowing the tenant to add more than one H&A a network It's a great improvement in term of H&A but not in term of Scalability So I'm going to talk about a new Proposal, I'm happy to see that there is there are some developer of this proposal Which is just before maybe we can just start a little demo. Yeah, thank you So this demo is pretty is pretty similar to the the previous one. So let's start by just creating a Virtual router Maybe you can say that Router process is the same for the end user. Yeah, the end user doesn't see that he's creating a H&A router Yeah, yeah, there is no change For the API so we can do everything With a reason for example So now I've just I have just Attached the virtual router to an external network and we can see that the two virtual return namespaces are created on the two virtual to L3 agent So now we just Start to keep TCP dump just to see where the traffic will go and we can do that on the both L3 agents on the both Router namespaces since all the interfaces is created are created Now we're going to to to attach the the router to a private network and Then we are going to start Virtual matching. Okay, now we have the VM starting We can associate a floating AP and Do the same thing just start a ping to see Where the traffic goes So it's the same terminal like before you have the second L3 agents on the bottom Right, which is for now is passive and the active L3 agents the active router is hosted on the top top right Exactly, so we can now we have the connectivity. We can see the the traffic is going through the L3 node on the top right and we are going to to just shut down the This node to see what's what's happened Okay, so traffic is now stopped and We will see that okay now we can see that the traffic now is go going through the the L3 node on the on the top on the bottom right so we just saw That we have a solution an approach to to reach a chair but as I said just before we we could improve the Scalability and that's the goal of the of this approach of this Solution which is a DVR DVR solution for distributed virtual router The goal of this approach is to distribute the routing To each compute nodes So to improve the east-west traffic, but not only also to improve the north-east Traffic especially for the virtual life for the floating IP and Some some services running on the on top of the L3 agent could stay it could be distributed as well probably the firewall as a service, but some others should be around on Something like a classical L3 node, which will be a service node If we look at what's happened Currently without the DVR We can see in this picture that if a VM1 wants to send some packets to a VM2 Hosted a plug on a different network The traffic will have to to to to go through The L3 agent so through a virtual router even if the even if the VM are hosted on the same compute nodes and If we look at this picture, and we can see what's happened in terms of east-west traffic We can see that there is a DVR instance Started on each compute node. So the traffic will go through the through each Instances of the DVR We don't have any more needs to To to go through the L3 node In this picture, we can see what's happened when we associate a floating IP to a VM. So the VM with the floating IP will be scheduled on the the compute node hosting the the VM so As the same as before no need to go through the L3 agent anymore and finally since the DVR is notable currently to To to implement the SNAT capabilities When we start a VM without any floating IP, we still need to have an SNAT mechanism to get an internet connection We can we can combine the two solution in one So we can have the service node which will be which will host the The SNAT mechanism, but also some services which Could not be hosted by by the DVR So we can see that on the on the top we can see that there is the service node running on the on top of the VRP and We still have the the DVR on each compute nodes and I think that's it So thank you. If you have any question It's open for question if you will with the approach of distributing virtual routers on the on the compute node another computer knows on the Network knows the scale to two or can you have more than two nodes that host the routers? So can you have for example five or ten network nodes that all have the same router? You want you want to distribute more than You want to have a more than one master slave, right that possible or not? Yeah Of course. Yeah, just a quick question about the VRR P capabilities are you also Using a single MAC address a generated MAC address for the VRR P And if you are are you pushing out the tunnel updates to fix all the locator records? And if not are you registering both locator records when you start the router? What was yeah a single Mac AP? Yeah, so so you are doing the tunnel update on failure as well. Yeah, is L2 populate required then? Yeah, probably, okay Yeah I'm from HP in case of VRR P I remain cream trying to create the NAT session even on all the nodes And if so, how? So, can you repeat? I mean the NAT session table Let's say on the on the network nodes, let's say we have multiple as you told multiple network nodes are there and the traffic or the outgoing traffic that may be created on one of them one of the node, but How about in the other nodes? I Sorry, I didn't get you. Yeah, you mean you are talking about the H&N network. All right. Yeah What do you mean exactly with the H&N? I mean in case of H&N at work. Is there any issue with the NAT? With the NAT? Yeah, no, I don't see any issue with the NAT. Okay You showed a demo of the VRR P solution. Is it actually implemented in Newton or not? It is currently in review Okay, so what you showed? Yeah, you can you can just download on the patches and just yeah, just have to test Thanks. I Think you made a documentation to yeah Yeah Yes documentation to deploy all the patch to apply all the patch all the patches and to test it Can you you can yeah check that on the wiki? Do you think it makes sense to you to Add the keep alive de capabilities to the L3 agent directly and not using external dependency for Keeping VIPs for virtual routers So you it's actually the case as an option if you want to enable H&A Automatically the L3 agent will be managed by the keep alive de backend for the VIPs But can L3 agents communicate directly to each other to You want to remove the keep alive de dependency? Yes, you want to improve to implement the VRP protocol directly in the in the agent finally Yes. Yeah. Yeah, but I think it's not the goal of the L3 adjunct. I think there's yeah Maybe maybe not but currently there's a implementation of keep alive de is pretty good And we are a lot of future like we can execute some script when there is a failure Which is great because we can for example, we can start a metadata the process when there is a failure and also Send another script. So thank you Have you considered a pacemaker for example instead of VRRP? That would be the option at all Actually with pacemaker you still have a downtime You currently you can install L3 agent using pacemaker. I know a lot of people is using it for example, we don't because it's the same For example with pacemaker you you have to run L3 agent in active passive But you have one L3 agent with which does nothing because pacemaker didn't run the process, right? Yes, yeah, so you you lose in scalability because you have to deploy the double of double number of L3 agent nodes and Half of them does nothing. So you lose a number of servers for nothing Yeah, yeah with a with a current implementation of the VRP solution we can an L3 agent is able to to host boss Slave version of a router and also a master so we don't yeah in the in the VRP in the VRP design your L3 agent could be Both master and backup for some networks for some routers. All right. Thanks. So In the case with the L3 agent on each compute node, how do you deal with VM live migration? For example for the floating IP. I don't know exactly since it is currently in Development exactly But I think it's something that should be in addressed. I don't know how I don't know when Having having the floating IPs at the edge of the network You know up north kind of sense in that case Question regarding both approach in the first approach. I saw your slides are showing Two L3 agent on two different network nodes And each of them have their own individual VMs does that imply that the first approach actually have active active mode So one one L3 agent is going to accommodate or serve the L3 functionality to one subset of the VM another L3 agent is going to serve another set of subset of the VMs I'm not sure to understand but as I said just before an L3 agent hosts both The master version of the slave version could in both cool the host the boss version But for two different Okay Like my second question regarding the second approach which use a VRP I assume in this case a very VRP can only handle the switch over for the tenant network from inside to outside Is that correct? Yeah, and how about from inside to from outside to inside direction So no, sorry, I didn't get you so from let's assume you have a router You have external network and though you say have the floating up your dress Tied to the external network in that case when you switch over to the another router or L3 agent How the traffic from outside to inside it is being handled? Yeah, you have to you have to to to to attach your UL3 agent all your L3 agent to the same external network Thanks to the provider network This case is to using the provider networks and to connect to your provider network to the external networks Oh, I see. Okay. So you have all your L3 agent connected to the same networks Okay, okay. Thank you very much Hi My name is Charles game from Erister. I have one question about the VRR P implementation as I know VRR P is active standby and Have to use the keyboard live to track the status of the pure lovers Have you ever considered the active active a mechanism like the future ARP or based on the any cast mechanism? It's it's fine. It's it's a proposal and Since we we have an active active design from the L3 point of view It's not mandatory to have an active active design. So you will replace VRP by something active active Do you have an example of protocol or of standard? Yeah, actually the VRR P Usually they used for the pairing of louders, but in case of the virtual any cast ARP mechanism can use the multiple louders It depends. It doesn't defend depend on the pairing mechanism So maybe more scalable. I think at this stage it's tough It's just a first implementation to bring the future. Maybe later. We could spin think about this kind of set up more scalable and with active active Architecture, right? Yeah, and also we just use a keyboard ID to manage the the interfaces But finally we could use something else. Yes maintaining the Keep tracking of the status. It may be as burden and cause the some kind of the scalable issue So yes, I think okay. Okay. Thank you This is weak from HP. So in this implementation There will be two L3 agents that will be listed in the neutron agent list, right? Like both of them will be listed, right? Yeah, so both the agents should have an It should have a unique external network IP assigned to their You know external networks, right? Yeah, so when it when a traffic comes from outside to inside and say it uses an external IP to reach not to to reach it into the cloud when a failover happens, right? So, you know, there will be a loss of traffic, right? So the one you showed had no loss of traffic because that was an east-west traffic But in a north-south traffic, I think in such an implementation will have loss of traffic, right? And I'm not sure to To to get you I mean when you when you have the failover your traffic Is moving on to the other node thanks to VRP on the both side in the north and south So I don't see where you have lost of traffic Because we are tracking TPC connection and we are tracking all the traffic thanks to VRP So I don't see where you have lost of Packets, okay, so when when you have a frame when you have a frame from external network coming inside to the router that the router that is The stand the router on the standby L3 agent or the second active L3 agent So that that router will will keep the same external network IP is it? Yeah, yeah, yeah, yeah, yeah, of course, of course Yeah, when you have the actually when you create a router the namespace is created on both nodes But the the floating IP or the external IP is created only on one nodes And when when you have the the failover process thanks to VRP The the keeper live D will just migrate the VIP So the external IP to the other nodes so you don't have problem with not and Features also IP is is migrating Yeah, I mean for for that Let's say we take them to net to network node one net N1 and N2 on N1 There is an outgoing packet for it We have done a source night and you created a session and that N1 goes down and N2 takes over now But that session is not there. So how the packet will be handled. That was my earlier question Yeah, we we we plan to use contract D to keep the session I was just gonna say maybe the answer is that we're you're gonna do In our broadcast on the failover and that's gonna reduce the packet loss and maybe that's what's missing here is that I Don't know it's not okay Any question? I have a question that you are introducing a internal RPC RPC mechanism between to communicate between the active and the standby So is there a redundancy for that RPC bus? Sorry, what you are using a RPC bus between the active and the standby to extend the RPC keep alarm messages, right? No, no, no, no, we have a if you look at the architecture we have so this is a picture under the hood where you have oh Could you? Could you turn on the screen please? Could you just turn on? Thank you So if you look at the picture you have a dedicated network for HA Which handles the VRP traffic so what what is the mechanism between these two to communicate? Is it again the same global RPC bus is used or the internal RPC bus is used? No, no, it's it's just VRP packets sent of through the HA network So when you create a router the router is scheduled on on both L3 agent and between the L3 agent You have you have a dedicated networks, which is called here HA and this networks handles the VRP traffic and when you have a downtime Let's say in the first L3 agents Thanks to VRP the Traffic will go to the second L3 agent. You don't have RPC or Does it need additional interfaces or it will take the management interface for me exchanging the keep alarm messages? No, it just it just it's just a virtual networks Not reachable by the tenons. So the tenon doesn't see the this network Do I answer the question or no? Yeah So you have up here you have the HA On a dedicated network, would it be possible to use whatever your neutron provider is to generate that dedicated network to get around that 255 limit Yeah, not with this implementation, but could be Okay, we have five minutes if you have more questions. Okay. Thank you very much. Thank you very much. Bye