 Hi everyone, welcome to the first Open Infra-Live episode of 2023. My name is Allison Price from the Open Infra Foundation and I'm very excited to be your host today. Open Infra-Live is a virtual series hosted by the Open Infra Foundation that we started a few years ago and have continued so we can provide a platform for the community to tell stories around open infrastructure and their open source strategies. Today I'm very excited because we have a group of folks open stack and Kubernetes, maybe just one of the most popular open source integration stories of today. So before I kick off, I do want to remind you we are live, so all of our participants today are open to questions throughout the presentation. We even have a demo that we may be showing at one point, so I hope you all tune in for that. But if you have questions, please drop them into the chat of wherever you are streaming today's episode. So today's episode is actually really special because it's a presentation that was given at an Open Infra meetup in the Triangle area. So if you're not participating in your local Open Infra user group, I encourage you to do so to get more information like this. Before we begin, I do want to thank all of our Open Infra Foundation members who make today episodes like today possible. All of our participants today are actually from one of our Platinum members, Red Hat. So we also want to thank Red Hat for giving the opportunity for these members to get on and share their story. So now I would like to pass it to the first presenter, Chris Jay, who will kick off today's episode. Welcome, Chris. Hey, thanks, Allison. Greetings Open Infra community. So my name is Chris Anychevsky. I am a field engineer and architect for Open Stack, Kubernetes, and really anything that runs in a data center. Joining me is my peer, Maciej Wensky, as well as two rock star software engineers for OpenShift and Kubernetes and Open Stack networking, Emilian Macchi and Luis Tomas Balovar. So we're super excited to bring you this topic of distributed architectures for Open Stack and Kubernetes. BGP or border gateway protocol is going to play a substantial role in our discussion. And the main goal for us is to arm you of the knowledge you need to transform what you might have in running like monolithic architectures into more sleek leaf spine topology for both your controllers and workers, you know, to increase your reliability, scalability, but I don't want to get ahead of myself. So let's dive dive in. Okay, so let let us set the record straight before we begin. We are here today speaking to you as a members of the Open Infra community. But it's worth noting that we also represent organization we work for Redhead. So anything that we present today is considered work in progress. And it's actively being discussed by the community. It's important to keep in mind that it may change in any time. Okay, with that out of the way, let me take you through a short journey or evolution of the data center. And we began with this flat, it was about probably 40 years ago, with the very first data center, where we just had the bare metal hardware switches, and then operating was sitting on top of it, and fairly straightforward and monolithic applications running on top. And then the next step of the of the evolution was to introduce introductions of the virtualization about 24 years ago or so. Fun fact, the virtualization was really introduced by IBM on the on the mainframe hardware. And but it was really popularized by VMware on the commodity hardware on the x86. And until today, as you might know, VMware is a main force in that topology. The networking has also evolved with this architecture from the flat single subnets, if you will. The concept of VLANs have been introduced where one could slice a monolithic network into different subnets, if you will. And then from the storage perspective, the local storage translated into a large storage array connected with some fast or ultra fast fiber channel connectivity to share the capability to the rest of the compute. Still, from the application perspective, it was fairly monolithic. The client server architecture became quite popular. And finally, stage three is the cloud architecture, what we know as a cloud architecture, the introductions of the containerizations, which provide a much more agile alternative to the VMs. However, both VMs and containers remain critical drivers for the compute. But with that agility of the compute, both software and storage have evolved as well. Software defined networking and software defined storage became more popular. We, our data center has stopped relying as much on the proprietary hardware to serve these services. And instead, everything is now being deployed on the commodity hardware. Application has changed as well from being monolithic. We changed into the microservices, very short lift app that are defined as a code, are moving very quickly. They support multi-tenancy. And they're scaling to the much larger numbers as we have ever seen. Okay, so the next one is when it comes to public clouds, we know that AWS has been kind of a king for quite some time. Even though there's a GCP and Azure that are keeping up technologically, AWS still keeping the kind of majority of the market share or big chunk of the market share. And what public cloud was always doing really well that was hard to match from the private cloud perspective. And when I say private cloud, the absolute leader today in the private cloud space is the OpenStack, together with the Kubernetes for the application. But it was always hard to replicate what AWS and other public clouds were doing well, which is the ability to break the infrastructures in the truly independent failure domains, what they call availability zones. And you might say, hey, but OpenStack had the concept of availability zone for a really long time. And that's true. However, how many deployments out there do you know where these availability zones are truly independent of each other? And if you kill one of them, it's not going to affect the rest of them. So this is kind of what we're trying to focus. This is the problem we've been trying to solve for a while. And we're going to talk a couple different methods that we were able to implement or work on from the OpenStack and Kubernetes perspective that helped us replicate this public cloud multiple failure domain model, if you will. So again, we're going to talk about most, we're going to focus on the two technologies, OpenStack for the infrastructure as a service and Kubernetes for the containers as a service. And we're going to start with the OpenStack. So starting with the kind of a very first architecture that we have worked on. And this from the triple open source project, we introduced this architecture in the Queens release. So it's been a while since we started working on it. And if you look at the diagram here on the left in this architecture, we're dedicating the first availability zone to run a control plane. So all your control services are running in a very first leaf or the network fabric, if you will. And together with that, you don't have to just use controllers, you can of course add the computes and hyperconverge or not hyperconverge software defined storage. And then availability zones one, two, and three, and etc. they are dedicated to to run your computes and storage at the same time. So there's some of the advantage of this architecture. First off, it's designed to be geographically distributed. And the requirements for the network latency are actually pretty forgiving. So as long as you have less than 100 milliseconds between the sites round trip latency, it should, you know, it's supported and it should it should run very, very smoothly. Another advantage is this is a it's been around for a while, it has been tested and popular, you know, adapted by quite a few open stack and Kubernetes users. So it's definitely proof tested, much simpler to deploy than, you know, pretty, pretty close to deploying a standard standardized open stack. There's some disadvantage or considerations that that we might talk about is that AZ0, so the first availability zone is pretty critical, right? If we lose it, if there is a disaster occurrence on the first AZ, the end users are no longer to create or destroy a new constructs, right? So they wouldn't be able to create new networks, storage, computes, etc. However, the workloads in the remaining AZ stay intact, right? So they there's operational and they they still provide services. Okay, and then we move on to the next version, which is a little bit newer. And it's not that this the second version is a is an upgrade of the version one. Look at it from the perspective of it having a satisfying a different use case. So as long as the version one was very good for the for geographically distributed architectures, the second version is aimed more for the for the availability zones in a relatively short geographical distance from from each other, but still separated with the different power distributions units and the network fabrics. And the change here is the control plane is no longer attached to single AZ. Instead, it is stretched over the layer two network, and the stretch across multiple availability zones. And this, this allows us to maximize the the SLA, it allows for better distributions of the of the resources, we can, we can lose any of the availability zones. And and there's no, there's no service failure to to our end users from from from any angles. Some caveats, the latency requirements for that is much tighter than on the version one. So if we can keep the latency between AZs in a in a single digit, or low double digit is, is, you know, that's it's required. And then the advantage, though, is we can in this architecture, we can also stretch the Kubernetes clusters with the also taking under consideration the the latencies. But this allows us to kind of build this very resilient architecture, not just for the for the open stack piece, but the Kubernetes itself, and we can, we can stretch it by using either the overline networks or some provider networks. Just a final thought. You can probably tell by just looking at this version, it still has a caveat of stretching that layer two network across multiple availability zones, which is not always preferred by a lot of network teams within organizations we we work with. So I want to introduce Machi Wenski, who's going to talk about the version three and how we evolve from there. Thank you, Chris. So version three of this deployment architecture is really focusing on that one primary thing. How do we distribute the control plane across the availability zones? And the best way to do this is to integrate our infrastructure deployment into rest of data center networking. And that is possible by using on spine and leaf architecture, where BGP is used to advertise both the control plane and data plane workloads. The challenges with stretching the control plane in in the previous iteration of this architecture, as Chris explained, were mostly due to using a layer two tunneling method for distributing the control plane networks. So that the control is running on multiple availability zones or in this architecture, where we'll be referring to those as leaves, had all access to that same control plane network. In the case of this architecture, the the control plane nodes are distributed across different availability zones and and they're able to advertise their vips using BGP. So that's that's a major change. There's a networking protocol introduced here. The networking protocol is BGP. And instead of stretching the control plane using layer two, we're going to use BGP to advertise the virtual IP addresses belonging to those control planes control plane nodes across the rest of the data center network. Now, as far as the data plane networks where our workload runs, we're still getting the same benefit we can distribute those workloads using different availability zones or leaf sites. But now since the control plane nodes are also distributed, when one of these availability zones goes goes down or two of them, we still have the ability to create or delete new storage compute on network on those availability zones. Let's go to the next. A little bit of an in-depth look into how the control plane is implemented. It involves rather a trivial change from the version two of this architecture. We'll be no longer using VRRP as the protocol to distribute the vips into the network. This is a layer two protocol, but we'll be using BGP. And the change is rather trivial. Instead of using VRRP, we'll be installing a VIP IP address on a local loopback interface of the control node. And a routing protocol in that BGP will pick up that VIP address and redistribute it to the rest of the control plane network. Thanks to doing that, by re-advertising that VIP, the rest of the data center will now have an active route to the controller in that given availability zone of leaf and direct all its traffic to that VIP. And let's take a look at how the layer three routing data plane works in this scenario. Now, we still get the same benefit. We still get a flexible workload placement. We're not limited by the network boundaries. And all that, all those changes are implemented using OVN BGP agent. The OVN BGP agent controls how traffic gets in and out of the data plane networks. And Lewis will give you a little bit more details about how that works in practice. Let's go to the next. Now, how are these routing changes implemented in real life? And you might have heard about the old project called Quagga. Quagga is an open source routing demon. It's been around for many years. But Quagga didn't have many updates for the past few years. It's been pretty much a stalled project. And a few years ago, I think it was four years ago, Quagga was forked into free range routing. Free range routing is the next iteration of Quagga. It supports many routing protocols. It runs on both Linux and BSD systems. The most important protocols that I wanted to talk about that are relevant to the spine and leave deployment and OpenStack are BGP, ECMP, and BFD. In OpenStack, Quagga, I'm sorry, free range routing is implemented using containers and it runs on all the controllers to facilitate the control plane networking. On all the computes to facilitate your data plane networking and network nodes. And I wanted to talk a little bit deeper about the routing protocols that I used on FRR. BGP is the de facto routing protocol of the Internet anywhere you want to route traffic between two large organizations, two continents, two countries. BGP is the routing protocol that is being used. BGP is a distance vector routing protocol. What does that actually mean? It means that the routing decisions are made based on the length of the autonomous path distance to its final destination. So every router that participates in a BGP routing protocol is assigned autonomous system number. And when BGP analyzes how far it is to get to its destination, it doesn't take into consideration like what the next hop is. It actually takes into consideration how long that path of autonomous systems is in order to get that. BGP is also adopted as the routing protocol of choice in Spine and Leaf network typologies. This is the deployment architecture of OpenStack when we're using the layer three routed BGP data plane and control plane networks. And it scales well. It scales so good that it's able to route the Internet traffic, which is the biggest computer network we know so far. Now BGP also supports Fiora extensions. This is not an extension. It's actually like an add-on. And BFD is one of those. BFD complements BGP. And in OpenStack, we're also using BFD to improve failure reconvergence time. So let's say one of the links goes down and in BGP, the timers that I used will hold the routes in the routing table for a certain amount of time. That amount of time is actually pretty long. BFD helps by monitoring the active BGP sessions and it will help you to take these routes out of the routing table and possibly replace those by another set of routes which can be used to get to the same destination. I like to refer to BFD as a BGP little helper. And the last protocol that FRR provides for us and that is very important to the implementation of spine and leaf and routed layer three routed BGP and control plane is ECMP. ECMP stands for equal cost multi-path routing. It's a very simple concept. Basically, it allows a router to use multiple routes as long as they have the same destination and all the other routing attributes are the same. ECMP can also help to load balance traffic so it can increase your throughput to that same destination if there are multiple routes present in the routing table to the same destination. And now I'll hand it over to Luis. Luis will talk about more in-depth details on how OVN BGP agent works in OpenStack. Thank you, Machi. So I'm going to explain a little bit about this agent and how it has been integrated into OpenStack to make sure that we are able to expose and provide connectivity through BGP to both VMs and load balancers regardless of they being on the provider networks, on the tenant networks, or with floating a piece associated to them. Just to mention that when we started this project, the main idea here was to not needing modifications on CoroVN or Neutron to apply this and this is the design we came up with basically for the agent. We have two main components, the driver and the watcher. The watcher somehow is doing something similar to the OVN controller in the sense that is reacting to events on the OVN southbound database and when certain events are detected then it's calling the driver to perform some actions which basically is about ensuring that this IP is exposed through BGP as well as there is connectivity to it. To do that, it first calls FRR as Machi has presented and the idea is to exercise it so that the IP is advertised as next hop, this node. Then we have both to do camera routing as well as obvious flows to ensure that the traffic gets redirected from the node to the OpenStack OVN overlay. In this case I mentioned OpenStack but the agent is designing such a way that you can also use it for implement different drivers that match your use case for Kubernetes, for playing VMs and these kind of things. It requires some things to work properly and the first one is to actually have FRR running and configuring such a way so that it advertises directly connected route and that's why we need to make the agent to add some IPs locally so that it gets advertised outside and it also needs to be connected to some BGP peers in the topology that was presented are the leaves and also there is a couple of extra requirements that even though the agent is taking care of by themselves like we need to have an IRP and NDP proxy enabled on the OBS provider bridges to ensure that the MAC tweaking is it's okay and for the same reason for the Tenant Networks we also need some static MAC entries for the OVN router gateway port as I'm going to explain later. At the moment we have three drivers available actually the last one was in Ten Days ago by the community we have two of them for BGP drivers which one for the Layer 3 data centers that we are commenting today but there is the new one added recently for also stretched data centers on Layer 2 and we also have an EDPN driver to handle that use case. So now we are going to describe a little bit about how the ingress traffic works and how the egress works. For the ingress I'm going to explain just the base case but there are some corner cases like small differences when it's a VM or a load balancer it's a floating AP associated to a VM but regardless of that the main thing is that there is two steps one is to make the infrastructure know that the IP is reachable to the local node and the other one is to ensure that the traffic flow is redirected to the OpenStack virtual networking. To do that as I mentioned before we are watching the self-bounded VM and detecting for instance that a port in OVM has a new chassis associated to it that means that it gets allocated to a node and then the agent called the driver the watcher called the driver and the driver start doing the next actions. The first one is like adding this IP of the VM to a dummy device that was created actually by the agent and that is isolated into a VRF to ensure that this is not just this is just to ensure that FRR gets to advertise that IP but not to actually lead the traffic to that dummy device that was created and then the second one is to actually redirect the traffic to the OpenStack and to do that what we do is like for instance if this is a VM on the provider network we add an IP rule and leverage the kernel networking so that whenever the traffic is destination is for this IP it will match this new IP rule and this will tell us to execute a specific routing table the routes totally not a specific routing table that was also created by the agent and in this basically we have some some routes I see the slides has disappeared I can continue also basically this in this routing table we add some some extra routes previous slide please we add some some extra route routes and these basically are the ones in church of redirecting the traffic to the OBS provider bridge normally named as VREX in this case. As I mentioned slightly different steps when when we use for instance want to advertise IPs on the network in the sense that it's not just advertising the IP we use the IP rules and routes to redirect the whole subnet side there but it needs to be done through the OBM gateway port this is the port that is connecting the tenant VMs to the provider VM and there is also some extra steps to be done like some static MAC entry need to be added into the kernel for this OBM router gateway port so that the traffic is properly redirected there. Just now there as I forgot to mention in normal layer 2 deployments you usually have this OBM provider bridges connected directly to the NICS or to a bond connected to the NICS but in this type of setup this completely disconnected and what we are using is the kernel routing to redirect the traffic from this physical NICS to the OBM overlay to this OBM provider bridges. But real quick before you move on sorry to interrupt we do have a few questions on this specific topic for the audience if you wouldn't mind so Michael actually has two questions regarding the OVN BGP agent how would it work if you needed to use VRF for ingress for provider traffic instead of a default routing table with IP rules? Well we are this for instance we are using for the EVP end driver but we are actually using kind of similar concept that we have a different VRF per tenant network and then the traffic through the BX LAN encapsulation is directly down there for the BGP we don't have such differentiation at the moment. The way we are configuring FRR is that it just advertised directly connected routes and then we leak the routes from this specific VRF that we created just for advertising those IPs but the traffic is not actually differentiated by different VRFs in this case. Okay and Michael actually had a second question. With the EVP end agent how can you maintain north south traffic via SNAT and use CRP MAC OVS rule to steer traffic into VRF? Wow that is a lot of acronyms Michael. Let me rethink how can you maintain north south traffic via SNAT? Well for the EVP end case it's usually we use it for not for the provider network but for the IPs on the tenant network and we actually need to use the the OVM router gateway port so we expose it not directly where the node is but where this OVM gateway port is created so and then from that point it goes through the tunnel like in the normal OVM setup to the compute node that is hosting the VM so there is a difference here like when the VM is created using the BGP driver and is created on the provider network we expose the IP right where the VM is but if we are using the EVP end or we are using also the BGP driver to expose a tenant network or IP we need to go to do this extra hop through the SNAT through the through the OVM gateway port. I was going to comment that on the future work that we are working on a solution to try to make this distributed and be able to escape this and go directly to where the VM is. We play around a little bit with how Neutron is creating these OVM routers because there is actually an option to to create them like fully distributed and and then I think OVM has also an option to do some kind of extra change on the Mac how they is how this is kind of advertised out so that there is no collision for that but this is still working progress and we don't have it we don't have it ready. Okay well thank you and I'll let you get back to your presentation. Yeah so for the last one if that was the actual intended of the question but we can get back after to tweet if not. For the egress I was going to mention that there is not that much dynamicity for the BGP driver there is a little bit more for the VPN one but for the BGP it's pretty much at the OVM BGP agent start time when we ensure that the proxy IRPR and DP are enabled on the OVS provider bridges that we want to use and also we add some extra flow to ensure that the Mac gets changed by the one on the on this OVM provider bridge in the in the way out so this is the only two steps that are needed to ensure that the the traffic leads leads it's able to leave the OVM overlay of OpenStack and it leaves it with the proper with the proper Mac and then in the last slide for me I have some information about what things that we are currently working on one of them is like as you have seen in this slide we are relying on Kernel networking to do this redirection between the traffic arriving to and the VMs of OpenStack and the OVM overlay and this pretty much goes opposite direction of constant DPDK which are you are supposed to do this on user land instead of Kernel space and for that we are working on a way to support the hardware floating and the OVS DPDK and the main idea here is to use OVM feature instead of Kernel networking to do this here we have a figure where we try to highlight this where instead of having the integration bridge usually called VR int of OpenStack that is connected to the external bridge of the provider like VRX we have some extra bridges in between and actually we have one extra OVM controller in there which controls this VR VGP that appears there in the figure and configures it with some virtual topology and this ensures with some IP with some OVM rules and OVM policies that the traffic can be redirected from them and therefore it's everything on either OVS or OVM flows we can take advantage of OVS DPDK for this also other items that we well for this feature actually it took a little bit more for us because as I mentioned before we were intending this to not needing modification in core OVM or Neutron and this feature will actually require some new features that are being merged at the moment in core OVM like multi bridge support to be able to have two OVM controllers running together in the same node as well as proxy IRP enabled by default natively in OVM so we are leveraging those features to make this work then for the other ideas that we also want to say is like distributed routine as mentioned before we are investigating the way to be able to expose these 10 nip's directly on the node where they are instead of through the OVM gateway node in one of the network nodes we are also investigating the possibility of using the phone dv instead of the cell phone dv for scalability purposes but we are missing some information and we are requesting some changes in core OVM to provide them and also recently for the evpn driver we have an api which is the networking vgp vpn for the vgp for the L2 data centers a new api was added based on address scope and the idea here is to also add it for the layer three data center the vgp driver to be able to use address scope as an api to decide what we need to expose and what not and how to avoid the overlapping api's problem here and with this I leave the floor for to Emilian to focus more on the Kubernetes side. Thanks Luis that was awesome and thanks for having me today so now I'm going to discuss about the same concept that you have seen with OpenStack applied to Kubernetes cluster and especially the control plane so the next slide please so historically we have seen the Kubernetes control plane running on OpenStack which would be deployed on a single layer to network domain and when you will deploy multiple racks in your data center the the most common use case would be to stretch the the layer to domain across the tracks using layer two encapsulation so something like vxlan or other protocols but one major problem with that is that the we create traffic bottlenecks and we potentially have layer two domain failures which is all coming from the limitations of encapsulation and obviously the the network latency would not be predictable between the Kubernetes machines in your OpenStack cloud between them so and the last piece is also like I said because you need to extend the encapsulation across multiple fabrics it can be very complex to maintain well to create initially and then to maintain once you have to change the the the racks layouts in your infrastructure so with that the next slide so in we acknowledge this you know this architecture as a like a v1 Chris said before and and now we are looking at what's next and what's next for us is having large Kubernetes deployments hosting critical applications with very high SLA requirements and for that we have been deployed OpenStack in DCN architecture mode which means distributed compute nodes if you if you have a look at the documentation that will be linked at some point we have we have design and architecture where OpenStack can be deployed on multiple availability zones and with a very low latency between the zones like it was mentioned before and the in in the data in the data center you will have at least three independent zones with again spinal leaf which was presented before BGP running in the spine and what I what I want to showcase here is the integration of BGP within your Kubernetes cluster so the next slide this is a very technical slide about how it could work in in practice so if you look at the picture you'll see at the top you see the the spine network and then you will see three availability zones where each of them have a few machines for for the Kubernetes cluster so you have the the control plane and the worker nodes and then you have the the leaf routers and and at the bottom you can see that there are squares for FRR which can run in a virtual machine or could be run also in your Kubernetes control plane cluster and this would be in charge of managing the virtual IPs for the Kubernetes cluster so yeah so the next slide is kind of like from where we come from and and where we are going so again the traditional architecture is more or less what we said before with you know the same cluster will be sharing the same layer too and and this this network would be stretched across multiple domains and initially for example in the product named OpenShift which is the Kubernetes distro for Red Hat we have been using KIPA LiveD and HAProxy as part of the internal load balancer managing the control plane VIP traffic we have seen limitations regarding scalability and also not being able to to to to distribute and route the traffic across multiple subnets so on the right side this is kind of like the new and modern architecture where we are going now where each leaf has its own network fabric and also actually network subnets where OpenStack machines would be running hosting the Kubernetes control plane and data plane and the VIP will be managed by either something within the control plane in Kubernetes or an external load balancer that will live in within the leaf and on that balancer we will use again BGP to to route the VIP traffic for the Kubernetes control plane so this architecture is increasing the SLA it's it's more scalable because the the the layer two networks are way smaller than they used to be in the traditional architecture the SLA is improved as well because if one domain goes down BGP will automatically route the VIP traffic to the domains where the the control plane is up and running and and the last thing I want to mention is this architecture is is plugged to the existing routing infrastructure that we have seen before with with my team where they work on the OpenStack side of things and and this is the Kubernetes cluster but being plugged onto the existing BGP infrastructure yeah so I have time for a demo and I am going to share my screen and I want to show you two things at least so before I share my terminal I kind of like want to show you what I'm what I have in my lab right now so on on on the left side this this blue network is kind of like the the internet for me I reach my my internal network through that through that public network and then I have three three leaves and each each of them has a dedicated network and subnet and actually each of them also has a router VM and load balancer VM which I will show later and for now my Kubernetes cluster is deployed within one leaf and we are working on a feature in Kubernetes running on OpenStack to to be distributed and routed across at least three different domains which is not supported today so in this demo that's why you see all the machines for Kubernetes running within one domain but in the next demo I hope I will show you the machines actually distributed and spread over all the domains so yeah thanks for sharing my terminal and so the first thing I want to show you is kind of like how BGP and the load balancer is is working here so like I said I had I have three load balancers one per one per leaf so this one on my terminal is the one from the first leaf so I have HAProxy running which which handles the load balancing for the for the Kubernetes API and ingress traffic so there is not much to to show here what I really want to show you is the the VIP that I have created and so I have two VIPs as you can see I have a 192, 168, 150 and 240 so 240 is the API VIP and I have FRR running on the load balancer which will help to route the the VIP traffic into my infrastructure so this FR instance is actually connected to my to my leaf router so that's the only neighbor that I that I peer with now and I will I will show you now what happens on my leaf router if I do if I list all the routes I can see that the VIP traffic in this leaf is going through my load balancer and if I go to the spine router if I list all the all the routes I can see that I have the two VIPs where each of them has three potential routes to to be to be reached so I have actually I have three three load balancers running and they have FRR so I can reach my VIPs through three different load balancers so what I want to show you quickly is what if I take down the VIP on my first load balancer as you can imagine BGP will update the the leaf and the spine so let's have a look I'm going to run a watch IPR on both the spine and the the and the leaf so you can see what happens in in live so I'm going to remove the API VIP from the load balancer one and on the leaf router you can see that the route was removed and on the spine you can see that the VIP is still reachable but not from the load balancer one node anymore obviously I still have two of the nodes to reach the VIP if I need so let's recreate the VIP because I need it okay let's close those terminals okay so now what I want to show you is reaching my Kubernetes cluster through a service that I'm going to create so I create a new project demo and this is just a simple web app that will that I will reach through ingress to get the the status of my web app so on the right side of the screen you will see a tcp dump running on two of my worker nodes in in the Kubernetes cluster and those two nodes are hosting the ingress router which is why I opened tcp dump on them and what you're going to see is when I try to reach my application the traffic will be load balance between those two workers and also I will I will highlight that in in my terminal if I can but I want to show you that the traffic is coming from different load balancer and I will show you what happens if I shut down the VIP on one of the load balancer all right so I'm going to do a curve so you can see that my app is alive and it went through the first worker as you can see if you look at the logs you can see that it's the the traffic is coming from the spine the spine router which is 192.168.68 so that's the IP address of the spine which means that it's coming from either load balancer 2 or load balancer 3 if it's coming from load balancer 1 you will see the IP address of of the of the of the leaf which is the case right now you can see 168 sorry 192.168.10.2 it's the IP address of the load balancer 1 that you see at the at the top right of the screen so this is load balanced as you can see and now I'm going to shut down the VIP again and you will see that the traffic will only only come from the the spine network because it will use the the the load balancer from the other leaves so it's coming from 68.88 which is the the other load balancer now if you if I bring the VIP back and you can see here that the the the last request coming from the load balancer 1 so again in this demo this is limited by the fact all the nodes are within the same failure domain and we are working upstream to implement the support of Kubernetes deployed on on multiple failure domains on OpenStack and and and the next demo you will see that the masters and the worker nodes for Kubernetes will be spread across those those domains and and BGP will still be used for routing the traffic exactly where it's available so I hope you like the demo and I guess we have some time for questions now. Hi I guess we do so if everyone can go back on camera it's almost everyone I think we have time for yeah just one question that was a great demo so thank you for sharing that Emilian so the question that I was going to ask and I know we had some interaction on YouTube and someone linked in as well but what so is this architecture available today I know this was kind of covered in the beginning and how can folks get started if they want to implement some of the things that you all talked about. I can go ahead for the for the agent I mean upstream we have some documentation in there in the repository of the OVN BGP agent about how to actually deploy and configure the agent itself with the different drivers different documentation you can use that for playing playing with the with the agent itself then there is some I mean there is no at least like a single doc that covers like all like for instance open stack deployment and this type of setup plus the agent plus what Emilian just has has shown there is no such a thing at the moment okay yeah and then so the functionality is implemented in Wallaby right this is where we introduced it so from the software perspective it's it's all there at least from the from you know from the open stack perspective and then we presented these three kind of versions and you know each of them has a pros and cons and let's say version one and version two they are available and they're actually being deployed by many open stack users out there. Cool and I mean that makes me ask so I know one of the things we talked about previously like is this only for open stack use cases or is it for other use cases as well? For I guess for Kubernetes we have seen this exact use case on other platforms and you know in the Kubernetes space we are working with the community not just open stack community but like the whole community to understand you know for example they can deploy Kubernetes on AWS and they already have this feature of being able to to distribute the cluster across multiple areas so that's that's what we are trying to to implement now with the open stack drivers if I can say yeah and I think the FRR is we're not the first you know that they're taking advantage of it there is another project in Kubernetes called Metal LB which takes advantage of the FRR to load balance the you know the workloads or the traffic coming from the paths on top of the Kubernetes so there's definitely ways to take advantage of this tech today. Yeah and for the agent it basically just talks to OVN so it's based on having an OVN cluster it doesn't matter if it's open stack, open C for extra things so it's just a matter of adapting and building your own driver for your use case in there but it's your work. Awesome well and it's an interesting thing of just combining all the different open source components which makes it a true open info story so I did want to I know we're out of time now but I did want to thank you all for sharing this not only at your local meet-up but also here on open info live I know we got a lot of engagement on different platforms so it seems like a popular topic that folks were interested in so thank you all. Yeah thanks so much for having us. Of course and before we wrap I want to promote we do have if you want to hear more talks like this I hope that you have the Vancouver Summit bookmarked in your calendar it is coming up June 13th through 15th where we'll have lots more demos a lot of technical talks and production use cases so registration and sponsorships are now open so we hope to see you all there very soon the schedule will be going live and I think just around three weeks all the track chairs are hard at work right now selecting sessions from the hundreds that we received so we're on February 8th is when we're targeting to publish the schedule so keep an eye out for that and then next week we have another episode of open info live where the large-scale SIG series is going to bring on another ops deep dive so we'll hear from Ubisoft about how they're running open stack at scale including some of their challenges some of their successes and they'll be open for questions there as well if you have ideas or technical demos that you've created that you'd like to share on open info live you can reach out to us at ideas.openinfo.live and share what you'd like to do or nominate someone else who you think has an interesting open source story as well we're here on open info live on Thursdays at 1500 UTC so we'll see you next time. Thanks everyone.