 Hello everyone everyone can hear me even in the back nice Thanks for joining in our session. I am really thrilled to be here and I'm Francesco Grimaldi team leader at desu tech Italian company like like me and Today we are here To talking about service the the BGP with Calico and Metal B I'm a team leader at desu tech. I'm not only manage my exceptional team, but I am With my team we provide the consultancy and trading about cloud native technology and Today we are here with my colleague evangelista Today we are here to shed light on How to use BGP To expose our application in kubernetes using Calico and the metal LB and as last thing about my Introduction, I'm honoured to be an authorized instructor for the Linux foundation this given to me the opportunity to share my knowledge and to train the students Hello everyone, I'm here with the same session obviously. I'm a journalist at Ranghi I'm also a manager for AI team in desu tech and we work as authorizer trainer for different company being company For example, I'm authorizer trainer for AWS AWS cloud and then Linux foundation egg and VMware So this is what we do on the day-to-day Like we said, we will start now. Okay, it will appear a little bit difficult with the next slide But let's explain it going going over Come on. Enjoy Let's take a look about the outline for today and Let's take a look about this light Can appear may can appear complex But today we are here to break it down and step-by-step To reach the final outcome Before to deep dive in in our setup in our solution Just take a quick review about our cluster kubernetes that will host our solution We have a kubernetes bunny a cluster at version one point twenty seven point six and We have a cluster comprising total of six nodes Three control plane and three working nodes We bootstrap it to this cluster so We bootstrap it to this cluster using a cuba dm that simplify the management and installation proxies and The nodes are virtual machines Okay, running in Ubuntu Linux as a guest overriding system We use a container D as a container runtime and we opted for Calico as CNI Take care about the worker IP nodes because will become important in the next slide and What about Service load balancer on kubernetes bunny a cluster unlike managed solution provided by cloud provided in Kubernetes bunny, uh, we don't have kubernetes one here don't support service load balancer out of the box So this means that we need an extra component extra configuration that will enable has to to enable this kind of feature and to use a service load balancer and Here tools like give VIP or metal LB comes into play service load balancer allow to expose our application outside of your cluster and Probably is better than classic node port because If you use a simple classic node port you probably in a production environment You need an external load balancer and an external balancer means more operational overhead Like tools like QVIP or metal LB Can work in two different manner can work in layer 2 mode or in BGP mode So what's about the layer 2 mode? Let's try to The cluster is a kubernetes vanilla cluster in a bare metal. Okay, so No, it's its own premises The cluster run on a vSphere environment and like the nodes are virtual machines running a Linux Ubuntu as a guest operating system and What about layer 2 mode? What are the benefits how works the layer 2 mode? Yeah, is it to install? Okay, it's a no complex config is a pretty universal solution. We don't need Special natural devices and no fancy router It's rapid to implement rapid or fast to implement What about the the drawbacks of layer 2 mode? In the networks some networks some environments can be Garp or style gap stand for gratuces harp that is the protocol the layer 2 mode Used to orchestrate the failover which fell over when we work in layer 2 mode For example, if we install metal LB in our kubernetes vanilla cluster Will installed System pods called speakers and among these speakers will be elected the one node as a leader and the leader node as a speaker the speaker leader, okay Have the remain rule the ability to intercept the external incoming traffic Okay, and root to internal traffic Imagine that the speaker leader node multiple IP configure on its own interface announced to external and And You have to know that when you work in layer 2 mode That is a big drawback that we don't have a real load balancing because as I mentioned here Lee You have a one speaker elected as leader that is the only node that can intercept the incoming traffic So this means node bottleneck Not bottlenecks means that all the traffic pass through these nodes and for example the bandwidth is limited to node capability in that moment but surely if the Leader node go down or maybe become unavailable Okay, there is a mechanism to failover and to elect a new leader That will manage the traffic the incoming traffic and Other problem that can we meet on layer 2 mode any other drawbacks can be for example Problems with updating ARP cache about the underlying network devices, okay Because the speaker that use gratucious harp to hot to update our cache of the network neighbor to update the cache and so to make the failover to work the failover and Another problem that can we have maybe we we work in the in the network with some latency one natural bottlenecks and so we can have Problems do during leader election and problems during leader election means possible them time because remember If you don't have a leader speaker a leader working Your application are not reachable. So the question now can be bgp is the right way Let's find out and This is a just a quick recap That where we are now as you can see we are a kubernetes cluster kubernetes vanilla cluster three control plane node three worker nodes okay, and We can proceed to the next step Calico as I mentioned it before we use Calico as CNI Calico is probably the the one of the most famous CNI in the cloud native word, okay, it's a pretty adopted and Calico is Is a pretty modular can be configured in multiple ways Can leverage can leverage on different natural features like bgp routing protocol Like vixlan or IP IP and capsulation protocol to build your overlay network to your cluster and In our setup in our kubernetes cluster we installed Calico Calico project using the official elm chart as you know elm is the package manager for Kubernetes that simplified installation process and the management About our deployments, and we here we use the official elm chart to install Calico As you can see we customized the installation with that Calico values dot yaml, okay, and as you can see we are enable we are enable bgp Okay, remember that and we are enable vixlan as a protocol to build our over overlay network in our cluster Okay, and that is the pod cedar block for our internal network for the cluster Times for demo just a quick demo about Installation process of Calico as we said we use the helm here We are installed elm using snap and After installing helm we can prepare our Calico values That is important because we want to customize our installation take a look We enable the bgp that we use in the next part and We are using a vxlan as a protocol of encapsulation for our overlay network and that is the The pod cedar block for the internal subnet of our cluster Here we are adding the official repo for the elm chart official repo of Calico and Now we can Update the repo and we are ready ready to go. We are ready to install Calico So we can use the command elm install Okay, just we can Set the version that we would like to install in my case 3.26 And now we can install Calico passing the our values file YAML file that will customize the installation based on our requirements nice as we can see the is deployed, okay and We can go to the next step as We as we mentioned before the load balancer is the service that allow has to expose Application in in our Kubernetes cluster as I as you can see in this high mage We have the load balancer, okay that we catch the incoming traffic and will run out the traffic inside our cluster Okay, so what about the metal it will be as I mentioned before metal it will be is a one of the tools that can enable service load balancer in Kubernetes bare metal cluster Kubernetes Vanilla cluster and Metal it'll be in in their Infrastructure have a two main components. The first one is the controller that have the rule of Assigning and allocate External IP for the service load balancer and the second rule the second company is the speaker One for each node deploy the tanks to the the demon set Okay, and the speaker is the component that have the rule to route the traffic the balance the traffic and the speaker can work in BGP on in layer 2 mode, but in one important thing because in our setup is a little bit different in What in our setup? We are leveraged on Metal LB just for the controller component. We need the metal LB Just for have a component that can help to allocate and assign IP address to our service load balancer. In fact, as you can see we deleted the demon sets of the speaker Okay, we delete that we don't need that Okay, because as we can see after we'll be calico to manage the traffic and Important we need to label all the control plane nodes in the slide You can see just the control plane zero one, but we need to execute this command on all three new all three Control plane nodes to exclude from external load balancer and The the configuration for metal LB in our setup is so simple. We just need to provide As a YAML manifest as a CRD for the the metal LB the the network that we won't dedicate for external traffic In my case can be for example See the block slash 24 or maybe a single P depends on our network Requires in my case. I use it even the single EP and even the entire block Just a quick demo to how we installed them the metal LB in this time. We choose to install Metal LB using the official YAML manifest provided by metal LB projects and we apply the manifest and At this point we can The manifest will install the system pods the controller and the speaker But as we see in the in the slide, we have to delete the speaker But because we don't need the speaker in our setup and just we have to label Okay, we have to label all the control plane nodes to exclude from the external load balancer Here we can pass the configuration for the IP address pool like we have an object kind IP address pool that is a CRD provided by metal LB project and Here we are passing an entire 24 blocks network Okay, that is that is a will be used for from for the external network Okay, and now as you can see we are Qubectl We are executing this common Qubectl apply to To create this object that will be read from the controller that will use this configuration to enable the controller and These are the same here. We can Enable just one AP. Okay, because I can tell to metal LB controller I pass a range of IP and try block or just one single AP that you can use So as for now The our infrastructure is made by a Kubernetes cluster. Okay Calico as a CNI component calico node on each node of the Kubernetes cluster and only the metal LB controller so the next step will be the real engine of our setup that is the BGP configuration for this BGP configuration you we need to understand what is BGP So BGP border gateway protocol is the protocol or use it to route traffic between autonomous system Okay, what are autonomous system autonomous system are a simple point that can Route traffic. Okay to reach our destination like internet on the worldwide network So if we think like this our network our infrastructure setup needs two autonomous system One is the router itself. The second one is the Kubernetes cluster Okay, the Kubernetes cluster. So let's deep and just one second in wires in our setup Like you can see based on the cluster only the worker node will made a BGP session Real really the calico node on each worker node will made the BGP session So our destination will be on the router. Okay will be root with a lot of path All the Kubernetes worker node made this path So let's start with the configuration over the router. Okay the first autonomous system Obviously, it depends on which router you are using. We are using a bias router So we use a set protocol BGP command over the router to make these autonomous system connection It's called it appear between autonomous system. And then like you can see all the AIP address of Kubernetes worker node are attached to the same remote autonomous system the real engine for this configuration is The configuration of calico. Okay the second autonomous system of our setup Calico enable different resource API for our Kubernetes cluster to manage and configure BGP We were we are using only three API three API. So BGP configuration is the first BGP filter It's the second one BGP peer is the third one What is BGP configuration BGP configuration as API in Kubernetes for Calico declared the Autonomous system number of the Kubernetes cluster and it also helps to The to declare with seed or block we need to advertise to the router Like you can see there are a lot of parameter in this BGP configuration We are showing you only service external IP a service load balancer IP What is the difference between these two the service load balancer IP read from the parameter status load balancer IP? So it's the IP assigned by the EPP pool Service external IP can share also single IP. This is the reason why we have used a two different EPP pool in metal LB because in this case with a subnet mask Slash 32 we can share a single IP. Why? When you when you use a setup made by Calico and metal LB you can't declare With service discovery you can't you can't declare a single IP Without using a slash 34 in the BGP configuration So when you declare a slash 24 for example, and you create a single IP for the external Load balancer so external IP for the load balancer You will not get a single route on the router But you will get the full subnet on the router because in the BGP configuration you are specified Slash 24 for this reason we have choose to show also the service external IP that can Can advertise a slash 32 Okay, so you can have a single IP for example if you use ingress ingress controller You will get the same IP For the next few years for the service load balancer for the external for the ingress engine x ingress contour and so on So you need to advertise the single IP to get a single route Okay over the obviously autonomous system remote of your setup But you need to understand that Calico with BGP enabled has also the pod seeder Okay So if you don't use filter on your configuration Calico will share and will advertise also the BGP the CNI of pod subnet on your cluster How it will share each micro subnet for each worker node because you know Calico can split The full pod subnet in each node So if you want to if you don't want to use this to have this you need to use filter like you can see We have made a reject there that can Obviously reject all the pod subnet The last configuration the last API we use is a BGP peer configuration. It's a BGP peer API that can Connect the communication with the remote autonomous system in this case via through there Okay, like you can see we have used the autonomous system number 64,513 for the Kubernetes cluster 64,512 for the router virus So Going on on the next slide. Well, you will see our configuration. I've also explained this step So you will see this made in the router and Kubernetes cluster and you will see the difference between the root propagation This is the I've explained this because it's outside of our presentation. So like you can see this is the command for the BGP configuration on virus router Okay, we declared the autonomous system of the router the worker and the autonomous system of the Kubernetes cluster Then next on we repeat this for each worker. Then I will show you let me move here Okay, like you can see we are repeating this for each worker node under the same autonomous system Why because I've said before in our infrastructure. We have only two autonomous system The first one is the router. The second one are all the free worker nodes So all the free worker nodes are the same autonomous system inside our configuration This needs to be clear. So now I'm saving the configuration Then I will show you how is the our configuration can show for this BGP protocol all the free path. So worker one worker two worker three. Let's move on on this Okay, as you can see all free neighbor for our configuration the last thing I want you to show here is When I show the root Here, let me stop one second like you can see on the router itself There are no BGP route if you read this this output the bee Declare a BGP route. There are no route now because we need to configure the cluster of Kubernetes So the calico site to make it to advertise So let's move on now. Okay The first one is the BGP configuration Obviously if you start with a clear cluster, you don't have this kind of of API object inside the cluster created So you will start creating it like you can see it's the same that you have seen in the slide So the same parameter to EP pool assigned also to metal LB 64,513 it's the cluster autonomous system then going on It's the BGP filter obviously where we declare obviously there is no one in the cluster. Okay, we created Where we reject all the pod subnet. It's a simple post post subnet declared in calico before then moving on The third component will be the BGPP Let's look at this obviously no BGPP you're configured in a clear cluster Yes, as you can see calico has a CLI for this for this kind of things in fact In the state of the fact, sorry for the Italian word state of the fact that you can say Calico you can read Calico city Okay, there. There's a CLI dedicated for the do this kind of configuration the same as Cube city. I'll apply the same Okay, now all three things are configured like you can see I get do a get for all things all the free API Yeah, and next we'll move to the router and let's see how in the same output the same command we get Both the root Let's look at this Yes, this was the previous one. Okay here. Yeah Just for a moment. It's not placed up Yeah Like you can see as for now. We have to be Declare it both the CDR we declared in the BGP configuration. Okay, and for each of these routes We have the three path worker one worker two worker three. So the three worker node of our Kubernetes cluster So as for now the configuration is made like this So we added the two configuration. We have a BGP session between Kubernetes cluster and router The next step will be let's try to Debron new application create a Kubernetes cluster you we will use the single IP so Slash 32 subnet mask for the people and Then we will try with a new VM to contact these IP address For in this in this light is important to know that the everything is connected to a different subnet Okay, the router has a two Ethernet interface Okay, like you can see one is on the node side So 10 10 10 and so on 10 10 10 0 the other one is a new network Okay, we can see it's a different network that where is the client VM for example, so Good to point to know that The Kubernetes service external IP is taken from a new network. So a third one. So rooted network Another one, let me add something so you can see it on the screen next to the video a client VM When he made a request you need to be configured with the the router as gateway Okay, if you pass the router as a gateway, you will see all the route propagated to the router So let's see that. Let's try to deploy Okay, let's start it obviously in Kubernetes you start with a new brand new namespace Okay Next to the ploy of the application. We will use a simple front-end. Okay a custom who am I okay? That's served also parameter about the client We serve in the request. Let's let's look at this I'm skipping all the comments so you can see a full comment directly A new deploy okay single pod. We don't mind it. So with the front-end simple front-end And next we will create the service load balancer Yeah, obviously them in space. So a new brand new deploy Okay Okay, like you can see all the chain is completed. So the ploy replicas at the pod and so on So let's try to expose this one with a service Let me add the comment Okay In this case, okay, we are creating directly a service for the deployment. We can do it also with the manifest YAML Okay, we need to pass two parameter and up until now type load balancer port 80 for port for the application and load balancer also Next step when we try to get a service and get a result of this operation You will see that the new the brand new service as an EP taken from the EP pool of metal a little bit Okay, so obviously namespace is missing here. Yeah this one Okay, as you can see we he has taken a 90p from the EP pool of the slash 32 Okay, but you can customize it because metal a little bee grant you a brand new Annotation to select which EP pool you want to use Okay, so you can create a different EP pool and choose from which EP pool you want the EP address the external IP This is the annotation I'll show you that this one Few seconds as you can see if you take the load balancer you can use this Annotation directly or in the manifest YAML what you want to select one of the EP pool In our configuration one is default the other one is single IP So as for now, let me Come back because it's going over on his own Yeah Okay, now next step will be I will take a new machine this one and Try typing in it The external IP Okay, so 10 11 140 50 as you can see from the client VM So I have brand new virtual machine on the other network I can connect to the application in this in this output You can see the name of the pod, but I will do the same command from CLI to let you know Which IP is a replying to my request? Let me show you just one second Yeah, this one as you can see inside all the parameters you get he got is the name of the pod the IP address of the service load balancer on the request and On the last part of this output the worker who is replying to my request So the worker The worker one I think is that one yet 10 10 11 is the worker one is replying to my request And obviously in the first part the IP You can see the IP of the pot itself. So all the chain The worker node the service load balancer and then the pot all free EP obviously So as you can understand the BGP This is only a small configuration a small setup in BGP. You can have multiple multiple Autonomous system with multiple BGP peer configured with multiple BGP filter Okay, and you can create a brandy new brain network as you can understand to share and advertise all the EP address So BGP obviously is so wide to understand Okay, so to talk about it's not a 30-minute talk But as for now, we have done a full architecture. Okay The client VM request for the router as a gateway and then connect to the application Okay, I hope you enjoyed this this little talk. Obviously, it's it's our first time. So I Need to to to share that I was a little bit anxious before Obviously, it's not my first languages. You can understand also that so it's pretty difficult another time But I think I hope you enjoyed it also. I enjoyed it I will try again to better improve my all everything But if you liked it, I will be glad to to understand each question and reply to that. Thank you Thank you. Thank you. How would you recommend automating? Sorry? Can you yeah? How would you recommend automating the BGP pairing? Is that something that's built into Calico as well? I mean traditionally you would do it with infrastructure as code, but Okay, what would you recommend for a deploy? You you are asking that you do infrastructure as code So you are asking how to automate the beer between the autonomous system in Calico. Yes between the router and Calico Okay, I think if you if you look at a production environment Okay, you will have all single IP. Okay, it's not so with metal LB and Calico It's not twice to use a more than 32 subnet mask Okay, because you will have a single route So I think you can create all BGP configuration You can have different one because we have created the default one But you can create different one And then it's a simple like you can see it's an apply Okay, so it's a manifest to create customized the manifest you can use different tool For example customites it itself the key of Kubernetes and then you can try automate try Automate this kind of process with the we whichever tool you want But it would need to be automated outside of the deployment you've described is So so can you can you it would need to be automated outside of the deployment that you describe? Okay, you are saying you want to automate inside the deployment itself. I was just wondering there is a solution I think it's a I think that the BGP configuration is an administrator task. Okay, you need to take a step back Okay, so the administrator itself need to configure all the infrastructure. Okay, and then I think the developer will The operator whichever you want can do the next step in the pre-application We'll use the people of metadata be you the administrator Configure it. Okay, and this is a separate question. But does Calico support BFD? Yeah, Calico only Calico enterprise support bidirectional Support the failure detection. Yeah. Yeah In fact, if you use project Calico in Calico community Yeah, probably you can have a slow failover based on the timeout. You can adjust the timeout can be tricky I know but if you to switch to Calico enterprise, you can have this kind of feature Okay, thank you for your help and fantastic presentation. Thank you. Thank you Good presentation. Thank you very much. Thank you. Are you familiar with BGP anycast? Do you know that term BGP? So you cast this anycast? Okay. Okay, it's very it's reminiscent of BGP anycast and with anycast they It's you could use UDB based traffic application, but I saw that you did TCP Yeah, how does it not break that the three-way handshake like the TCP three-way handshake, okay? you you can understand that it's a Lot tricky in this case But you can also inspect better configuration for the BGP API resource, okay? I'm sure you can find some section Customized that you can customize to to to make this a configuration. Okay. It's a lot more bigger Got it. And then when you did the curl I noticed that it was node one that responded if you you repeated the curl will always be no one Or would it be randomly sorry sir? I don't understand a lot when you did the curl command You did a curl to figure out which node responded. It was node one if you if you ran it again would always be node one Okay, it's a it's a simple you can add a simple equal cost multipath in this configuration So you will have a different worker or responding for your sorry. It's my preference. Is that how you did it? No, it's a it's a simple next step on the router. Okay equal cost multipath You will add on the router that as the route itself so you can add the configuration there Thank you, sir. Thank you Yes, can you can you approach the microphone? Yeah, sure. What happens if you lose the pods running the controller? The pod yeah, you which one the calico node or the metal LB controller Okay You need to understand that metal it's a young project. Okay, so it's a beta project also So in a production environment, it's a little bit tricky Okay, so but obviously it comes with the I think it's it's not a problem because when The controller of metal a b gives you like any pump give you a single IP then ends its work Okay, so if you lost a metal a big controller, you don't lose the external IP Okay, this is important because the bgp session is maintained by calico node itself Then also the metal a little bit control is under deployment So if the pods go down you will have a new pod obviously Metal a big control is a normal pod. It's by default. It's not privileged pod so it doesn't go on Kubernetes control plane, but it's allocated on a simple node the work or not So if you in fact when you deploy it it it can go on One of the worker, okay But you can obviously customize it with a node selector whichever you want and you can grant privilege to the Kubernetes to the metal a b controller. Okay, but by default it's not like this. Thank you Yes, thank you Thank you everybody