 Thank you. Thanks for the intro and welcome to KCD Chennai and I know that you guys are busy with Saturday course, but still thanks for taking your time to attend this fantastic event organized by KCD team. And I'm very excited to present to you guys about talk about deconstructing Kubernetes for networking and knowledge profit or make one call it as a profit right. So this is based on the Microsoft internal training presented by Stephen Griffith. We enjoyed this training and I found it it's very useful for me personally. So I thought, you know, I don't share this information to the outside world. And the most of the things are all open to Kubernetes and wherever you see AKS, I try to keep it as neutral. But some places you might see AKS let's say the VMSS scale set or AKS but still the commands and the constructs are the concepts are same across the Kubernetes right. And let's let's proceed with the sessions since we have a 40 plus slides I'm trying to condense into 25 minutes session and then see how best I can know responsible on this provided time. And the session objective is, you know, I'll set the context by introducing the layers around the Kubernetes and starting with the, what is parts what is services what is components on their own the networking and then you know we'll dive into the some of the networking constructs like you know how one can approach the networking problems are solid issues and you're not attempting to troubleshoot right and then quickly I'll walk you through with the steps and the tools I used over a period of time to achieve the required findings or maybe the solution to make it work. See the, all the tools and the steps are you know, put it in a graphical way. The reason is in the presented time it's so difficult to set up for the whole thing and record as is. And Kubernetes network. First of all, we have to agree that accept that Kubernetes hard right. It's not like I'm a certified I'm a master in the Kubernetes area right. So we have to accept the fact that Kubernetes hard but network isn't magic and also network is also equally hard. And we see many people shy away discussing about the networking concepts because of the complexity involved. And at the end of the day, it's all wild zones to you know, simple IP tables and build with a lot of overlay networking concepts around the IP tables and then we have the CNI or the Kubernetes you call and we will deconstruct the whole thing in coming slides. Okay, so the Kubernetes networking as a key concepts are, I mean I'm going to cover the some of this key concepts right so the first thing the fundamental thing is about the cubelette. This cubelette is responsible or the agents are a demon set responsible for certain things right so there are like 145 plus parameters you see about in the after the docker sheen removal in 1.24 is expected to have 120 parameters but still it is a huge but so if you SSH into one of the nodes, one of the Kubernetes nodes using the PS command you can see this you know what is the network plugin used in the underlying architecture right. So you see that briefly in the diagram you see a network plugin as a cubelette in this case whereas you would also see this CNI from the respective provider if it is an AKS then Azure CNI or other provider also will have the equivalent you know CLM I think there are like 15-20 providers you provide a CNI network on top of the default networking and these agents and the node agents of the cubelette is responsible for various things and I suggest you take time to go through it and then we will proceed with you know how you can actually log into one of the SSH into one of the nodes. The commands are same still maybe the cloud provider is different but the commands are same and you could use one of those in a box and then try to run a spin a you know troubleshooting the container what I see the mcr at microsoft.net and then there is an image which I am using to troubleshoot and then you can run the top command or atop command and then see the container DE these are the some of these agents in the cubelette also you will find it as a first thing here and you will find other agents are equally running in those machines. And those are like runners are demon set or sorry the agent or a process so they are responsible for certain networking concepts which we will see in a moment. And the key components around the Kubernetes networking it's all most I mean most mostly revolves around cubelette or a CNI right by default we started with the cubelette when the community started but over the period of time due to the limitation of the cubelette. Now we are branching towards the CNI are a layer on top of the default one and the respective providers like you see the Azure has got their own and GCP AWS and VMware and CEM and there are like as I said the 20 plus networking layers are CNI plugins are available to achieve the some robust networking concepts around you know the default one. And the parts actually it's been talked a while parts of the fundamental constructs are very low level compute execution of your application and it's it's all. It could it can have a multiple container or maybe as a side card proxy yesterday we talked about service mess you know as a side card proxy though these parts were responsible to run a container manage this containers and also manage the IP address for this containers so. And then we'll jump into the services also services also you so if you look at the the way the slides are organized right so I'm trying to get deeper deeper and then show you the you know how you can actually slice and dice these networking concepts. So when we drill down to the service level service levels you know there are four types of services cluster IP node port and the load balancer and external name. And I believe there are one more is also there but there are four basically four types of majorly you know use things. These are like it operates in the layer for it means that it's it doesn't know anything about the application in the back and right. It's it that's what has been prescribed here let's say if I create a cluster IP gives an IP of no cluster to private IP in the cluster to work with a little safe I spin. If I define as a load balancer it's me a public IP from the respective load balancer I mean sorry cloud providers load balancer and then and it's kind of abstraction and you should know how to run this command as a cube CTL get SVC would take you take you there. And then the second one I mean the next one is the endpoints when you run the command like a cube CTL endpoints and parts come on dash or dot wide. You would see the back and the I mean back is having a single IP address are endpoints whereas the front has got four different parts and there are like you know four different copies of the parts and then you see that the IP address are you know in the range of 10.24. I mean the idea here isn't it matches the endpoints what we see it in the command and also the parts what's been assigned from the Cuban it is right. So, so how this matching is happened I think if you look at the notes I provided I say it's a quick reminder so when you define on the Amal file in the selector label it's try to associate this endpoints with you know IP address what we have. And ingress actually this works in a layer seven it's been talked about I think a lot of ingress available in the community space and see that ingress control by default we in Azure we provide Azure application get the ingress controller you have a choice to run engine access and ingress or a traffic or on why this will essentially will help you to route the traffic in a designated way it works in the application layer that's very important concepts you should be our often the communities network. And then let's deep dive. As in that you are you are an SRE development and sorry and the DevOps engineer you are going to debug some issues right so first thing that the networking concept starts with you know the Amal spec what you see what is the port what is IP address defined how these selectors and labels are you know mapped each other right. In the right side there is a kind of service which has been created for the deployment. In the bottom they say there is a selector labels it's matching as a flask application dash a right so to the service been created on top of this container this part the replicas it is nothing but the copy of your part and they and then if you see. One fine day you get an issue saying that hey I'm trying to do a curl command. Maybe in the left side you see it's working fine but we're in the right side. It's failing right so one of the part is failing to you know maybe respond in a way that it's showing is an operation timed out right. It is a normal thing which you might have seen or come across so that the thing I'm trying to say here isn't how would you approach this problem or troubleshoot this let's deep dive and then see. In a high level if you look at. I mean there's one thing you have to be very aware here is the cube net you will not I mean in cube net you will find row tables created by default. And then but in case of a CNI networking and you will not see that row tables created right I mean you have to be very. Mindful about whether you are troubleshooting a cube net issue or a plugin or CNI plugin right so based on that be the underlying network stack of the the approach would differ. So in this case of I mean in this case I mean here in this case we are trying to. Check what are the I mean various knobs off in the networking complexities the fundamentals are same again. Don't get away by the screenshot of Azure or a case but so when you get into the system or when you get in I mean when you look at the resources what has been created by the managed provider. You ought to check the default route that's going via the firewall or getting blocked by certain things right you always have a. Networking concepts define I mean the route tables configured to separately for ingress and egress so make sure that you know they there is no overlapping IPs are created this one of the I mean these are like some of the checkpoints you try to check but when you further. Be bound to check right so there isn't that shouldn't be an energy created or associate to the the V net by default when you. Create a cluster in the you know a cubanet right in the left hand side left hand side in the cubanet I mean the left hand left hand side as you know the cubanet by default you see that. The energies are attached to the subnet but when in case of CNI you know it's been attached to sometime in the V net or in the node level right so make sure. Nothing has been assigned to the subnet so the default setting up the subnet sorry energy changes depends on the the plug in what he choose make sure these you know energies are not assigned or maybe propagate to the top to bottom make sure you know the energies are not blocking your access right. We have any when you drill down further is again the one more we have to know to check it in the Azure so you go to the particular resource group and then see whether the energies have been blocking that access endpoints and then it's confirmed that doesn't seems to be blocking anything. And then let's deep dive into cluster by you know get into that machine and then start troubleshooting actually what is actually stopping that a curl command which we try to troubleshoot so far we were troubleshooting or maybe looking at. What's outside of the cluster by looking in the portal NSG or maybe you know the definition what we said in the you know start of the cluster in the spark spec and other thing but in case of when we get into the cluster and you're make sure. The fundamental concepts as like you know the the cubanet versus CNI the concepts are clear right see as I said the CNI in CNI. You will not you will not see the route table created but in case of a cubanet you will see the route tables created to make sure you know the things are improper. And also these are like kind of a definition I mean maybe the difference but I'm sure that most of the things are same as you know even though the providers are different and. Yeah one thing actually I want to say here the networks I mean the cubanet the fundamentally the cubanets are the network concepts are put in a way that you know it's it's actually nothing but an overlay on top of your. The network provided by the nodes and in case of CNI you can define a range of a subnet range and also you have a flexibility to set you know let's say if you are setting up. 30 IP address per node and plus one node one IP for that particular node 31 IP address fundamentally the CNI would operate within that range. But in case of next version of CNI I believe the dynamic allocation of IP address also coming in picture so I think we are most I mean the focus here is mostly into CNI because. I mean if you look at the cubanet it's cubanet maybe we all started playing with the cubanet but when you get into the production grade application the CNI is what I see or we see are mostly used are majorly considered right the CNI version to which is going to be a more dynamic allocation. So I encourage to check CNI version to also and then as I mean as we discussed the cubanet versus Azure CNI in the first diagram in the top in the top most when the node IP address is like 10.10.504 5.4 has been. That's your nodes IP address but in case of I need actually okay yeah so the the parts I mean for the Azure CNI the IP address are assigned from the subnet CIDR you can see that 10. I mean you can see the difference like there's a 1024 in the top whereas 10.10 is in the bottom and similarly for the service side and the service that also it changes. But the external load balancer remains same both are going to be having in the same it's a public IP address at the end of the day it's a public IP address the moment you create a service type of the load balancer you get a public IP address and then there's no difference here. Let's let's get into the what's actually happening right when we do a curl command like here. I mean we got an operation time right this is a problem statement we are trying to troubleshoot so so that's a public IP address you can make sure the public IP address from the respective providers are in the proper and in the right format. But in the case you know we are troubleshooting with the a case cluster so I'm I'm trying to make sure the Azure load balancer you know functional and also properly configured and the ports are listening. And then 5000 is a port where we are trying to reach and then in the side by side in the screenshot you see 5000 is also configured back and is also 5000. The slide I mean from the Kubernetes view. Kubernetes I mean doesn't understand whether you are running in operating with the AKS or AKS it works in a way that because it's all other things are put as an abstraction or layer around Kubernetes are extended. So the Kubernetes view you will see these load balancer having the IP address and also ports with the incoming and outgoing and in external IP. So that's a service view. And the Linux view if you access it to the SSH into one of the machine. What is the Linux kernel view looks like for this. So if you do a pseudo IP tables of dash D and that dash NL and cube services grab up for that application. You will see that the game and set and the tool chains are there were multiple tool chains and then finally it reaches the destination of 2081. That is our load balancer IP right. So we're trying to see what I mean how many tool signs are you know how many hops it takes to not hops actually the tool signs are involved to reach the destination. If you introduce an additional NVA and then you would see that additional tool chains also gets added here from the Kubernetes view. And if you do the service in points and surveys and you will see that things are matching and reports 5000 is what we are looking trying to reach and which is all properly configured. And then again with the Kubernetes view and then from the end points it actually reaches the our managed cluster here. The case no right. Inside the case note that is a part and then when you do this cube CTL get end points and come apart. You see that the IP address of the end points and parts assigned IP address are matching. So so far good and then get into the Linux view kernel view and similarly in the Linux tables. I mean when you I mean the same Linux node when it tried to do this IP tables and then the commands of Cube service with some height and certain hash. You will see that the the cube dash step which is actually matching with the IP tables of what is that net listed right. And also the part IP is also precisely matching. That is that's a destination we are trying to put the traffic to write. And then there's another view. So when you do the CRI CTL PS you will see that the container name that flashed application which you are trying to curl. I mean the hash of container name and then you can also CRI CTL inspector dash dash output and then go to the template of the particular container ID. And then try to see that you know what's actually the loopback address and other things by the way these. I will see here the slide these commands are properly put it in a way that you know you can try to do the same thing on your own. I will put it in my GitHub so that you can grab the slide and then all these steps are you know we are trying to get deeper and deeper. And then it's OK. So far good. Have you have you confirmed or does it work in a way we expected. No still the still you know the the timeout issues are happening. And then we see that we are trying to get into the machine and also we confirm that and the ransom commands and then we confirm that things are fine. And then there is no firewall issues and there is no energy issues here. And we looked in the kernel view. Sorry the Linux kernel view looked in the Azure view or a provider view. We looked from the QCTL endpoints and service view and but still the nodes are not receiving traffic rights. I mean the timeout issues I mean the timeout errors are not getting solved right. And there's one more avenue we can check. We can also check with the load balancer metrics whether the particular part which is running inside the cluster which is receiving the traffic or not right. And you see that in the load balancer metrics in the bottom towards the bottom there are like a couple of red marks are there that shows that. And there's somewhere you know something is breaking with the load balancer. I mean the traffic is not actually really hitting the load balancer or reaching the road balancer right. So we could also use the I mean we can now use the tools like a wire shock and by the way in the wire shock is the UI version. But when when we use the T-Shark which is equivalent to the command line version of wire shock we'll be able to specifically set filters. Let's say when we right now we are doing a curl command that's OK but we can actually set the commands like you know the HTTP record that method is a get method we are interested. And then the ports you know there's a source port destination port and we can actually try to collect this you know the the the logs of same like wire shock equivalent to T-Shark log and then see how these you know patch packets are in exchange between each other. And then this is a command to run it and then packet flow test with the T-Shark and I have to talk about the hey is the load generator written in Go language. It's a very very useful and also nimble utility to generate some load test. It's available in GitHub and then I'm commanding I mean using a command here in the top saying that you know hey generate a load for this particular IP address which I'm trying to resolve or maybe trying to debug and then someone in the working scenario you see that when I run the wire shock you see the proper response in the green color section right. You see the twenty eighty one is reachable on five thousand port and then it's it's working as expected but in case of a non-working scenario and there's no traffic it is actually it's again time to issue and then we can now get into a TCP dump. The for the same scenario how TCP dump would help us to dump out the traffic between you know to client and server right. So here in the bottom you see that the pseudo of TCP dump and then you specify the command asset here so we'll get the dump of the TCP and then analyze that right. And we could also use the CRI CTL to know to check whether the listening ports are in a proper and mapping in a map to the required port which is you know which has been looked from outside. And there is another utility I have used here it's a pseudo in a center and that would also tell us that you know whether the connections can be established from outside to the inside cluster. And by the way these are all these commands are running from the individual machines are in the particular one of the agent nodes. And finally we reached to the application code right so we started somewhere outside and we layer after layer we reached to the particular application in the application code we see that the host has been set as a zero dot zero dot zero dot zero right. That's been kind of another listening port inside that application code and by the way this is how the flask I mean this is a template of flash code I we copied I mean I copied from the Internet and just put it for a sake you know for troubleshooting. And this is how the application code looks like but when we get into the Docker file when we create for the for the flask you know we only create a Docker file this is how it looks like so for the working file. And we are injecting with the parameters in a dash dash host of zero zero zero but in the non working scenario. It's been hard coded with 127 maybe the non working scenario might work with them with the developer in the development environment but when we move to the Kubernetes world we are forcing to work on 127. That is the issue we are you know we were able to find out from this whole discovery. So underlying I mean in a nutshell and the dash dash host which has been set as 127 which was actually not worked as expected but when we change back to zero dot zero which means that flexibility in you know assigning some IP address from the host. We are not actually forcing any IP address right we are just saying free IP address I mean free to pick from the host right. So this actually I mean the left side worked properly and then right side was actually failing that is the issue here which we troubleshoot all the way. And we could also look at you know whether it's listening in the right IP address I mean this is just a command I put it for a useful sake right that I see I mean CRI CTL inspect that's a command and template I think you will find it in the the slider which you get it later. The key takeaway from out of this right so the way we have to approach this. There are two ways you can actually start troubleshooting it's we always start from top to bottom approach top is nothing but I will look at in the portal respective cloud provider portal and then we try to debug one after the other. But once if you are familiar with this concept you can actually do the reverse also but see we have been you know maybe we found that the troubleshooting from bottom to top layer is the most important. The most convenient or maybe most useful to isolate some of this issue issue because the the easiest thing is like top to bottom right you look at from the Azure portal or maybe the specific cloud provider portal load balancer NSG. That's anybody can do but some of the hard issues right you have to start from the bottom to top stack so that you know you'll be able to do. And at the end of the day the communities are the overlay networking concepts you discussed it's all it's all built on IP tables and the routing rules and that's what you know don't get you know scared off maybe are afraid to talk about this or learn more about it and. And in the process of elimination start from the start to high level and start with the route tables and the energies and the general networking routing issues or if you wanted to follow up the high level approach. Start from the criminal or maybe as I said into the node and then you know. Go to the top top you as some of the tools I found or maybe it's been used internally to we see you know it's used for troubleshooting and I found that useful to share it here and keep CTL I know you must be knowing about some of this keep CTL troubleshooting commands and as such into the respective cloud provider provider agent notes and by the way it's not recommended to it's not advice to. Always as such into the notes unless there is a cover shooting requirement right and we strongly discourage you know changing anything in the notes and because it's like at the end of the day these are like a temporal notes right so the note goes down we'll get another note so don't try to customize anything in the notes and make use of the respective providers load balancer insights. That will get some kind of you know clues to mail down you know what what's going on inside. Then these are like a five commands you know CRI CTL and NS Center and then to troubleshoot effectively let's see tcb dump or tcb dump was in Windows where I'm in Windows world which has been now ported to Linux also and there are other tools of the system internal tools if you look at system internal tools by Marcus no week was. There are like 30 plus our tools mostly built for windows has been now ported back to window and it's our Linux world as an so so you will find the some of the good benefits from. System internal tools going to Linux world and one of them is the tcb dump and T shark is nothing but the wire shark equivalent in the command line and LS office the file open you know to check whether the files are opened or not and that's all about it and I know it's a. It's a kind of a quick run through and but I'm sure that you will enjoy when you start debugging line by line or maybe the commands as in the and described in the slides and. If you have any questions to reach out to me and the slides I will share we post this conference I will share it in the bottom of the handle right so you can take a look at it if you have any questions let me know and yeah.