 Hello everybody, welcome to this presentation about Kubernetes Advanced Networking Testing with Kyn. My name is Antonio Hea and I'm a contributor in Kubernetes and Kyn projects. For those of you that are not familiar with Kyn, Kyn is a tool that is able to deploy Kubernetes in Docker containers. It was created by Benjamin Elders some time ago to improve the testing of the Kubernetes project. One of the most important features of Kyn is that it allows you to create multi-node scenarios very useful to test conformance and it's very optimized for performance and stability. It's able to boot a craster in less than 30 seconds. The way that Kyn works is using special node images. These node images are container images that come preloaded with systemd, kubelet and containerd binaries and the rest of the Kubernetes components container images. The way that Kyn works is using these images to create a cluster and once the images are created and running, it turns kubatmin on top to configure the cluster. In this slide you can see what is the most common deployment of Kyn for testing with one control plane node and two worker nodes. For this presentation we are going to focus in the networking. What we are interested in is how Kyn implements the networking. Kyn by default uses Docker and the Docker networking creates a Linux bridge. Then all the container images are attached to this Linux bridge using BS interfaces. On top of that we have a lot of IP table rules in different layers. For example, Docker uses IP tables to implement forwarding from the host to the containers and Kyn or the Kubernetes cluster uses IP tables by Qproxy to create the services abstraction for networking and Kynet uses it to create to work the pod-to-pod communication. Another important feature from Kyn is that it has an API and we can use this API to create plugins so we don't need to wait for the project to implement the feature that we want. We just can use this API and this is what we are going to demonstrate in this presentation how to use Kyn API to create complex network scenarios. One of the most common requests in Kyn that we didn't implement officially is to be able to simulate nodes with multiple interfaces and multiple networks. This usually this is common in bare metal scenarios when you want to provide different networks for different functionalities in this case. In the example we have an storage network and an external network. So let's explain how we can create a plugin to extend Kyn without having to have the feature in the Kyn project itself. So what we are going to do is to deploy a Kyn cluster. The Kyn cluster as we saw before is going to create several nodes attached to a Linux bridge. If we want to create new networks we can use Docker directly to create a new network. The common Docker network create is going to create a new Linux bridge. As we can see in the image this new Linux bridge is going to be isolated. The next step is to connect the containers the node cluster containers to this Linux bridge. So again Docker has another command that allows us to do that and automatically creates the interfaces inside the nodes. This is very useful for example to test scenarios and to use the Docker capabilities because we can with one just simple command we can run an external container simulating an NFS server that bots can use to test the multi-network functionality. So we are going to demonstrate how to do this. If you see my screen you can see that I have a repository with different plugins. In this case I created a plugin that is called Vermetta. This plugin uses the config file to define the topology. If you can see you can you can leverage the kind API to keep having all the kind functionalities and extend it to have a new field that allows you to define the external networks. So we can just tell the command to create a new cluster with this configuration where we tell that we want to add two new external networks to the cluster config. Once we have the cluster deployed we can see that we can still use the kind commands with our new cluster and we can see that we are going to have the new networks that we define. So here we have external and storage and we can see that these new networks are attached to the commands that to the containers can control brain and can work it. So let's now try to simulate an external NFS server that is going to share with a container one of the host folders. Let's create one folder that we want to share. It already exists okay let me clean it so you can see that we start from a clean environment and now we are going to create a container with an NFS server in the storage network. So this is the command that says docker run a container with the name NFS in the network storage and these are the options to mount the new recently created folder into the container so we can export it okay now we can see that we have a new container called NFS here you go and it is running so the next thing that we are going to do is we are going to create a pod that is able to mount this volume and we are going to see how the cluster is able to use both networks one of the good things of kind is that it uses the docker custom network that allows to resolve the container name so for the server we don't have to use a piece we can use the name of the container here directly so this spot is going to mount this new folder and it's going to create a file called dates with the time in a loop so let's apply the manifest and once it is running it is being created when it is running we should be able to see the file in the host tmp NFS share dates okay it took some time to create it but you can see how the file is being updated by the pod every five seconds one of the more complex scenarios to test are the multicluster by multicluster i mean different kubernetes clusters in different regions or in different data centers the problematic here is that the network between this cluster is not always reliable and sometimes it goes through internet and you have latency and you have packet drops and all this thing so for testing these scenarios we need to emulate this connection between cluster this intra cluster connectivity that is not always easy to do so how can we do this in kind well first what we are going to do is to create one cluster in its own network so its cluster is going to be isolated in its own network what we are going to do this because we want to emulate the inter cluster communication and for that we are going to connect all these clusters to a container to a special container that contains the that is able to emulate the one but we have a problem with docker that docker doesn't allow to change the gateway of the docker network so what we are going to do is a small we can work around this with netlink so at the end these docker containers are network name spaces with vf interfaces so what we are going to do is to use netlink to replace the gateways this way we achieve that each cluster send the traffic to this one emulator container okay so let me show you how to do this so in this case we are going we have a new plugin called multi cluster and i created a configuration file this is up to you you can define you can create the configurations that you want for this what i use it is a template that allows you to to indicate the cluster name and the number of nodes that you want per cluster and the network characteristics because in multi cluster we need different standards per cluster they cannot overlap if they overlap we cannot route and the traffic from cluster to cluster is not going to work once we have the configuration file we just need to run this plugin needs sudo privilege because the maintenance replaced gateway work around so we tell sudo multi cluster create config and it starts to create the clusters once it once it finished we can see that we are going to have two clusters and each cluster is going to have its own network okay so we can create an hyper server in one of the clusters so we are going to test the connectivity between two applications in different clusters so in this case we are going to use this image that has an hyper server okay we created one hyper server in us and we're going to create another hyper server in europe so let's see so we can see that the pod is running okay okay once we have the pod running we are going to expose the pod the pod in us so the pod from europe can reach it we need to specify the part the port hyper listening the port 5001 by the port so we expose the port and we are going to remember the IP address because right now what we are going to do let me try to split the terminal we are going to connect to the pod in europe okay and we are going to test the tcp connectivity so we can report every second and connect to the cluster that is 10 and this is 128 254 and we can see okay that we are in an ideal situation we have 30 gigabytes per second between europe and us okay we can check the ip of one of the nodes in us i have a query that is able to give us the ip of the node so we can check the latency thinking one of these ip is and we can see that we have close to 100 0.100 milliseconds between the cluster but all of us know that that's not real so let's and this is the thing we can we can see that we have this special container that we talked about before that is the one container and this is the one that we are going to use to emulate the one so the clusters behave as if they were deploying in in a real scenario so we are going to get into this container and we are going to emulate a latency of 100 milliseconds to emulate the latency what we are going to use is the traffic control from the linux this is and we know that each cluster is connected to the eth1 and eth2 interfaces if you have more clusters you just need to map them the commands to the corresponding interfaces and we are going to tell to add a 100 milliseconds today okay we can see that the pin starts to go 100 milliseconds this is in one direction let's do it for the other direction so we see that we have 200 milliseconds and what do i want to demonstrate with this okay so we have more latency okay we can leave with that but let's see what happens to the ipad okay you see now we have 43 megabits per second that is why it's important to emulate real scenarios if you are testing a multicluster and you are not emulating the latency you are in a very happy scenario because when somebody deploys your application whatever in a multicluster scenario and with long distances like europe to acl to us they are going to have latency and the latency impact directly on the bandwidth of the tcp connections for our last example we are going to simulate a multicluster scenario and demonstrate a new feature that is going to be released in this new kubernetes 1.21 version that is called topology aware hints so basically users start to deploy more multicluster scenarios to improve the resilience of their clusters but current implementation on service doesn't provide a good way to indicate the traffic to remain on zone so it usually is spread all over the cluster however the users may want to give some affinity to improve performance and reduce costs for that this new feature give a new field in the pointer laces to provide this information so qproxy or other service implementation can use it to to implement the service affinity to the zone we can demonstrate this new and test this new feature with kai an example of how to do this is i created a plugin that deploy a multi-zone cluster so you just need to give the parameters with the topology that you want it means the zones the number of zones we are going to set two zones and the number of node person let's add one node person in this case the control plane node is not considered as part of the zone once we have our cluster working we are going to create a set a deployment with your replicas one pod per node so for that i have a template here that you can use so we are going to deploy three replicas and this this program is a server that replies with the name of the pod so we are going to create a deployment and we are going to expose the deployment to create a service so we have a new service and this new service has to have end point slices per each pod the way that this feature work is adding a new field to the end point slices with the topology information so i have a query here that will allow us to extract this information so this basically is iterating over the point slice and for each endpoint it prints the name of the pod of the point and the new field call hints right now it's missing the name of the endpoint slice server deployment so we can see that these are the name of the pods but there is no value in the hints field this is because the service has to be annotated with an special annotation to enable the topology hints so we are going to annotate the service this is the annotation service dot Kubernetes dot io topology our our hints equal auto so once we annotate the service we should be able to see the topology hints okay so we have this endpoint that i prefer to have traffic from zone zero and the other this other one that is prefer to have traffic from zone one so how can we test it so the most simple way to test this is to send traffic to the pods from one specific zone in this case i have this query query that is going to give us the name of the nodes and the zone it belongs so that's using the well known label for the nodes topology dot Kubernetes dot io so okay and we can see that the kind worker two is in zone one so if we query the if we query the service from this node the feature should give us affinity to the pod in the zone one only so for doing that we can exit directly from the in this case worker two it's missing an R kind worker two and we could this special state p of the service that has a handler hostname that returns the hostname of the pod we do that i did something wrong okay exit if we do that we can see that everything goes to the same endpoint that if we check again is the endpoint prefer for zone one if we do the same but for the other node the kind worker that is in zone zero we should see the other two endpoints okay so ns6 and n bkhm that we see here that are the ones preferred for zone zero well i think that with these three examples we were able to demonstrate how to use kind to emulate and test complexity into these scenarios my recommendation is to try to keep it always these scenarios as close as possible to the reality as we saw in the multi multi cluster example adding latency to the to the test make a big difference another important point is the kind is not a replacement for system testing there are things that you are not going to be able to emulate in containers with this if you are interested in helping or contributing or you have some idea you want to create your own plugins or collaborate with me or with the kind community in these plugins you have the links to the slack channels and if you're interested in the in this presentation code you have the link to my own repo so you can go through the examples on your own thank you very much for attending this presentation and goodbye