Kubernetes Advanced Networking Testing with KIND - Antonio Ojea, RedHat¶

Playlists: KubeCon + CloudNativeCon Europe 2021

Antonio Ojea shows how KIND implements networking with Docker and how to use it for advanced Kubernetes network testing.

Categories: Networking

Topics: kind, networking testing, linux bridge, iptables, kubernetes testing

Transcript¶

▶ 0:00 hello everybody welcome to this presentation about kubernetes advanced networking testing with kine my name is antonio here and i'm a contributor in kubernetes and kind projects for those of you that are not familiar with kind kind is a tool that is able to deploy kubernetes in docker containers it was created by benjamin elder some time ago to improve the testing of the kubernetes project one of the most important features of kind is that it allows you to create multi-node scenarios very useful to test conformance and

▶ 0:43 it's very optimized for performance and stability it's able to boost to with the crust and in less than 30 seconds the way that kind works is using a special node images these node images are container images that come preloaded with systemd cumulating container d binaries and the rest of the kubernetes components container images the way that kind works is using these images to create a cluster and once the images are created and running it runs cubadmin on top

▶ 1:25 to configure the cluster so in the slide you can see what is the most common deployment of kind for testing with one control plane node and two worker nodes for this presentation we are going to focus in the networking so what we are interested in is how kind implements the networking kind in by default uses docker and the docker networking creates a linux bridge then then all the container images are attached to this linux bridge using

▶ 2:08 bs interfaces on top of that we have a lot of iptable rules in different layers so for example docker uses ip tables to implement forwarding from the horse to the containers and kind or the kubernetes cluster uses iptables by q proxy to create the services abstraction for networking and kinetic ec to create to warp the pod to pod communication another important feature from kind is that it has an api and we can use this api to create plugins so we don't need to wait

▶ 2:53 for the project to implement the feature that we want we just can use this api and this is what we are going to demonstrate in this presentation how to use kine api to create complex network scenarios one of the most common requests in kind that we didn't implement it officially is to be able to simulate nodes with multiple interfaces and multiple networks this usually this is common in bare metal scenarios when you want to provide different networks for different functionalities

▶ 3:37 in this case in the example we have an installer a storage network and an external network so let's explain how we can create a plugin to stand kind without having to have the feature in the kind project itself so what we are going to do is to deploy a kind cluster the kind cluster as we saw before is going to create several nodes attaches to attach to a to a linux bridge if we want to create new networks we can we can use docker directory to create a new network the command docker network create is

▶ 4:21 going to create a new linux bridge as we can see in the image this new linux bridge is going to be isolated the next step is to connect the containers the node cluster containers to this linux bridge so again docker has another command that allows us to do that and automatically creates the interfaces inside the nodes this is very useful for example to test scenarios and to use the docker capabilities because we can with one just simple command we can run an external container

▶ 5:06 simulating an nfs server that bots can can use to test the multi the multi-network functionality so we are going to demonstrate how to do this did you see my screen you can see that i have a repository with different plugins in this case i created a plugin that is called bermeta this plugin uses the config

▶ 5:48 file to define the topology as you can see you can you can leverage the kind api to keep having all the kind functionalities understand it to have a new field that allows you to define external networks so we can just tell the command to create a new cluster with this configuration where we tell that we want to add two new external networks to the class config

▶ 6:35 once we have the cluster deploy we can see that we can still use the kind commands with our new cluster and we can see that we are going to have the new networks that we define so here we have external and storage and we can see that these new networks are attached to the commands that to the containers can

▶ 7:18 control plane and quarkit so let's now try to simulate uh an external nfs server that is going to share with a container one of the hosts folders let's create one folder that we want to share it already sees okay let me clean it so you can see that we start from a clean environment

▶ 8:01 and now we are going to create a container with an nfs server in the storage network so this is the command that says docker run a container with the name nfs in the network storage and these are the the options to mount the new recently created folder into the container so we can export it okay now we can see that we have a new container called nfs here we go and it is running so the next thing that we are going to do is we are going to create a pot

▶ 8:46 that is able to mount this volume and we are going to see how the cluster is able to use both network one of the good things of kind is that it uses the docker custom networks that allow to resolve the container name so for the server we don't have to use a piece we just can use the name of the container here directory so this spot is going to mount this new folder and it's going to create a file called dates with the time in a loop so let's apply the manifest and once it is running

▶ 9:36 it has been created when it is running we should be able to see the file in the host tmp and fshare dates okay it took some time to to create it but you can see how the file is being

▶ 10:20 updated by the pod every five seconds one of the more complex scenarios to test are the multi-cluster by multi-cluster i mean different kubernetes clusters in different regions or in different data centers the problematic here is that the network between this cluster is not always readable and sometimes it goes through internet and you have latency and you have packet drawers

▶ 11:03 and all this thing so for testing these scenarios we need to emulate this connection between cluster this intra cluster connectivity that is not always easy to do so how can we do this in kind well first what we are going to do is to create one cluster in each in its own network so its cluster is going to be as isolated in its own network what we are going to do this because we want to emulate the inter cluster com communication and for that we are going to connect

▶ 11:46 all these clusters to a container to a special container that contains the that is able to emulate the the one but we have a problem with docker that docker doesn't allow to change the gateway of the docker network so what we are going to do is a small we can work around this with net link so at the end these docker containers are network name displays name spaces with bs interfaces so what we are going to do

▶ 12:31 is to use netlink to replace the gateways this way we achieve that each cluster send the traffic to this one emulator container okay so let me show you how to do this so in this case we are going we have a new plugin called multicaster and i created a configuration file this is up to you you can define you can create the configurations that you want

▶ 13:12 for this what i used is a template that allows you to to indicate the cluster name and the number of nodes that you want per cluster and the network characteristics because in multicluster we need different subnets per cluster they can overlap if they overlap we cannot route and the traffic from cluster to cluster is not going to work once we have the configuration file we just need to run this plugin needs sudo

▶ 13:54 privilege because the maintenance replaced gateway work around so we tell sudo multicaster create config and it starts to create the clusters okay once it once it finished we can see that

▶ 14:48 we are going to have two clusters and each cluster is going to have its own network okay so we can create an iperf server in one of the clusters so we are going to test the connectivity between two applications in different clusters so in this case we are going to use this image that has an ipad server okay

▶ 15:35 we created one hyperserver in us and we are going to create another hyper server in europe so let's see oh sorry so we can see that the body running okay okay once we have the port running we are going to expose the pod the pod in us so the ball from

▶ 16:17 europe can reach it because for iperf we need to specify the part the port repair listening the port 5001 by the four so we expose the port and we are going to remember the ip address because right now what we are going to do let me try to split the terminal we are going to connect to the pole in europe

▶ 17:13 okay and we are going to test the tcp connectivity so we can report every second and connect to the cluster that is 10 and this is one two eight two five four and we can see okay that we are in a an ideal situation we have 30 gigabits per second between europe and us okay we can check the ip of one of the nodes in us

▶ 18:05 i have a query that is able to give us the eyepiece of the nose so we can check the latency pinging one of these ips and we can see that we have close to 100 0.100 milliseconds between the cluster but all of us know that that's not real so less and this is the thing we can we can see that we have this special container that we talked before

▶ 18:49 that is the one container and this is the one that we are going to use the more the to emulate the one the clusters behave as if they were deploying in in a real scenario so we are going to get into this container and we are going to emulate a latency of 100 milliseconds to emulate the latency what we are going to use is the traffic control from the linux this is this and we know that each cluster is connected to the eth1

▶ 19:33 and eth2 interfaces if you have more clusters you just need to map then the commands to the corresponding interfaces and we are going to tell to add a 100 milliseconds today okay we can see that the pin starts to go 100 milliseconds this is in one directions let's do it for the other direction so we see that we have 200 milliseconds and what do i want to demonstrate with this okay so we have more latency okay we can live with that but let's see what happens to the ipad okay you see now we have

▶ 20:19 43 megabits per second that is why it's important to emulate real scenarios if you are testing a multicluster and you are not emulating the latency you are in a very happy scenario because when somebody deploys your application whatever in a multicultural scenario and with long distances like europe to acl to us they are going to have latency and the latency impact directly on the bandwidth of the tcp connections for our last example we are going to simulate

▶ 21:02 a multi-sony scenario and demonstrate a new feature that is going to be released in this new kubernetes 1.21 version that is called topology aware things so basically users start to deploy more multi-sound scenarios to improve the resilience of their clusters but current implementation of service doesn't provide a good way to indicate the traffic to remain on zone so it usually is a spread all over the cluster however the users may want to give some affinity to improve performance and reduce costs for that

▶ 21:46 this new feature give a new field in the endpoint slices to provide this information so q proxy or other services implementation can use it to to implement the service affinity to the song we can demonstrate this new and test this new feature with kai an example on how to do this is i've created a plugin that deploy a multi-song cluster so you just need to give the parameters with the topology that you want

▶ 22:30 it means the the songs the number of songs we are going to say two zones and the number of notes person let's add one node person in this case the control plane node is not considered as part of the zone once we have our cluster working we are going to create a deployment with your replicas one part per node so for that i have a template here that you can use so we are going to deploy

▶ 23:14 three replicas and this this program is a server that replies with the name of the bot so we are going to create a deployment and we are going to expose the deployment to create a service so we have a new service and this new service has to have endpoint slices per each spot the way

▶ 24:00 that this fit will work is adding a new field to the endpoint slices with the topology information so i have a query here that will allow us to extract this information so this basically is iterating over the point slice and for each endpoint it prints the name of the pot of the point and the new field called hints right now it's missing the name of the pointed lice server deployment so we can see that these are the name of

▶ 24:45 the pots but there is no value in the hints field this is because the service has to be annotated with a special annotation to enable the topology hints so we are going to annotate the service this is the notation service.kubernetes topology aware hints equal auto so once we annotate the service we should be able to see the topology hints okay so we have this endpoint that are preferred to have traffic from zones here zero and the other this other one that is prefer to have traffic from

▶ 25:30 some one so how can we test it so the most simple way to test this is to send traffic to the pods from one specific zone in this case i have this query that is going to give us the name of the nodes and the zone it belongs so that's using the well-known label for the nodes topology.kubernetes.oh so okay and we can see that the kind worker 2 is in song one so if we query the if we query

▶ 26:16 the service from this node the feature should give us affinity to the pod in the song one only so for doing that we can exec directory from the in this case worker 2 it's missing our kind worker 2 and we call this a special this is the pld service

▶ 26:59 that has a handler first name that returns the hostname of the post we do that i did something wrong okay exit if we do that we can see that everything goes to the same endpoint that if we check again is the point prefer for someone if we do the same but for the other node the kind worker that is in zone zero we should see the other two endpoints okay so ns6 and

▶ 27:46 and b chi hm that we see here that are the ones prefer for zone zero well i think that with these three examples we were able to demonstrate how to use kind to emulate and test complex it scenarios my recommendation is to try to keep it always these scenarios as close as possible to the reality as we saw in the multi multi-cluster example adding latency to the to the test make a big difference another important point is the

▶ 28:31 kind is not a replacement for system testing there are things that you are not going to be able to emulate in containers with this if you are interested in helping or contributing or you have some idea you want to create your own plugins or collaborate with me or with the kind community in this plugins you have the links to the luck channels and if you're interested in the in this presentation code you have the link to my own repo so you can go through the examples on your own thank you very much for attending this

▶ 29:14 presentation and goodbye