{"id":6193,"date":"2019-03-29T23:00:02","date_gmt":"2019-03-29T23:00:02","guid":{"rendered":"http:\/\/howk.de\/w1\/blog-kube-proxy-subtleties-debugging-an-intermittent-connection-reset\/"},"modified":"2019-03-29T23:00:02","modified_gmt":"2019-03-29T23:00:02","slug":"blog-kube-proxy-subtleties-debugging-an-intermittent-connection-reset","status":"publish","type":"post","link":"https:\/\/howk.de\/?p=6193","title":{"rendered":"Blog: kube-proxy Subtleties: Debugging an Intermittent Connection Reset"},"content":{"rendered":"

Author:<\/strong> Yongkun Gui<\/a>, Google<\/p>\n

I recently came across a bug that causes intermittent connection resets. After
\nsome digging, I found it was caused by a subtle combination of several different
\nnetwork subsystems. It helped me understand Kubernetes networking better, and I
\nthink it\u2019s worthwhile to share with a wider audience who are interested in the same
\ntopic.<\/p>\n

The symptom<\/h2>\n

We received a user report claiming they were getting connection resets while using a
\nKubernetes service of type ClusterIP to serve large files to pods running in the
\nsame cluster. Initial debugging of the cluster did not yield anything
\ninteresting: network connectivity was fine and downloading the files did not hit
\nany issues. However, when we ran the workload in parallel across many clients,
\nwe were able to reproduce the problem. Adding to the mystery was the fact that
\nthe problem could not be reproduced when the workload was run using VMs without
\nKubernetes. The problem, which could be easily reproduced by a simple
\napp<\/a>, clearly has something to
\ndo with Kubernetes networking, but what?<\/p>\n

Kubernetes networking basics<\/h2>\n

Before digging into this problem, let\u2019s talk a little bit about some basics of
\nKubernetes networking, as Kubernetes handles network traffic from a pod
\nvery differently depending on different destinations.<\/p>\n

Pod-to-Pod<\/h3>\n

In Kubernetes, every pod has its own IP address. The benefit is that the
\napplications running inside pods could use their canonical port, instead of
\nremapping to a different random port. Pods have L3 connectivity between each
\nother. They can ping each other, and send TCP or UDP packets to each other.
\nCNI<\/a> is the standard that solves
\nthis problem for containers running on different hosts. There are tons of
\ndifferent plugins that support CNI.<\/p>\n

Pod-to-external<\/h3>\n

For the traffic that goes from pod to external addresses, Kubernetes simply uses
\nSNAT<\/a>. What it does
\nis replace the pod\u2019s internal source IP:port with the host\u2019s IP:port. When
\nthe return packet comes back to the host, it rewrites the pod\u2019s IP:port as the
\ndestination and sends it back to the original pod. The whole process is transparent
\nto the original pod, who doesn\u2019t know the address translation at all.<\/p>\n

Pod-to-Service<\/h3>\n

Pods are mortal. Most likely, people want reliable service. Otherwise, it\u2019s
\npretty much useless. So Kubernetes has this concept called “service” which is
\nsimply a L4 load balancer in front of pods. There are several different types of
\nservices. The most basic type is called ClusterIP. For this type of service, it
\nhas a unique VIP address that is only routable inside the cluster.<\/p>\n

The component in Kubernetes that implements this feature is called kube-proxy.
\nIt sits on every node, and programs complicated iptables rules to do all kinds
\nof filtering and NAT between pods and services. If you go to a Kubernetes node
\nand type iptables-save<\/code>, you\u2019ll see the rules that are inserted by Kubernetes
\nor other programs. The most important chains are KUBE-SERVICES<\/code>, KUBE-SVC-*<\/code>
\nand KUBE-SEP-*<\/code>.<\/p>\n