{"id":4299,"date":"2018-12-31T14:12:38","date_gmt":"2018-12-31T14:12:38","guid":{"rendered":"http:\/\/howk.de\/w1\/blog-grpc-load-balancing-on-kubernetes-without-tears\/"},"modified":"2018-12-31T14:12:38","modified_gmt":"2018-12-31T14:12:38","slug":"blog-grpc-load-balancing-on-kubernetes-without-tears","status":"publish","type":"post","link":"https:\/\/howk.de\/?p=4299","title":{"rendered":"Blog: gRPC Load Balancing on Kubernetes without Tears"},"content":{"rendered":"

Author<\/strong>: William Morgan (Buoyant)<\/p>\n

Many new gRPC users are surprised to find that Kubernetes’s default load
\nbalancing often doesn’t work out of the box with gRPC. For example, here’s what
\nhappens when you take a simple gRPC Node.js microservices
\napp<\/a> and deploy it on Kubernetes:<\/p>\n

\"\"<\/p>\n

While the voting<\/code> service displayed here has several pods, it’s clear from
\nKubernetes’s CPU graphs that only one of the pods is actually doing any
\nwork—because only one of the pods is receiving any traffic. Why?<\/p>\n

In this blog post, we describe why this happens, and how you can easily fix it
\nby adding gRPC load balancing to any Kubernetes app with
\nLinkerd<\/a>, a CNCF<\/a> service mesh and service sidecar.<\/p>\n

Why does gRPC need special load balancing?<\/h1>\n

First, let’s understand why we need to do something special for gRPC.<\/p>\n

gRPC is an increasingly common choice for application developers. Compared to
\nalternative protocols such as JSON-over-HTTP, gRPC can provide some significant
\nbenefits, including dramatically lower (de)serialization costs, automatic type
\nchecking, formalized APIs, and less TCP management overhead.<\/p>\n

However, gRPC also breaks the standard connection-level load balancing,
\nincluding what’s provided by Kubernetes. This is because gRPC is built on
\nHTTP\/2, and HTTP\/2 is designed to have a single long-lived TCP connection,
\nacross which all requests are multiplexed<\/em>—meaning multiple requests can be
\nactive on the same connection at any point in time. Normally, this is great, as
\nit reduces the overhead of connection management. However, it also means that
\n(as you might imagine) connection-level balancing isn’t very useful. Once the
\nconnection is established, there’s no more balancing to be done. All requests
\nwill get pinned to a single destination pod, as shown below:<\/p>\n

\"\"<\/p>\n

Why doesn’t this affect HTTP\/1.1?<\/h1>\n

The reason why this problem doesn’t occur in HTTP\/1.1, which also has the
\nconcept of long-lived connections, is because HTTP\/1.1 has several features
\nthat naturally result in cycling of TCP connections. Because of this,
\nconnection-level balancing is “good enough”, and for most HTTP\/1.1 apps we
\ndon’t need to do anything more.<\/p>\n

To understand why, let’s take a deeper look at HTTP\/1.1. In contrast to HTTP\/2,
\nHTTP\/1.1 cannot multiplex requests. Only one HTTP request can be active at a
\ntime per TCP connection. The client makes a request, e.g. GET \/foo<\/code>, and then
\nwaits until the server responds. While that request-response cycle is
\nhappening, no other requests can be issued on that connection.<\/p>\n

Usually, we want lots of requests happening in parallel. Therefore, to have
\nconcurrent HTTP\/1.1 requests, we need to make multiple HTTP\/1.1 connections,
\nand issue our requests across all of them. Additionally, long-lived HTTP\/1.1
\nconnections typically expire after some time, and are torn down by the client
\n(or server). These two factors combined mean that HTTP\/1.1 requests typically
\ncycle across multiple TCP connections, and so connection-level balancing works.<\/p>\n

So how do we load balance gRPC?<\/h1>\n

Now back to gRPC. Since we can’t balance at the connection level, in order to
\ndo gRPC load balancing, we need to shift from connection balancing to request<\/em>
\nbalancing. In other words, we need to open an HTTP\/2 connection to each
\ndestination, and balance requests<\/em> across these connections, as shown below:<\/p>\n

\"\"<\/p>\n

In network terms, this means we need to make decisions at L5\/L7 rather than
\nL3\/L4, i.e. we need to understand the protocol sent over the TCP connections.<\/p>\n

How do we accomplish this? There are a couple options. First, our application
\ncode could manually maintain its own load balancing pool of destinations, and
\nwe could configure our gRPC client to use this load balancing
\npool<\/a>. This approach gives
\nus the most control, but it can be very complex in environments like Kubernetes
\nwhere the pool changes over time as Kubernetes reschedules pods. Our
\napplication would have to watch the Kubernetes API and keep itself up to date
\nwith the pods.<\/p>\n

Alternatively, in Kubernetes, we could deploy our app as headless
\nservices<\/a>.
\nIn this case, Kubernetes will create multiple A
\nrecords<\/a>
\nin the DNS entry for the service. If our gRPC client is sufficiently advanced,
\nit can automatically maintain the load balancing pool from those DNS entries.
\nBut this approach restricts us to certain gRPC clients, and it’s rarely
\npossible to only use headless services.<\/p>\n

Finally, we can take a third approach: use a lightweight proxy.<\/p>\n

gRPC load balancing on Kubernetes with Linkerd<\/h1>\n

Linkerd<\/a> is a CNCF<\/a>-hosted service
\nmesh<\/em> for Kubernetes. Most relevant to our purposes, Linkerd also functions as
\na service sidecar<\/em>, where it can be applied to a single service—even without
\ncluster-wide permissions. What this means is that when we add Linkerd to our
\nservice, it adds a tiny, ultra-fast proxy to each pod, and these proxies watch
\nthe Kubernetes API and do gRPC load balancing automatically. Our deployment
\nthen looks like this:<\/p>\n

\"\"<\/p>\n

Using Linkerd has a couple advantages. First, it works with services written in
\nany language, with any gRPC client, and any deployment model (headless or not).
\nBecause Linkerd’s proxies are completely transparent, they auto-detect HTTP\/2
\nand HTTP\/1.x and do L7 load balancing, and they pass through all other traffic
\nas pure TCP. This means that everything will just work.<\/em><\/p>\n

Second, Linkerd’s load balancing is very sophisticated. Not only does Linkerd
\nmaintain a watch on the Kubernetes API and automatically update the load
\nbalancing pool as pods get rescheduled, Linkerd uses an exponentially-weighted
\nmoving average<\/em> of response latencies to automatically send requests to the
\nfastest pods. If one pod is slowing down, even momentarily, Linkerd will shift
\ntraffic away from it. This can reduce end-to-end tail latencies.<\/p>\n

Finally, Linkerd’s Rust-based proxies are incredibly fast and small. They
\nintroduce <1ms of p99 latency and require <10mb of RSS per pod, meaning that
\nthe impact on system performance will be negligible.<\/p>\n

gRPC Load Balancing in 60 seconds<\/h1>\n

Linkerd is very easy to try. Just follow the steps in the Linkerd Getting
\nStarted Instructions<\/a>—install the
\nCLI on your laptop, install the control plane on your cluster, and “mesh” your
\nservice (inject the proxies into each pod). You’ll have Linkerd running on your
\nservice in no time, and should see proper gRPC balancing immediately.<\/p>\n

Let’s take a look at our sample voting<\/code> service again, this time after
\ninstalling Linkerd:<\/p>\n

\"\"<\/p>\n

As we can see, the CPU graphs for all pods are active, indicating that all pods
\nare now taking traffic—without having to change a line of code. Voila,
\ngRPC load balancing as if by magic!<\/p>\n

Linkerd also gives us built-in traffic-level dashboards, so we don’t even need
\nto guess what’s happening from CPU charts any more. Here’s a Linkerd graph
\nthat’s showing the success rate, request volume, and latency percentiles of
\neach pod:<\/p>\n

\"\"<\/p>\n

We can see that each pod is getting around 5 RPS. We can also see that, while
\nwe’ve solved our load balancing problem, we still have some work to do on our
\nsuccess rate for this service. (The demo app is built with an intentional
\nfailure—as an exercise to the reader, see if you can figure it out by
\nusing the Linkerd dashboard!)<\/p>\n

Wrapping it up<\/h1>\n

If you’re interested in a dead simple way to add gRPC load balancing to your
\nKubernetes services, regardless of what language it’s written in, what gRPC
\nclient you’re using, or how it’s deployed, you can use Linkerd to add gRPC load
\nbalancing in a few commands.<\/p>\n

There’s a lot more to Linkerd, including security, reliability, and debugging
\nand diagnostics features, but those are topics for future blog posts.<\/p>\n

Want to learn more? We\u2019d love to have you join our rapidly-growing community!
\nLinkerd is a CNCF<\/a> project, hosted on
\nGitHub<\/a>, and has a thriving community
\non Slack<\/a>, Twitter<\/a>,
\nand the mailing lists<\/a>. Come and
\njoin the fun!<\/p>\n","protected":false},"excerpt":{"rendered":"

Author: William Morgan (Buoyant) Many new gRPC users are surprised to find that Kubernetes’s default load balancing often doesn’t work out of the box with gRPC. For example, here’s what happens when you take a simple gRPC Node.js microservices app and deploy it on Kubernetes: While the voting service displayed here has several pods, it’s […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"yoast_head":"\nBlog: gRPC Load Balancing on Kubernetes without Tears - Howk IT-Dienstleistungen<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/howk.de\/?p=4299\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Blog: gRPC Load Balancing on Kubernetes without Tears - Howk IT-Dienstleistungen\" \/>\n<meta property=\"og:description\" content=\"Author: William Morgan (Buoyant) Many new gRPC users are surprised to find that Kubernetes’s default load balancing often doesn’t work out of the box with gRPC. For example, here’s what happens when you take a simple gRPC Node.js microservices app and deploy it on Kubernetes: While the voting service displayed here has several pods, it’s […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/howk.de\/?p=4299\" \/>\n<meta property=\"og:site_name\" content=\"Howk IT-Dienstleistungen\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/howk.de\" \/>\n<meta property=\"article:published_time\" content=\"2018-12-31T14:12:38+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/kubernetes.io\/images\/blog\/grpc-load-balancing-with-linkerd\/Screenshot2018-11-0116-c4d86100-afc1-4a08-a01c-16da391756dd.34.36.png\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/howk.de\/?p=4299#article\",\"isPartOf\":{\"@id\":\"https:\/\/howk.de\/?p=4299\"},\"author\":{\"name\":\"admin\",\"@id\":\"https:\/\/howk.de\/#\/schema\/person\/b029bd02d4f35dce869ef54c81a100c5\"},\"headline\":\"Blog: gRPC Load Balancing on Kubernetes without Tears\",\"datePublished\":\"2018-12-31T14:12:38+00:00\",\"dateModified\":\"2018-12-31T14:12:38+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/howk.de\/?p=4299\"},\"wordCount\":1347,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/howk.de\/#organization\"},\"articleSection\":[\"Hi Tech\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/howk.de\/?p=4299#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/howk.de\/?p=4299\",\"url\":\"https:\/\/howk.de\/?p=4299\",\"name\":\"Blog: gRPC Load Balancing on Kubernetes without Tears - Howk IT-Dienstleistungen\",\"isPartOf\":{\"@id\":\"https:\/\/howk.de\/#website\"},\"datePublished\":\"2018-12-31T14:12:38+00:00\",\"dateModified\":\"2018-12-31T14:12:38+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/howk.de\/?p=4299#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/howk.de\/?p=4299\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/howk.de\/?p=4299#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/howk.de\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Blog: gRPC Load Balancing on Kubernetes without Tears\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/howk.de\/#website\",\"url\":\"https:\/\/howk.de\/\",\"name\":\"Howk IT-Dienstleistungen\",\"description\":\"Howk IT Services - Howk IT-Dienstleistungen\",\"publisher\":{\"@id\":\"https:\/\/howk.de\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/howk.de\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/howk.de\/#organization\",\"name\":\"HowK\",\"url\":\"https:\/\/howk.de\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/howk.de\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/howk.de\/w1\/wp-content\/uploads\/2013\/12\/howk-logo.png\",\"contentUrl\":\"https:\/\/howk.de\/w1\/wp-content\/uploads\/2013\/12\/howk-logo.png\",\"width\":170,\"height\":170,\"caption\":\"HowK\"},\"image\":{\"@id\":\"https:\/\/howk.de\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/howk.de\",\"http:\/\/de.linkedin.com\/in\/howkde\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/howk.de\/#\/schema\/person\/b029bd02d4f35dce869ef54c81a100c5\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/howk.de\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/b5a20f4d07bca1b73f25cff58a1116c4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/b5a20f4d07bca1b73f25cff58a1116c4?s=96&d=mm&r=g\",\"caption\":\"admin\"},\"url\":\"https:\/\/howk.de\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Blog: gRPC Load Balancing on Kubernetes without Tears - Howk IT-Dienstleistungen","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/howk.de\/?p=4299","og_locale":"en_US","og_type":"article","og_title":"Blog: gRPC Load Balancing on Kubernetes without Tears - Howk IT-Dienstleistungen","og_description":"Author: William Morgan (Buoyant) Many new gRPC users are surprised to find that Kubernetes’s default load balancing often doesn’t work out of the box with gRPC. For example, here’s what happens when you take a simple gRPC Node.js microservices app and deploy it on Kubernetes: While the voting service displayed here has several pods, it’s […]","og_url":"https:\/\/howk.de\/?p=4299","og_site_name":"Howk IT-Dienstleistungen","article_publisher":"https:\/\/www.facebook.com\/howk.de","article_published_time":"2018-12-31T14:12:38+00:00","og_image":[{"url":"https:\/\/kubernetes.io\/images\/blog\/grpc-load-balancing-with-linkerd\/Screenshot2018-11-0116-c4d86100-afc1-4a08-a01c-16da391756dd.34.36.png"}],"author":"admin","twitter_card":"summary_large_image","twitter_misc":{"Written by":"admin","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/howk.de\/?p=4299#article","isPartOf":{"@id":"https:\/\/howk.de\/?p=4299"},"author":{"name":"admin","@id":"https:\/\/howk.de\/#\/schema\/person\/b029bd02d4f35dce869ef54c81a100c5"},"headline":"Blog: gRPC Load Balancing on Kubernetes without Tears","datePublished":"2018-12-31T14:12:38+00:00","dateModified":"2018-12-31T14:12:38+00:00","mainEntityOfPage":{"@id":"https:\/\/howk.de\/?p=4299"},"wordCount":1347,"commentCount":0,"publisher":{"@id":"https:\/\/howk.de\/#organization"},"articleSection":["Hi Tech"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/howk.de\/?p=4299#respond"]}]},{"@type":"WebPage","@id":"https:\/\/howk.de\/?p=4299","url":"https:\/\/howk.de\/?p=4299","name":"Blog: gRPC Load Balancing on Kubernetes without Tears - Howk IT-Dienstleistungen","isPartOf":{"@id":"https:\/\/howk.de\/#website"},"datePublished":"2018-12-31T14:12:38+00:00","dateModified":"2018-12-31T14:12:38+00:00","breadcrumb":{"@id":"https:\/\/howk.de\/?p=4299#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/howk.de\/?p=4299"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/howk.de\/?p=4299#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/howk.de\/"},{"@type":"ListItem","position":2,"name":"Blog: gRPC Load Balancing on Kubernetes without Tears"}]},{"@type":"WebSite","@id":"https:\/\/howk.de\/#website","url":"https:\/\/howk.de\/","name":"Howk IT-Dienstleistungen","description":"Howk IT Services - Howk IT-Dienstleistungen","publisher":{"@id":"https:\/\/howk.de\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/howk.de\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/howk.de\/#organization","name":"HowK","url":"https:\/\/howk.de\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/howk.de\/#\/schema\/logo\/image\/","url":"https:\/\/howk.de\/w1\/wp-content\/uploads\/2013\/12\/howk-logo.png","contentUrl":"https:\/\/howk.de\/w1\/wp-content\/uploads\/2013\/12\/howk-logo.png","width":170,"height":170,"caption":"HowK"},"image":{"@id":"https:\/\/howk.de\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/howk.de","http:\/\/de.linkedin.com\/in\/howkde"]},{"@type":"Person","@id":"https:\/\/howk.de\/#\/schema\/person\/b029bd02d4f35dce869ef54c81a100c5","name":"admin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/howk.de\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/b5a20f4d07bca1b73f25cff58a1116c4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/b5a20f4d07bca1b73f25cff58a1116c4?s=96&d=mm&r=g","caption":"admin"},"url":"https:\/\/howk.de\/?author=1"}]}},"_links":{"self":[{"href":"https:\/\/howk.de\/index.php?rest_route=\/wp\/v2\/posts\/4299"}],"collection":[{"href":"https:\/\/howk.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/howk.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/howk.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/howk.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4299"}],"version-history":[{"count":0,"href":"https:\/\/howk.de\/index.php?rest_route=\/wp\/v2\/posts\/4299\/revisions"}],"wp:attachment":[{"href":"https:\/\/howk.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4299"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/howk.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4299"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/howk.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4299"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}