{"id":6546,"date":"2019-04-15T19:00:03","date_gmt":"2019-04-15T19:00:03","guid":{"rendered":"http:\/\/howk.de\/w1\/blog-process-id-limiting-for-stability-improvements-in-kubernetes-1-14\/"},"modified":"2019-04-15T19:00:03","modified_gmt":"2019-04-15T19:00:03","slug":"blog-process-id-limiting-for-stability-improvements-in-kubernetes-1-14","status":"publish","type":"post","link":"https:\/\/howk.de\/?p=6546","title":{"rendered":"Blog: Process ID Limiting for Stability Improvements in Kubernetes 1.14"},"content":{"rendered":"<p><strong>Author: Derek Carr<\/strong><\/p>\n<p>Have you ever seen someone take more than their fair share of the cookies? The one person who reaches in and grabs a half dozen fresh baked chocolate chip chunk morsels and skitters off like Cookie Monster exclaiming \u201cOm nom nom nom.\u201d<\/p>\n<p>In some rare workloads, a similar occurrence was taking place inside Kubernetes clusters. With each Pod and Node, there comes a finite number of possible process IDs (PIDs) for all applications to share. While it is rare for any one process or pod to reach in and grab all the PIDs, some users were experiencing resource starvation due to this type of behavior. So in Kubernetes 1.14, we introduced an enhancement to mitigate the risk of a single pod monopolizing all of the PIDs available.<\/p>\n<h2 id=\"can-you-spare-some-pids\">Can You Spare Some PIDs?<\/h2>\n<p>Here, we\u2019re talking about the greed of certain containers. Outside the ideal, runaway processes occur from time to time, particularly in clusters where testing is taking place. Thus, some wildly non-production-ready activity is happening.<\/p>\n<p>In such a scenario, it\u2019s possible for something akin to a fork bomb taking place inside a node. As resources slowly erode, being taken over by some zombie-like process that continually spawns children, other legitimate workloads begin to get bumped in favor of this inflating balloon of wasted processing power. This could result in other processes on the same pod being starved of their needed PIDs. It could also lead to interesting side effects as a node could fail and a replica of that pod is scheduled to a new machine where the process repeats across your entire cluster.<\/p>\n<h2 id=\"fixing-the-problem\">Fixing the Problem<\/h2>\n<p>Thus, in Kubernetes 1.14, we have added a feature that allows for the configuration of a kubelet to limit the number of PIDs a given pod can consume. If that machine supports 32,768 PIDs and 100 pods, one can give each pod a budget of 300 PIDs to prevent total exhaustion of PIDs. If the admin wants to overcommit PIDs similar to cpu or memory, they may do so as well with some additional risks. Either way, no one pod can bring the whole machine down. This will generally prevent against simple fork bombs from taking over your cluster.<\/p>\n<p>This change allows administrators to protect one pod from another, but does not ensure if all pods on the machine can protect the node, and the node agents themselves from falling over. Thus, we\u2019ve introduced a feature in this release in alpha form that provides isolation of PIDs from end user workloads on a pod from the node agents (kubelet, runtime, etc.). The admin is able to reserve a specific number of PIDs&ndash;similar to how one reserves CPU or memory today&ndash;and ensure they are never consumed by pods on that machine. Once that graduates from alpha, to beta, then stable in future releases of Kubernetes, we\u2019ll have protection against an easily starved Linux resource.<\/p>\n<p>Get started with <a href=\"https:\/\/github.com\/kubernetes\/kubernetes\/releases\/tag\/v1.14.0\" target=\"_blank\">Kubernetes 1.14<\/a>.<\/p>\n<h2 id=\"get-involved\">Get Involved<\/h2>\n<p>If you have feedback for this feature or are interested in getting involved with the design and development, join the <a href=\"https:\/\/github.com\/kubernetes\/community\/tree\/master\/sig-node\" target=\"_blank\">Node Special Interest Group<\/a>.<\/p>\n<h3 id=\"about-the-author\">About the author:<\/h3>\n<p>Derek Carr is Senior Principal Software Engineer at Red Hat. He is a Kubernetes contributor and member of the Kubernetes Community Steering Committee.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Author: Derek Carr Have you ever seen someone take more than their fair share of the cookies? The one person who reaches in and grabs a half dozen fresh baked chocolate chip chunk morsels and skitters off like Cookie Monster exclaiming \u201cOm nom nom nom.\u201d In some rare workloads, a similar occurrence was taking place [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.9.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Blog: Process ID Limiting for Stability Improvements in Kubernetes 1.14 - Howk IT-Dienstleistungen<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/howk.de\/?p=6546\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Blog: Process ID Limiting for Stability Improvements in Kubernetes 1.14 - Howk IT-Dienstleistungen\" \/>\n<meta property=\"og:description\" content=\"Author: Derek Carr Have you ever seen someone take more than their fair share of the cookies? The one person who reaches in and grabs a half dozen fresh baked chocolate chip chunk morsels and skitters off like Cookie Monster exclaiming \u201cOm nom nom nom.\u201d In some rare workloads, a similar occurrence was taking place [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/howk.de\/?p=6546\" \/>\n<meta property=\"og:site_name\" content=\"Howk IT-Dienstleistungen\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/howk.de\" \/>\n<meta property=\"article:published_time\" content=\"2019-04-15T19:00:03+00:00\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/howk.de\/?p=6546#article\",\"isPartOf\":{\"@id\":\"https:\/\/howk.de\/?p=6546\"},\"author\":{\"name\":\"admin\",\"@id\":\"https:\/\/howk.de\/#\/schema\/person\/b029bd02d4f35dce869ef54c81a100c5\"},\"headline\":\"Blog: Process ID Limiting for Stability Improvements in Kubernetes 1.14\",\"datePublished\":\"2019-04-15T19:00:03+00:00\",\"dateModified\":\"2019-04-15T19:00:03+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/howk.de\/?p=6546\"},\"wordCount\":552,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/howk.de\/#organization\"},\"articleSection\":[\"Hi Tech\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/howk.de\/?p=6546#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/howk.de\/?p=6546\",\"url\":\"https:\/\/howk.de\/?p=6546\",\"name\":\"Blog: Process ID Limiting for Stability Improvements in Kubernetes 1.14 - Howk IT-Dienstleistungen\",\"isPartOf\":{\"@id\":\"https:\/\/howk.de\/#website\"},\"datePublished\":\"2019-04-15T19:00:03+00:00\",\"dateModified\":\"2019-04-15T19:00:03+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/howk.de\/?p=6546#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/howk.de\/?p=6546\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/howk.de\/?p=6546#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/howk.de\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Blog: Process ID Limiting for Stability Improvements in Kubernetes 1.14\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/howk.de\/#website\",\"url\":\"https:\/\/howk.de\/\",\"name\":\"Howk IT-Dienstleistungen\",\"description\":\"Howk IT Services - Howk IT-Dienstleistungen\",\"publisher\":{\"@id\":\"https:\/\/howk.de\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/howk.de\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/howk.de\/#organization\",\"name\":\"HowK\",\"url\":\"https:\/\/howk.de\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/howk.de\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/howk.de\/w1\/wp-content\/uploads\/2013\/12\/howk-logo.png\",\"contentUrl\":\"https:\/\/howk.de\/w1\/wp-content\/uploads\/2013\/12\/howk-logo.png\",\"width\":170,\"height\":170,\"caption\":\"HowK\"},\"image\":{\"@id\":\"https:\/\/howk.de\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/howk.de\",\"http:\/\/de.linkedin.com\/in\/howkde\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/howk.de\/#\/schema\/person\/b029bd02d4f35dce869ef54c81a100c5\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/howk.de\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/b5a20f4d07bca1b73f25cff58a1116c4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/b5a20f4d07bca1b73f25cff58a1116c4?s=96&d=mm&r=g\",\"caption\":\"admin\"},\"url\":\"https:\/\/howk.de\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Blog: Process ID Limiting for Stability Improvements in Kubernetes 1.14 - Howk IT-Dienstleistungen","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/howk.de\/?p=6546","og_locale":"en_US","og_type":"article","og_title":"Blog: Process ID Limiting for Stability Improvements in Kubernetes 1.14 - Howk IT-Dienstleistungen","og_description":"Author: Derek Carr Have you ever seen someone take more than their fair share of the cookies? The one person who reaches in and grabs a half dozen fresh baked chocolate chip chunk morsels and skitters off like Cookie Monster exclaiming \u201cOm nom nom nom.\u201d In some rare workloads, a similar occurrence was taking place [&hellip;]","og_url":"https:\/\/howk.de\/?p=6546","og_site_name":"Howk IT-Dienstleistungen","article_publisher":"https:\/\/www.facebook.com\/howk.de","article_published_time":"2019-04-15T19:00:03+00:00","author":"admin","twitter_card":"summary_large_image","twitter_misc":{"Written by":"admin","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/howk.de\/?p=6546#article","isPartOf":{"@id":"https:\/\/howk.de\/?p=6546"},"author":{"name":"admin","@id":"https:\/\/howk.de\/#\/schema\/person\/b029bd02d4f35dce869ef54c81a100c5"},"headline":"Blog: Process ID Limiting for Stability Improvements in Kubernetes 1.14","datePublished":"2019-04-15T19:00:03+00:00","dateModified":"2019-04-15T19:00:03+00:00","mainEntityOfPage":{"@id":"https:\/\/howk.de\/?p=6546"},"wordCount":552,"commentCount":0,"publisher":{"@id":"https:\/\/howk.de\/#organization"},"articleSection":["Hi Tech"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/howk.de\/?p=6546#respond"]}]},{"@type":"WebPage","@id":"https:\/\/howk.de\/?p=6546","url":"https:\/\/howk.de\/?p=6546","name":"Blog: Process ID Limiting for Stability Improvements in Kubernetes 1.14 - Howk IT-Dienstleistungen","isPartOf":{"@id":"https:\/\/howk.de\/#website"},"datePublished":"2019-04-15T19:00:03+00:00","dateModified":"2019-04-15T19:00:03+00:00","breadcrumb":{"@id":"https:\/\/howk.de\/?p=6546#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/howk.de\/?p=6546"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/howk.de\/?p=6546#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/howk.de\/"},{"@type":"ListItem","position":2,"name":"Blog: Process ID Limiting for Stability Improvements in Kubernetes 1.14"}]},{"@type":"WebSite","@id":"https:\/\/howk.de\/#website","url":"https:\/\/howk.de\/","name":"Howk IT-Dienstleistungen","description":"Howk IT Services - Howk IT-Dienstleistungen","publisher":{"@id":"https:\/\/howk.de\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/howk.de\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/howk.de\/#organization","name":"HowK","url":"https:\/\/howk.de\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/howk.de\/#\/schema\/logo\/image\/","url":"https:\/\/howk.de\/w1\/wp-content\/uploads\/2013\/12\/howk-logo.png","contentUrl":"https:\/\/howk.de\/w1\/wp-content\/uploads\/2013\/12\/howk-logo.png","width":170,"height":170,"caption":"HowK"},"image":{"@id":"https:\/\/howk.de\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/howk.de","http:\/\/de.linkedin.com\/in\/howkde"]},{"@type":"Person","@id":"https:\/\/howk.de\/#\/schema\/person\/b029bd02d4f35dce869ef54c81a100c5","name":"admin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/howk.de\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/b5a20f4d07bca1b73f25cff58a1116c4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/b5a20f4d07bca1b73f25cff58a1116c4?s=96&d=mm&r=g","caption":"admin"},"url":"https:\/\/howk.de\/?author=1"}]}},"_links":{"self":[{"href":"https:\/\/howk.de\/index.php?rest_route=\/wp\/v2\/posts\/6546"}],"collection":[{"href":"https:\/\/howk.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/howk.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/howk.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/howk.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6546"}],"version-history":[{"count":0,"href":"https:\/\/howk.de\/index.php?rest_route=\/wp\/v2\/posts\/6546\/revisions"}],"wp:attachment":[{"href":"https:\/\/howk.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6546"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/howk.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6546"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/howk.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6546"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}