{"id":6743,"date":"2019-04-24T19:00:02","date_gmt":"2019-04-24T19:00:02","guid":{"rendered":"http:\/\/howk.de\/w1\/blog-hardware-accelerated-ssl-tls-termination-in-ingress-controllers-using-kubernetes-device-plugins-and-runtimeclass\/"},"modified":"2019-04-24T19:00:02","modified_gmt":"2019-04-24T19:00:02","slug":"blog-hardware-accelerated-ssl-tls-termination-in-ingress-controllers-using-kubernetes-device-plugins-and-runtimeclass","status":"publish","type":"post","link":"https:\/\/howk.de\/?p=6743","title":{"rendered":"Blog: Hardware Accelerated SSL\/TLS Termination in Ingress Controllers using Kubernetes Device Plugins and RuntimeClass"},"content":{"rendered":"

Authors:<\/strong> Mikko Ylinen (Intel)<\/p>\n

Abstract<\/h2>\n

A Kubernetes Ingress is a way to connect cluster services to the world outside the cluster. In order
\nto correctly route the traffic to service backends, the cluster needs an Ingress controller. The
\nIngress controller is responsible for setting the right destinations to backends based on the
\nIngress API objects\u2019 information. The actual traffic is routed through a proxy server that
\nis responsible for tasks such as load balancing and SSL\/TLS (later \u201cSSL\u201d refers to both SSL
\nor TLS ) termination. The SSL termination is a CPU heavy operation due to the crypto operations
\ninvolved. To offload some of the CPU intensive work away from the CPU, OpenSSL based proxy
\nservers can take the benefit of OpenSSL Engine API and dedicated crypto hardware. This frees
\nCPU cycles for other things and improves the overall throughput of the proxy server.<\/p>\n

In this blog post, we will show how easy it is to make hardware accelerated crypto available
\nfor containers running the Ingress controller proxy using some of the recently created Kubernetes
\nbuilding blocks: Device plugin framework and RuntimeClass. At the end, a reference setup is given
\nusing an HAproxy based Ingress controller accelerated using Intel® QuickAssist Technology cards.<\/p>\n

About Proxies, OpenSSL Engine and Crypto Hardware<\/h2>\n

The proxy server plays a vital role in a Kubernetes Ingress Controller function. It proxies
\nthe traffic to the backends per Ingress objects routes. Under heavy traffic load, the performance
\nbecomes critical especially if the proxying involves CPU intensive operations like SSL crypto.<\/p>\n

The OpenSSL project provides the widely adopted library for implementing the SSL protocol. Of
\nthe commonly known proxy servers used by Kubernetes Ingress controllers, Nginx and HAproxy use
\nOpenSSL. The CNCF graduated Envoy proxy uses BoringSSL but there seems to be community interest
\nin having OpenSSL as the alternative<\/a> for it too.<\/p>\n

The OpenSSL SSL protocol library relies on libcrypto that implements the cryptographic functions.
\nFor quite some time now (first introduced in 0.9.6 release), OpenSSL has provided an ENGINE
\nconcept<\/a> that allows these cryptographic operations to be offloaded to a dedicated crypto
\nacceleration hardware. Later, a special dynamic<\/em> ENGINE enabled the crypto hardware specific
\npieces to be implemented in an independent loadable module that can be developed outside the
\nOpenSSL code base and distributed separately. From the application\u2019s perspective, this is also
\nideal because they don\u2019t need to know the details of how to use the hardware, and the hardware
\nspecific module can be loaded\/used when the hardware is available.<\/p>\n

Hardware based crypto can greatly improve Cloud applications\u2019 performance due to hardware
\naccelerated processing in SSL operations as discussed, and can provide other crypto
\nservices like key\/random number generation. Clouds can make the hardware easily available
\nusing the dynamic ENGINE and several loadable module implementations exist, for
\nexample, CloudHSM<\/a>, IBMCA<\/a>, or QAT Engine<\/a>.<\/p>\n

For Cloud deployments, the ideal scenario is for these modules to be shipped as part of
\nthe container workload. The workload would get scheduled on a node that provides the
\nunderlying hardware that the module needs to access. On the other hand, the workloads
\nshould run the same way and without code modifications regardless of the crypto acceleration
\nhardware being available or not. The OpenSSL dynamic engine enables this. Figure 1 below
\nillustrates these two scenarios using a typical Ingress Controller container as an example.
\nThe red colored boxes indicate the differences between a container with a crypto hardware
\nengine enabled container vs. a \u201cstandard\u201d one. It\u2019s worth pointing out that the configuration
\nchanges shown do not necessarily require another version of the container since the configurations
\ncould be managed, e.g., using ConfigMaps.<\/p>\n

\n\"Figure
\n

Figure 1. Examples of Ingress controller containers<\/p>\n<\/figcaption><\/figure>\n

Hardware Resources and Isolation<\/h2>\n

To be able to deploy workloads with hardware dependencies, Kubernetes provides excellent extension
\nand configurability mechanisms. Let\u2019s take a closer look into Kubernetes the device plugin framework<\/a>
\n(beta in 1.14) and RuntimeClass<\/a> (beta in 1.14) and learn how they can be leveraged to expose crypto
\nhardware to workloads.<\/p>\n

The device plugin framework, first introduced in Kubernetes 1.8, provides a way for hardware vendors
\nto register and allocate node hardware resources to Kubelets. The plugins implement the hardware
\nspecific initialization logic and resource management. The pods can request hardware resources in
\ntheir PodSpec, which also guarantees the pod is scheduled on a node that can provide those resources.<\/p>\n

The device resource allocation for containers is non-trivial. For applications dealing with security,
\nthe hardware level isolation is critical. The PCIe based crypto acceleration device functions can
\nbenefit from IO hardware virtualization, through an I\/O Memory Management Unit (IOMMU), to provide
\nthe isolation: an IOMMU group<\/em> the device belongs to provides the isolated resource for a workload
\n(assuming the crypto cards do not share the IOMMU group with other devices). The number of isolated
\nresources can be further increased if the PCIe device supports the Single-Root I\/O Virtualization
\n(SR-IOV) specification. SR-IOV allows the PCIe device to be split further to virtual functions<\/em> (VF),
\nderived from physical function<\/em> (PF) devices, and each belonging to their own IOMMU group. To expose
\nthese IOMMU isolated device functions to user space and containers, the host kernel should bind
\nthem to a specific device driver. In Linux, this driver is vfio-pci and it makes each device
\navailable through a character device in user space. The kernel vfio-pci driver provides user space
\napplications with a direct, IOMMU backed access to PCIe devices and functions, using a mechanism
\ncalled PCI passthrough<\/em>. The interface can be leveraged by user space frameworks, such as the
\nData Plane Development Kit (DPDK). Additionally, virtual machine (VM) hypervisors can provide
\nthese user space device nodes to VMs and expose them as PCI devices to the guest kernel.
\nAssuming support from the guest kernel, the VM gets close to native performant direct access to the
\nunderlying host devices.<\/p>\n

To advertise these device resources to Kubernetes, we can have a simple Kubernetes device plugin
\nthat runs the initialization (i.e., binding), calls kubelet\u2019s Registration<\/code> gRPC service, and
\nimplements the DevicePlugin gRPC service that kubelet calls to, e.g., to Allocate<\/code> the resources
\nupon Pod creation.<\/p>\n

Device Assignment and Pod Deployment<\/h2>\n

At this point, you may ask what the container could do with a VFIO device node? The answer comes
\nafter we first take a quick look into the Kubernetes RuntimeClass.<\/p>\n

The Kubernetes RuntimeClass was created to provide better control and configurability
\nover a variety of runtimes<\/em> (an earlier blog post<\/a> goes into the details of the needs,
\nstatus and roadmap for it) that are available in the cluster. In essence, the RuntimeClass
\nprovides cluster users better tools to pick and use the runtime that best suits for the pod use case.<\/p>\n

The OCI compatible Kata Containers runtime<\/a> provides workloads with a hardware virtualized
\nisolation layer. In addition to workload isolation, the Kata Containers VM has the added
\nside benefit that the VFIO devices, as Allocate<\/code>\u2019d by the device plugin, can be passed
\nthrough to the container as hardware isolated devices. The only requirement is that the
\nKata Containers kernel has driver for the exposed device enabled.<\/p>\n

That\u2019s all it really takes to enable hardware accelerated crypto for container workloads. To summarize:<\/p>\n

    \n
  1. Cluster needs a device plugin running on the node that provides the hardware<\/li>\n
  2. Device plugin exposes the hardware to user space using the VFIO driver<\/li>\n
  3. Pod requests the device resources and Kata Containers as the RuntimeClass in the PodSpec<\/li>\n
  4. The container has the hardware adaptation library and the OpenSSL engine module<\/li>\n<\/ol>\n

    Figure 2 shows the overall setup using the Container A illustrated earlier.<\/p>\n

    \n\"Figure
    \n

    Figure 2. Deployment overview<\/p>\n<\/figcaption><\/figure>\n

    Reference Setup<\/h2>\n

    Finally, we describe the necessary building blocks and steps to build a functional
    \nsetup described in Figure 2 that enables hardware accelerated SSL termination in
    \nan Ingress Controller using an Intel® QuickAssist Technology (QAT) PCIe device.
    \nIt should be noted that the use cases are not limited to Ingress controllers, but
    \nany OpenSSL based workload can be accelerated.<\/p>\n

    Cluster configuration:<\/h3>\n