{"id":8357,"date":"2019-06-20T19:00:02","date_gmt":"2019-06-20T19:00:02","guid":{"rendered":"http:\/\/howk.de\/w1\/blog-future-of-crds-structural-schemas\/"},"modified":"2019-06-20T19:00:02","modified_gmt":"2019-06-20T19:00:02","slug":"blog-future-of-crds-structural-schemas","status":"publish","type":"post","link":"https:\/\/howk.de\/?p=8357","title":{"rendered":"Blog: Future of CRDs: Structural Schemas"},"content":{"rendered":"

Authors:<\/strong> Stefan Schimanski (Red Hat)<\/p>\n

CustomResourceDefinitions were introduced roughly two years ago as the primary way to extend the Kubernetes API with custom resources. From the beginning they stored arbitrary JSON data, with the exception that kind<\/code>, apiVersion<\/code> and metadata<\/code> had to follow the Kubernetes API conventions. In Kubernetes 1.8 CRDs gained the ability to define an optional OpenAPI v3 based validation schema.<\/p>\n

By the nature of OpenAPI specifications though\u2014only describing what must be there, not what shouldn\u2019t, and by being potentially incomplete specifications\u2014the Kubernetes API server never knew the complete structure of CustomResource instances. As a consequence, kube-apiserver\u2014until today\u2014stores all JSON data received in an API request (if it validates against the OpenAPI spec). This especially includes anything that is not specified in the OpenAPI schema.<\/p>\n

The story of malicious, unspecified data<\/h2>\n

To understand this, we assume a CRD for maintenance jobs by the operations team, running each night as a service user:<\/p>\n

\n
apiVersion: <\/span>operations\/v1\n<\/span><\/span>kind: <\/span>MaintenanceNightlyJob\n<\/span><\/span>spec:\n<\/span> <\/span>shell: <\/span>>\n<\/span> grep backdoor \/etc\/passwd ||\n<\/span> echo \u201cbackdoor:76asdfh76:\/bin\/bash\u201d >> \/etc\/passwd || true<\/span>\n<\/span> <\/span>machines: <\/span>[\u201caz1-master1\u201d,\u201daz1-master2\u201d,\u201daz2-master3\u201d]\n<\/span> <\/span>privileged: <\/span>true<\/span><\/code><\/pre>\n<\/div>\n
The privileged field is not specified by the operations team. Their controller does not know it, and their validating admission webhook does not know about it either. Nevertheless, kube-apiserver persists this suspicious, but unknown field without ever validating it.<\/p>\n
When run in the night, this job never fails, but because the service user is not able to write \/etc\/passwd<\/code>, it will also not cause any harm.<\/p>\n
The maintenance team needs support for privileged jobs. It adds the privileged<\/code> support, but is super careful to implement authorization for privileged jobs by only allowing those to be created by very few people in the company. That malicious job though has long been persisted to etcd. The next night arrives and the malicious job is executed.<\/p>\n
Towards complete knowledge of the data structure<\/h2>\n
This example shows that we cannot trust CustomResource data in etcd. Without having complete knowledge about the JSON structure, the kube-apsierver cannot do anything to prevent persistence of unknown data.<\/p>\n
Kubernetes 1.15 introduces the concept of a (complete) structural OpenAPI schema\u2014an OpenAPI schema with a certain shape, more in a second\u2014which will fill this knowledge gap.<\/p>\n
If the provided OpenAPI validation schema provided by the CRD author is not structural, violations are reported in a NonStructural<\/code> condition in the CRD.<\/p>\n
A structural schema for CRDs in apiextensions.k8s.io\/v1beta1<\/code> will not be required. But we plan to require structural schemas for every CRD created in apiextensions.k8s.io\/v1<\/code>, targeted for 1.16.<\/p>\n
But now let us see what a structural schema looks like.<\/p>\n
Structural Schema<\/h2>\n
The core of a structural schema<\/strong> is an OpenAPI v3 schema made out of<\/p>\n