Isopod

Abstract

Kubernetes is an open-source cluster orchestration system for containerized workloads to reduce idiosyncrasy across cloud vendors [2]. Using Kubernetes, Cruise has built a multi-tenant platform with thousands of cores and tens of terabytes of memory. Such a scale is possible in part thanks to the declarative abstraction of Kubernetes, where desired states are described in YAML manifests [5]. However, YAML as a data serialization format is unfit for workload specification. Structured data in YAML are untyped and prone to wrong indents and missing fields. Due to poor meta-programming support, composing YAML with control logic---loops and branches---suffers from YAML fragmentation and indentation tracking (example at bit.ly/yml-hell). Moreover, YAML manifests are often generated by filling a shared template with cluster-specific parameters---the image tag and the replica count might differ in development and production environments. Existing templating tools---Helm [11], Kustomize [9], Kapitan [7] and the likes---assume these parameters are statically known and use CLIs to query dynamic ones, such as secrets stored in HashiCorp Vault [10]. Such scheme is hard to test, since side effects escape through CLIs, and highly depends on the execution environment, since CLI versions vary across machines or might not exist. Not least, YAML manifests describe the eventual state but not how existing workloads will be affected. Blindly applying the manifest---for example, from a stale version of code---can be disastrous and cause unexpected outages. Isopod presents an alternative configuration paradigm by treating Kubernetes objects as first-class citizens. Without intermediate YAML artifacts, Isopod renders Kubernetes objects directly in Protocol Buffers [8], so they are strongly typed and consumed directly by the Kubernetes API. With Isopod, configurations are scripted in Starlark [3], a Python dialect by Google also used by Bazel [1] and Buck [4] build systems. To replace CLI dependencies, Isopod extends Starlark with runtime built-ins to access services and utilities such as Vault, Kubernetes apiserver, Base64 encoder, and UUID generator, etc. Isopod uses a separate runtime for unit tests to mock all built-ins, providing test coverage that was not possible before. Isopod is also hermetic and secure. The common reliance on the kubeconfig file for cluster authentication leaks secrets to disk, a security risk if working from a shared host, such as a cluster node or CICD worker. Instead, Isopod builds Oauth2 tokens [6] to the target cluster using the Identity & Access Management (IAM) service of the cloud vendor. Application secrets are stored in Vault and queried at runtime. Hence, no secrets escape to the disk. In fact, Isopod prohibits disk IO except for loading Starlark modules from other scripts. No external libraries can be loaded unless explicitly implemented as an Isopod built-in. Distributed as a single binary, Isopod is self-contained with all dependencies. Finally, Isopod is extensible. Protobuf packages of Kubernetes API groups added in the future can be loaded in the same way. Because built-ins are modular and pluggable, users can easily implement and register new built-ins with the Isopod runtime to support any Kubernetes vendors. Isopod offers many other features, such as object life cycle management and parallel rollout to multiple clusters, which is impossible if using kubeconfig. In dry-run mode, Isopod displays intended actions from the current code change as a YAML diff against live objects in the cluster to avoid unexpected configuration change. Since the adoption of Isopod, the PaaS team at Cruise has migrated 14 applications and added another 16 without outage or regression, totaling around 10,000 lines of Starlark. The migration results in up to 60% reduction in code size and 80% faster rollout due to code reuse, cluster parallelism, and the removal of YAML intermediaries. All unit tests take less than 10 secs to finish. Isopod is open source at github.com/cruise-automation/isopod.