CernVM-FS ephemeral publishers on Kubernetes

Abstract

The CernVM File System (CernVM-FS) is a global read-only POSIX file system that provides scalable and reliable software distribution to numerous scientific collaborations. It gives access to more than a billion binary files of experiment application software stacks and operating system containers to end user devices, grids, clouds, and supercomputers. CernVM-FS is asymmetric by construction. Writing into the repository is a centralized operation called publishing, while reading is allowed for many clients from many locations. The classic publishing process needs a dedicated “release manager machine” that provides the editable repository copy. This classic approach was improved thanks to the introduction of the CernVM-FS Gateway that provides concurrent access to the repository backend storage through a REST API. In this contribution, we present further improvements to the CernVM-FS publishing process. Our main contribution is the construction of ephemeral containers that are created on demand and used to provide a temporary, editable repository copy for a single publish operation. The container construction makes careful use of Linux namespaces and a user-space implementation of overlayfs. We further show that both the gateway and the containers used for publishing can be instantiated as pods in a kubernetes cluster. Thus, we demonstrate a kubernetes-native CernVM-FS publishing workflow.