CernVM-FS Container Image Integration

Abstract

Linux containers have gained widespread use in high energy physics, be it for services using container engines such as containerd/kubernetes, for production jobs using container engines such as Singularity or Shifter, or for development workflows using Docker as a local container engine. Thus the efficient distribution of the container images, whose size usually ranges from a few hundred megabytes to a few tens of gigabytes, is becoming a pressing concern. Because container images show similar characteristics than scientific application stacks, unpacking the images in CernVM-FS can remedy the distribution issues provided that the container engine at hand is able to use such unpacked images from CernVM-FS. In this contribution, we willl report on recent advances in the integration of Singularity, Docker, and containerd with CernVM-FS. We show improvements in the publishing of container images from a Docker registry that rely on new means of directly ingesting image tarballs. Well also show a repository file system structure for storing container images that are optimized for storing both container engines using flat root file systems (Singularity) as well as container engines using layers (containerd, Docker). To evaluate the benefits of our approach, we show concrete use cases and figures for production and development images from LHC experiments stored in the recently created unpacked.cern.ch repository.