Building HEP Software with Spack: Experiences from Pilot Builds for Key4hep and Outlook for LCG Releases

Valentin Volkl,Dmitri Konstantinov,Thomas Madlener,Andre Sailer,Ivan Razumov,Tao Lin,Gerardo Ganis,Joseph Wang,C Biscarat,S Campana,S Roiser,C.I Rovelli,G.A Stewart,B Hegner

doi:10.1051/epjconf/202125103056

Abstract

Consistent, efficient software builds and deployments are a common concern for all HEP experiments. This paper describes the evolution of the usage of the Spack package manager in HEP in the context of the LCG stacks and the current Spack-based management of Key4hep software. Whereas previously Key4hep software used Spack only for a thin layer of FCC experiment software on top of the LCG releases, it is now possible to build the complete stack, from system libraries to FCC-, iLCSoft- and CEPC software packages with Spack. This pilot build doubles as a prototype for a Spack-based LCG release. The workflows and mechanisms that can be used for this purpose, potential for improvement as well as the roadmap towards a complete LCG release in spack are discussed.

Highlights

Spack [1] is a widely used build tool and package manager for scientific software, and is one of the candidates to replace LCGCMake [2], the tool currently used to build/manage the LCG stacks
The workflows and mechanisms that can be used for this purpose, potential for improvement as well as the roadmap towards a complete LCG release in spack are discussed
Investigations of the use of Spack for LCG builds were initiated by the HEP Software

Summary

Introduction

Spack [1] is a widely used build tool and package manager for scientific software, and is one of the candidates to replace LCGCMake [2], the tool currently used to build/manage the LCG stacks. Abridged example of the recipe for the DD4hep package, showing how versions, patches and build options can be declared in Spack. "Variants" are custom configurations of the packages that result in different installations These are mapped to build options and enable or disable optional parts of the package. This allows to install multiple configurations of the same version of a package This unique identifier helps reproducibility and allows the installation in arbitrary prefixes – when setting up a package the dependencies are identified by hash and loaded via RPATH and environment variables (depending on the build system type, these may include standard variables like CMAKE_PREFIX_PATH or custom ones as set in the package recipe). The LCG stacks are deployments of almost 500 HEP and general-purpose software packages that are used as dependencies for numerous experiments like ATLAS and LHCb. New LCG builds are released several times per year, in 10-20 configurations for different compilers, release/debug build types, operating systems and architectures. This section details the problems encountered during the pilot build, and lists workarounds and relevant development lines in spack

Concretizer

Data Packages

Binary caches

Distribution of spack on read-only file systems

Conclusion and Outlook