Automated Discovery of Container Executables

Abstract

Linux container technologies such as Docker and Singularity offer encapsulated environments for easy execution of software. In high performance computing, this is especially important for evolving and complex software stacks with conflicting dependencies that must co-exist. Singularity Registry HPC (“shpc”) was created as an effort to install containers in this environment as modules, seamlessly allowing for typically hidden executables inside containers to be presented to the user as commands, and as such significantly simplifying the user experience. A remaining challenge, however, is deriving the list of important executables in the container. In this work, we present a new modular methodology that allows for discovering new containers in large community sets, deriving container entries with relevant executables therein, and fully automating both recipe generation and updates over time. As an exemplar outcome, we have employed this methodology to add to the Registry over 8,000 containers from the BioContainers community that can be maintained and updated by the software automation. All software is publicly available on the GitHub platform, and can be beneficial to container registries and infrastructure providers for automatically generating container modules, thus lowering the usage entry barrier and improving user experience.