Optimizing ATLAS data storage: The impact of compression algorithms on ATLAS physics analysis data formats

Abstract

The increased footprint foreseen for Run-3 and HL-LHC data will soon expose the limits of currently available storage and CPU resources. Data formats are already optimized according to the processing chain for which they are designed. ATLAS events are stored in ROOT-based reconstruction output files called Analysis Object Data (AOD), which are then processed within the derivation framework to produce Derived AOD (DAOD) files. Numerous DAOD formats, tailored for specific physics and performance groups, have been in use throughout the ATLAS Run-2 phase. In view of Run-3, ATLAS has changed its analysis model, which entailed a significant reduction of the existing DAOD flavors. Two new formats, unfiltered, skimmable on read and designed to meet the requirements of the majority of the analysis workflows, have been proposed as replacements: DAOD_PHYS and DAOD_PHYSLITE, a smaller format containing already calibrated physics objects. As ROOT-based formats, they natively support four lossless compression algorithms: lzma, lz4, zlib and zstd. In this study, the effects of different compression settings on file size, compression time, compression factor and reading speed are investigated considering both DAOD_PHYS and DAOD_PHYSLITE formats. Moreover, the impact of the AutoFlush parameter controlling how in-memory data structures are serialized to ROOT files, has been evaluated. This study yields new quantitative results that can serve as a paradigm on how to make compression decisions for different ATLAS use cases. As an example, for both DAOD_PHYS and DAOD_PHYSLITE, the lz4 library exhibits the fastest reading speed, but results in the largest files, whereas the lzma algorithm provides larger compression factors at the cost of significantly slower reading speeds. In addition, guidelines for setting appropriate AutoFlush values are outlined.