The Quest to solve the HL-LHC data access puzzle

X Espinal,A Sciabà,I Vukotic,F Wuerthwein,M Schulz,S Jezequel

doi:10.1051/epjconf/202024504027

Abstract

HL-LHC will confront the WLCG community with enormous data storage, management and access challenges. These are as much technical as economical. In the WLCG-DOMA Access working group, members of the experiments and site managers have explored different models for data access and storage strategies to reduce cost and complexity, taking into account the boundary conditions given by our community.Several of these scenarios have been evaluated quantitatively, such as the Data Lake model and incremental improvements of the current computing model with respect to resource needs, costs and operational complexity.To better understand these models in depth, analysis of traces of current data accesses and simulations of the impact of new concepts have been carried out. In parallel, evaluations of the required technologies took place. These were done in testbed and production environments at small and large scale.We will give an overview of the activities and results of the working group, describe the models and summarise the results of the technology evaluation focusing on the impact of storage consolidation in the form of Data Lakes, where the use of streaming caches has emerged as a successful approach to reduce the impact of latency and bandwidth limitation.We will describe the experience and evaluation of these approaches in different environments and usage scenarios. In addition we will present the results of the analysis and modelling efforts based on data access traces of the experiments.

Highlights

The WLCG strategy paper [1] set out the path towards computing for the High-Luminosity LHC (HL-LHC) era, building up from the input provided by the HSF [2] Community White Paper [3]
The working group is investigating the potential benefits of data caching infrastructures and promoting their deployment within a consolidated storage infrastructure labeled as WLCGData Lake (Fig. 1), on referred to as the Data Lake
In the upcoming year we will increase our understanding about the operations and performance of the different caching infrastructures that are running worldwide for ATLAS and CMS

Summary

Introduction

The WLCG strategy paper [1] set out the path towards computing for the High-Luminosity LHC (HL-LHC) era, building up from the input provided by the HSF [2] Community White Paper [3]. The estimates for the data volumes and computing show a major step up from the current needs and a program of work was established from the WLCG point of view to address this future challenge. One of the charges is addressed by the DOMA Access Working Group to evaluate future data access scenarios. The DOMA Access Working Group collected information from the experiments about the evolution of their computing models and future plans for user data analysis. The working group is investigating the potential benefits of data caching infrastructures and promoting their deployment within a consolidated storage infrastructure labeled as WLCGData Lake (Fig. 1), on referred to as the Data Lake

Compact analysis objects

File usability and data access patterns

Findings

Conclusions