Abstract
Rucio, the distributed data management system of the ATLAS experiment already manages more than 400 Petabytes of physics data on the grid. Rucio was incrementally improved throughout LHC Run-2 and is currently being prepared for the HL-LHC era of the experiment. Next to these improvements the system is currently evolving into a full-scale generic data management system for application beyond ATLAS, or even beyond high-energy physics. This contribution focuses on the development roadmap of Rucio for LHC Run-3, such as event level data management, generic meta-data support and increased usage of networks and tapes. At the same time Rucio is evolving beyond the original ATLAS requirements. This includes additional authentication mechanisms, generic database compatibility, deployment and packaging of the software stack in containers, and a project paradigm shift to a full-scale open source project..
Highlights
Managing large volumes of research data is a major challenge for any scientific project or experiment
The ATLAS Experiment [1] at the Large Hadron Collider (LHC) [2] at CERN [3] is a typical example for such massive data requirements
A complete new instance of Rucio has been setup only using Kubernetes/Helm for the WLCG Data Organization Management Access (DOMA) third party copy (TPC) working group to initiate a constant flow of TPC transfers between test endpoints
Summary
Managing large volumes of research data is a major challenge for any scientific project or experiment. The data requirements of these experiments are ever growing, leading to an unprecedented amount of required storage space and data organisation. These storage systems are typically heterogeneous and are distributed at multiple geographical locations under different administrative domains. The ATLAS Experiment [1] at the Large Hadron Collider (LHC) [2] at CERN [3] is a typical example for such massive data requirements. In this article we describe the evolution of Rucio to a generic, open-source, data management system for application beyond ATLAS.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
