Abstract

The ATLAS Distributed Data Management system stores more than 150PB of physics data across 120 sites globally. To cope with the anticipated ATLAS workload of the coming decade, Rucio, the next-generation data management system has been developed. Replica management, as one of the key aspects of the system, has to satisfy critical performance requirements in order to keep pace with the experiment's high rate of continual data generation. The challenge lies in meeting these performance objectives while still giving the system users and applications a powerful toolkit to control their data workflows. In this work we present the concept, design and implementation of the replica management in Rucio. We will specifically introduce the workflows behind replication rules, their formal language definition, weighting and site selection. Furthermore we will present the subscription component, which offers functionality for users to proclaim interest in data that has not been created yet. This contribution describes the concept and the architecture behind those components and will show the benefits made by this system.

Highlights

  • DQ2 is the current Distributed Data Management system of the ATLAS collaboration: Manages 150 PB of data, 750 storage endpoints on more than 120 sites globally, 1000 users

  • Replication requests can be both flexible and specific based on Rucio Storage Element (RSE) expressions

  • A replication rule consists of an RSE expression, defining a set of possible destination RSEs and the number of replicas it should create

Read more

Summary

Martin Barisits

DQ2 is the current Distributed Data Management system of the ATLAS collaboration: Manages 150 PB of data, 750 storage endpoints on more than 120 sites globally, 1000 users. Rucio[1] is the generation ATLAS distributed data management system to follow DQ2

Rucio concepts
Rucio Daemons
Replication Rules
RSE Selection
GGeaeramrmanan recmreoavteess flreapglication
Conclusion

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.