The ATLAS EventIndex: architecture, design choices, deployment and first operation experience

D Barberis,F Prokoshin,J Sánchez,R Többicke,S González De La Hoz,A Favareto,S.E Cárdenas Zárate,J Cranshaw,C Glasman,E.J Gallas,J Hřivnáč,J Salt Cairols,D Malon,R Yuan,Á Fernández Casaní

doi:10.1088/1742-6596/664/4/042003

Abstract

The EventIndex is the complete catalogue of all ATLAS events, keeping the references to all files that contain a given event in any processing stage. It replaces the TAG database, which had been in use during LHC Run 1. For each event it contains its identifiers, the trigger pattern and the GUIDs of the files containing it. Major use cases are event picking, feeding the Event Service used on some production sites, and technical checks of the completion and consistency of processing campaigns. The system design is highly modular so that its components (data collection system, storage system based on Hadoop, query web service and interfaces to other ATLAS systems) could be developed separately and in parallel during LSI. The EventIndex is in operation for the start of LHC Run 2. This paper describes the high-level system architecture, the technical design choices and the deployment process and issues. The performance of the data collection and storage systems, as well as the query services, are also reported.

Highlights

■ Trigger patterns ■ References to the events at each processing stage (RAW, ESD, (x)AOD, NTUP) in all permanent files on storage generated by the ATLAS Production System ● Size and constraints ■ ATLAS collects a few billion real events each year of data taking and generates more than twice that number of simulated events ■ ~350 B/event ! 2 TB of raw information (6 TB after internal replication in Hadoop) in the EventIndex only for LHC Run 1 " The trigger rate for Run 2 is more than twice that for Run 1 " Simulated data not counted yet ■ Needs clever storage structure with smart search and retrieve tools
● The EventIndex infrastructure that was designed, developed and deployed in 2012-2015, is in operation ■ Run1 Tier-0 processing data fully indexed ■ Run1 reprocessed data being filled in ■ Run2 new data indexed in real time
● Initial use cases all satisfied with good performance ● Work continuing on system optimisation and increased functionality

Summary

Dario Barberis

D. Barberis (1), S.E. Cárdenas Zárate (2), J. (1) Università di Genova and INFN, Genova, Italy — (2) Universidad Técnica Federico Santa Maria, Valparaíso, Chile —. (4) Instituto de Física Corpuscular (IFIC), University of Valencia and CSIC, Valencia, Spain —. (5) University of Oxford, Oxford, UK — (6) Universidad Autónoma de Madrid, Madrid, Spain —. (7) LAL, Université Paris-SuDdaarnido CBNaRrbSe/IrNis2:PA3,TOLrAsSay,EFvreanntcIen—de(8x) CERN, Geneva, Switzerland

What is the ATLAS EventIndex?

Event list

Data Collection

Development Status

Summary and Outlook