The ATLAS Eventlndex: data flow and inclusion of other metadata

D Barberis,F Prokoshin,E J Gallas,J Hrivnac,S Gonzalez De La Hoz,R Toebbicke,J Sanchez,S E Cárdenas Zárate,D Malon,A Fernandez Casani,Atlas Collaboration ,A Favareto,J Salt,R Yuan,C Garcia Montoro

doi:10.1088/1742-6596/762/1/012028

Abstract

The ATLAS EventIndex is the catalogue of the event-related metadata for the information collected from the ATLAS detector. The basic unit of this information is the event record, containing the event identification parameters, pointers to the files containing this event as well as trigger decision information. The main use case for the EventIndex is event picking, as well as data consistency checks for large production campaigns. The EventIndex employs the Hadoop platform for data storage and handling, as well as a messaging system for the collection of information. The information for the EventIndex is collected both at Tier-0, when the data are first produced, and from the Grid, when various types of derived data are produced. The EventIndex uses various types of auxiliary information from other ATLAS sources for data collection and processing: trigger tables from the condition metadata database (COMA), dataset information from the data catalogue AMI and the Rucio data management system and information on production jobs from the ATLAS production system. The ATLAS production system is also used for the collection of event information from the Grid jobs. EventIndex developments started in 2012 and in the middle of 2015 the system was commissioned and started collecting event metadata, as a part of ATLAS Distributed Computing operations.

Highlights

The ATLAS experiment [1] runs a complex system that employs detectors with millions of readout channels, and gathers billions of events in data taking periods that may extend to several years
On the Grid the EventIndex uses ProdSys2 [10], a second generation ATLAS production system, to index datasets as soon as they appear in the ATLAS Metadata Interface (AMI) catalogue and are marked as valid and completed
It should be noted that HBase, Hadoop and Tomcat keeps some results in cache/memory so the second search is usually faster than the first one

Summary

Home Search Collections Journals About Contact us My IOPscience

This content has been downloaded from IOPscience. Please scroll down to see the full text. Ser. 762 012028 (http://iopscience.iop.org/1742-6596/762/1/012028) View the table of contents for this issue, or go to the journal homepage for more. Download details: IP Address: 188.184.3.56 This content was downloaded on 11/05/2017 at 10:00 Please note that terms and conditions apply. You may be interested in: Integration of the Eventlndex with other ATLAS systems D. D Barberis, S E Cárdenas Zárate, A Favareto, A Fernandez Casani, E J Gallas, C Garcia Montoro, S Gonzalez de la Hoz, J Hrivnac, D Malon, F Prokoshin2*, J Salt , J Sanchez, R Toebbicke , R Yuan on behalf of the ATLAS Collaboration

Introduction

Published under licence by IOP Publishing Ltd

Searched tag Files

Conclusions and outlook