Abstract
The ATLAS experiment is scaling up Big Data processing for the next LHC run using a multilevel workflow system comprised of many layers. In Big Data processing ATLAS deals with datasets, not individual files. Similarly a task (comprised of many jobs) has become a unit of the ATLAS workflow in distributed computing, with about 0.8M tasks processed per year. In order to manage the diversity of LHC physics (exceeding 35K physics samples per year), the individual data processing tasks are organized into workflows. For example, the Monte Carlo workflow is composed of many steps: generate or configure hard-processes, hadronize signal and minimum-bias (pileup) events, simulate energy deposition in the ATLAS detector, digitize electronics response, simulate triggers, reconstruct data, convert the reconstructed data into ROOT ntuples for physics analysis, etc. Outputs are merged and/or filtered as necessary to optimize the chain. The bi-level workflow manager - ProdSys2 - generates actual workflow tasks and their jobs are executed across more than a hundred distributed computing sites by PanDA - the ATLAS job-level workload management system. On the outer level, the Database Engine for Tasks (DEfT) empowers production managers with templated workflow definitions. On the next level, the Job Execution and Definition Interface (JEDI) is integrated with PanDA to provide dynamic job definition tailored to the sites capabilities. We report on scaling up the production system to accommodate a growing number of requirements from main ATLAS areas: Trigger, Physics and Data Preparation.
Highlights
Multilevel Workflow System in the ATLAS ExperimentM Borodin, K De2, J Garcia Navarro, D Golubkov, A Klimentov, T Maeno and A Vaniachine on behalf of the ATLAS Collaboration 1 Department of Elementary Particle Physics, National Research Nuclear University "MEPhI," Moscow, 117513, Russia 2 Physics Department, University of Texas Arlington, Arlington, TX 76019, United
The multi-purpose nature of the ATLAS experiment [1] at the LHC resulted in continuous growth in use cases for Big Data processing, as more data and new requirements emerge
To prepare the production for Run 2 challenges, PanDA has been upgraded with the Job Execution and Definition Interface (JEDI) [5], the production system enhanced with the Database Engine for Tasks (DEfT) [7], and the Distributed Data Management (DDM) system being upgraded to Rucio [8]
Summary
M Borodin, K De2, J Garcia Navarro, D Golubkov, A Klimentov, T Maeno and A Vaniachine on behalf of the ATLAS Collaboration 1 Department of Elementary Particle Physics, National Research Nuclear University "MEPhI," Moscow, 117513, Russia 2 Physics Department, University of Texas Arlington, Arlington, TX 76019, United. States of America 3 Instituto de Fisica Corpuscular, Universidad de Valencia, E-46980 Paterna, Spain 4 Experimental Physics Department, Institute for High Energy Physics, Protvino, 142281, Russia 5 Big Data Laboratory, National Research Centre "Kurchatov Institute" Moscow, 123182, Russia 6 Physics Department, Brookhaven National Laboratory, Bldg. 11973, United States of America 7 High Energy Physics Division, Argonne National Laboratory, 9700 South Cass
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
