Abstract
High performance computing facilities present unique challenges and opportunities for HEP event processing. The massive scale of many HPC systems means that fractionally small utilization can yield large returns in processing throughput. Parallel applications which can dynamically and efficiently fill any scheduling opportunities the resource presents benefit both the facility (maximal utilization) and the (compute-limited) science. The ATLAS Yoda system provides this capability to HEP-like event processing applications by implementing event-level processing in an MPI-based master-client model that integrates seamlessly with the more broadly scoped ATLAS Event Service. Fine grained, event level work assignments are intelligently dispatched to parallel workers to sustain full utilization on all cores, with outputs streamed off to destination object stores in near real time with similarly fine granularity, such that processing can proceed until termination with full utilization. The system offers the efficiency and scheduling flexibility of preemption without requiring the application actually support or employ check-pointing. We will present the new Yoda system, its motivations, architecture, implementation, and applications in ATLAS data processing at several US HPC centers.
Highlights
With the increased data volume recorded during LHC Run2 and beyond, it becomes critical for the experiments to efficiently use all CPU power available to them, and to leverage computing resources they don’t own
After processing each event range, the Event Service saves the output file to a secure location, such that Event Service jobs can be terminated practically at any time with minimal data losses. Another requirement for efficient running on high performance computing resources (HPC) systems is that the application has to leverage MPI mechanisms in order to be able to run on many compute nodes simultaneously
For this purpose we have developed an MPI-based implementation of the Event Service (Yoda), which is able to run on HPC compute nodes with no internet connectivity with the outside world
Summary
This content has been downloaded from IOPscience. Please scroll down to see the full text. Ser. 664 092025 (http://iopscience.iop.org/1742-6596/664/9/092025) View the table of contents for this issue, or go to the journal homepage for more. Download details: IP Address: 137.138.124.206 This content was downloaded on 24/02/2016 at 13:49 Please note that terms and conditions apply. 21st International Conference on Computing in High Energy and Nuclear Physics (CHEP2015) IOP Publishing. Journal of Physics: Conference Series 664 (2015) 092025 doi:10.1088/1742-6596/664/9/092025
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
