Scaling up ATLAS Event Service to production levels on opportunistic computing platforms

D Benjamin,T Wenaus,P Van Gemmeren,F Wang,J Caballero,On Behalf Of The Atlas Collaboratio ,P Nilsson,M Ernst,A Vaniachine,W Guan,D Lesny,V Tsulaia,T Maeno,J Hover

doi:10.1088/1742-6596/762/1/012027

Abstract

Continued growth in public cloud and HPC resources is on track to exceed the dedicated resources available for ATLAS on the WLCG. Examples of such platforms are Amazon AWS EC2 Spot Instances, Edison Cray XC30 supercomputer, backfill at Tier 2 and Tier 3 sites, opportunistic resources at the Open Science Grid (OSG), and ATLAS High Level Trigger farm between the data taking periods. Because of specific aspects of opportunistic resources such as preemptive job scheduling and data I/O, their efficient usage requires workflow innovations provided by the ATLAS Event Service. Thanks to the finer granularity of the Event Service data processing workflow, the opportunistic resources are used more efficiently. We report on our progress in scaling opportunistic resource usage to double-digit levels in ATLAS production.

Highlights

Continued growth in public cloud and HPC resources is on track to exceedbthuelle n 1BJeacnauuasrye 2o0f 1s4pecific aspects of opportunistic resources such as preemptive job scheduling dedicated resources available for ATLAS on the WLCG
Requirements: We have to be agile in how we use these opportunistic resources: processiInNgTisRaOn DimUpoCrtTaInOt mNeans for maximizing ATLAS computing throughput. Quick start when they become available within buTdhgeeAt CcAoTnswtroarinkstsh.oEpxsaemriepsle, sfoorfmseurclyh konpopwonrtuansisAtiIcHEreNsPou(Arcrteifsicaiarel Intelligen ceQuinicHkigehxitEwnehregny tahnedy are about to disappear
Mainly focusing on high-energy physics, talks related to nuclear physics, astrophysics, laser and condensed matter physics, earth physics, biophysics, and others, are most welcome

Summary

Introduction

Continued growth in public cloud and HPC resources is on track to exceedbthuelle n 1BJeacnauuasrye 2o0f 1s4pecific aspects of opportunistic resources such as preemptive job scheduling dedicated resources available for ATLAS on the WLCG. In scaling opportunistic resource usage to double-digit levels in ATLAS production. Motivations: Using opportunistic resources efficiently and fully for ATLAS Requirements: We have to be agile in how we use these opportunistic resources: processiInNgTisRaOn DimUpoCrtTaInOt mNeans for maximizing ATLAS computing throughput

Results

Conclusion