Stability and scalability of the CMS Global Pool: Pushing HTCondor and glideinWMS to new limits

J Balcas,J Marra Da Silva,D Mason,M Mascheroni,A Perez-Calero Yzquierdo,K Hurtado Anampa,K Larson,A Tiradani,D Hufnagel,J Letts,F Aftab Khan,B Bockelman

doi:10.1088/1742-6596/898/5/052031

Abstract

The CMS Global Pool, based on HTCondor and glideinWMS, is the main computing resource provisioning system for all CMS workflows, including analysis, Monte Carlo production, and detector data reprocessing activities. The total resources at Tier-1 and Tier-2 grid sites pledged to CMS exceed 100,000 CPU cores, while another 50,000 to 100,000 CPU cores are available opportunistically, pushing the needs of the Global Pool to higher scales each year. These resources are becoming more diverse in their accessibility and configuration over time. Furthermore, the challenge of stably running at higher and higher scales while introducing new modes of operation such as multi-core pilots, as well as the chaotic nature of physics analysis workflows, places huge strains on the submission infrastructure. This paper details some of the most important challenges to scalability and stability that the CMS Global Pool has faced since the beginning of the LHC Run II and how they were overcome.

Highlights

The main components of this Global Pool include a glideinWMS frontend and factories, the HTCondor Central Manager and Condor Connection Broker (CCB), deployed in 24-core 48GB (RAM) virtual machines (VMs) running on hypervisors with 10 Gbps ethernet connectivity
About 20 schedds dedicated to Monte Carlo simulation and other data processing activities are deployed at CERN and Fermi National Accelerator Laboratory (FNAL) on 16-core 64 GB physical machines and 24core 48 GB VMs
About ten schedds for physics analysis jobs are run at CERN on 24-core 48GB VMs and some physical machines at UCSD

Summary

Introduction

The main components of this Global Pool include a glideinWMS frontend and factories, the HTCondor Central Manager and Condor Connection Broker (CCB), deployed in 24-core 48GB (RAM) virtual machines (VMs) running on hypervisors with 10 Gbps ethernet connectivity. The main limits to the scalability of this system come from the I/O between the pool components, the combinatorics at the Negotiator, which has to process queued jobs against all potentially matching pilot execution slots, generically known as startds, as well as the speed of individual elements themselves.

Results

Conclusion