High Level Trigger Research Articles

Development of the CMS detector for the CERN LHC Run 3

Since the initial data taking of the CERN LHC, the CMS experiment has undergone substantial upgrades and improvements. This paper discusses the CMS detector as it is configured for the third data-taking period of the CERN LHC, Run 3, which started in 2022. The entire silicon pixel tracking detector was replaced. A new powering system for the superconducting solenoid was installed. The electronics of the hadron calorimeter was upgraded. All the muon electronic systems were upgraded, and new muon detector stations were added, including a gas electron multiplier detector. The precision proton spectrometer was upgraded. The dedicated luminosity detectors and the beam loss monitor were refurbished. Substantial improvements to the trigger, data acquisition, software, and computing systems were also implemented, including a new hybrid CPU/GPU farm for the high-level trigger.

Autoencoders for real-time SUEP detection

Confining dark sectors with pseudo-conformal dynamics can produce Soft Unclustered Energy Patterns (SUEP), at the Large Hadron Collider: the production of dark quarks in proton–proton collisions leading to a dark shower and the high-multiplicity production of dark hadrons. The final experimental signature is spherically symmetric energy deposits by an anomalously large number of soft Standard Model particles with a transverse energy of O(100)MeV\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\,\ ext {MeV}$$\\end{document}. Assuming Yukawa-like couplings of the scalar portal state, the dominant production mode is gluon fusion, and the dominant background comes from multi-jet QCD events. We have developed a deep learning-based Anomaly Detection technique to reject QCD jets and identify any anomalous signature, including SUEP, in real-time in the High-Level Trigger system of experiments like the Compact Muon Solenoid at the Large Hadron Collider. A deep convolutional neural autoencoder network has been trained using QCD events by taking transverse energy deposits in the inner tracker, electromagnetic calorimeter, and hadron calorimeter sub-detectors as 3-channel image data. Due to the sparse nature of the data, only ∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document}0.5% of the total ∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document}300k\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${300}\\,\ extrm{k}$$\\end{document} image pixels have nonzero values. To tackle this challenge, a nonstandard loss function, the inverse of the so-called Dice Loss, is exploited. The trained autoencoder with learned spatial features of QCD jets can detect 40% of the SUEP events, with a QCD event mistagging rate as low as 2%. The model inference time has been measured using the Intel® CoreTM\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{\ exttt {TM}}$$\\end{document} i5-9600KF processor and found to be ∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document}20ms\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${20}\ extrm{ms}$$\\end{document}, which perfectly satisfies the High-Level Trigger system’s latency of O(102)ms\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {O}(10^2)~\ extrm{ms}$$\\end{document}. Given the virtue of the unsupervised learning of the autoencoders, the trained model can be applied to any new physics model that predicts an experimental signature anomalous to QCD jets.

Design and implementation of neural network based conditions for the CMS Level-1 Global Trigger upgrade for the HL-LHC

Abstract The CMS detector will be upgraded to maintain, or even improve, the physics acceptance under the harsh data taking conditions foreseen during the High-Luminosity LHC operations. In particular, the trigger system (Level-1 and High Level Triggers) will be completely redesigned to utilize detailed information from sub-detectors at the bunch crossing rate: the upgraded Global Trigger will use high-precision trigger objects to provide the Level-1 decision. Besides cut-based algorithms, novel machine-learning-based algorithms will also be included in the Global Trigger to achieve a higher selection efficiency and detect unexpected signals. Implementation of these novel algorithms is presented, focusing on how the neural network models can be optimized to ensure a feasible hardware implementation. The performance and resource usage of the optimized neural network models are discussed in detail.

A full-function Global Common Module prototype for ATLAS Phase-II upgrade

The High Luminosity Large Hadron Collider (HL-LHC [1]), an upgrade of the LHC, is set to become operational in 2029, aiming to achieve instantaneous luminosities 5–7.5 times larger than the nominal value of the LHC. However, unlocking the full physics potential at this much higher luminosity level necessitates a tenfold increase in the data bandwidth processed by ATLAS. This poses significant challenges to the design of the Trigger and Data Acquisition systems. To address these challenges, a baseline architecture has been chosen for the ATLAS Phase-II upgrade, relying on a single-level hardware trigger known as the Level-0 Trigger. This trigger has a maximum rate of 1 MHz and a latency of 10 μs. Central to this upgrade is the inclusion of a new subsystem — the Global Trigger [2]. This component performs complex algorithms, akin to those currently used in Phase-I high-level trigger software (such as Topoclustering), on full-granularity calorimeter data. The Global Trigger is divided into three sublayers: the Multiplexer Processor (MUX) layer, the Global Event Processor (GEP) layer, and the Global to Central Trigger Processor [3] interface (gCTPi). A full-function Global Common Module (GCM) hardware prototype has been designed to fulfill the requirements of all three sublayers of the Global Trigger, featuring different firmware loads. This GCM prototype, based on the ATCA [4] front board form factor, incorporates two of the latest AMD (Xilinx) Versal Premium devices VP1802 [5]. These devices boast double the density of the Virtex UltraScale+ FPGA VU13P used in the previous design [6] and include an integrated SoC with a completely new architecture. To handle high-speed I/Os, this GCM prototype employs twenty 12-channel 25.7 Gb/s FireFly [7] optical engines. The estimated maximum power consumption of this GCM prototype is 400 W, which falls within the cooling capabilities of the ATLAS ATCA shelf. To ensure power integrity, signal integrity, and thermal performance, extensive PCB simulations and thermal simulations have been done to guide the layout design of the GCM prototype. This paper provides an in-depth overview of the design process for this full-function GCM prototype hardware, with a particular focus on technology choices and simulation results.

Run-3 Commissioning of CMS Online HLT reconstruction using GPUs

The software-based High-Level Trigger (HLT) of CMS reduces the data readout rate from 100 kHz (obtained from Level 1 trigger) to around 5 kHz. It makes use of all detector subsystems and runs a streamlined version of CMS reconstruction. Run-2 of the LHC saw the reconstruction algorithms run on a CPU farm. However, the need to have increased computational power as we approach the high luminosity phase of LHC demands the use of Graphical Processing Units (GPUs) to reign in the cost, size and power consumption of the HLT farm. Parallelization of the reconstruction algorithms, on top of the multi-threading functionality introduced in Run-2, allowed parts of the Hadronic Calorimeter (HCAL), Electromagnetic Calorimeter (ECAL) and Pixel Tracker reconstruction to be offloaded to NVIDIA GPUs. In order to ensure the reproducibility of physics results on any machine, the HLT configuration was designed to run seamlessly with and without GPUs, that is, the algorithms were automatically offloaded to a GPU when one was available and otherwise fell back to running on the CPU. This contribution will describe the development of GPU-based algorithms for the HLT and the challenges they presented, along with the comprehensive validation and commissioning activity undertaken by CMS to ensure the successful operations of the new HLT farm.

Running GPU-enabled CMSSW workflows through the production system

The CMS experiment at CERN accelerates several stages of its online reconstruction by making use of GPU resources at its High Level Trigger (HLT) farm for LHC Run 3. Additionally, during the past years, computing resources available to the experiment for performing offline reconstruction, such as Tier-1 and Tier-2 sites, have also started to integrate accelerators into their systems. In order to make efficient use of these heterogeneous platforms, it is essential to adapt both the CMS production system and the CMSSW reconstruction code to make use of GPUs. The CMSSW offline reconstruction can now partially run on GPUs, inheriting from the work done at the HLT. Parts of the production systems infrastructure have also been adapted to successfully map, schedule and run the available GPU-enabled workflows on different sites across the computing grid. This talk will describe the process of commissioning GPU-enabled CMSSW workflows through the production system and will present first results from the deployment of GPU-enabled offline reconstruction workflows.

Optimization of opportunistic utilization of the ATLAS high-level trigger farm for LHC Run 3

Optimization of opportunistic utilization of the ATLAS high-level trigger farm for LHC Run 3

MiniDAQ-3: Providing concurrent independent subdetector data-taking on CMS production DAQ resources

The data acquisition (DAQ) of the Compact Muon Solenoid (CMS) experiment at CERN, collects data for events accepted by the Level-1 Trigger from the different detector systems and assembles them in an event builder prior to making them available for further selection in the High Level Trigger, and finally storing the selected events for offline analysis. In addition to the central DAQ providing global acquisition functionality, several separate, so-called “MiniDAQ” setups allow operating independent data acquisition runs using an arbitrary subset of the CMS subdetectors. During Run 2 of the LHC, MiniDAQ setups were running their event builder and High Level Trigger applications on dedicated resources, separate from those used for the central DAQ. This cleanly separated MiniDAQ setups from the central DAQ system, but also meant limited throughput and a fixed number of possible MiniDAQ setups. In Run 3, MiniDAQ-3 setups share production resources with the new central DAQ system, allowing each setup to operate at the maximum Level-1 rate thanks to the reuse of the resources and network bandwidth. Configuration management tools had to be significantly extended to support the synchronization of the DAQ configurations needed for the various setups. We report on the new configuration management features and on the first year of operational experience with the new MiniDAQ-3 system.

Repurposing of the Run 2 CMS High Level Trigger Infrastructure as a Cloud Resource for Offline Computing

Repurposing of the Run 2 CMS High Level Trigger Infrastructure as a Cloud Resource for Offline Computing

Impact of the high-level trigger for detecting long-lived particles at LHCb

Impact of the high-level trigger for detecting long-lived particles at LHCb

Fast b-tagging at the high-level trigger of the ATLAS experiment in LHC Run 3

The ATLAS experiment relies on real-time hadronic jet reconstruction and b-tagging to record fully hadronic events containing b-jets. These algorithms require track reconstruction, which is computationally expensive and could overwhelm the high-level-trigger farm, even at the reduced event rate that passes the ATLAS first stage hardware-based trigger. In LHC Run 3, ATLAS has mitigated these computational demands by introducing a fast neural-network-based b-tagger, which acts as a low-precision filter using input from hadronic jets and tracks. It runs after a hardware trigger and before the remaining high-level-trigger reconstruction. This design relies on the negligible cost of neural-network inference as compared to track reconstruction, and the cost reduction from limiting tracking to specific regions of the detector. In the case of Standard Model HH → bb̅bb̅, a key signature relying on b-jet triggers, the filter lowers the input rate to the remaining high-level trigger by a factor of five at the small cost of reducing the overall signal efficiency by roughly 2%.

The ATLAS Readout System for LHC Runs 2 and 3

The ReadOut System (ROS) is a central part of the data acquisition (DAQ) system of the ATLAS Experiment at the CERN Large Hadron Collider (LHC). The system is responsible for receiving and buffering event data from all detector subsystems and serving these to the High Level Trigger (HLT) system via a 10 GbE network, discarding or transporting data onward once the trigger has completed its selection process. The ATLAS ROS was completely replaced during the 2013–2014 experimental shutdown in order to meet the demanding conditions expected during LHC Run 2 and Run 3 (2015–2025). The ROS consists of roughly one hundred Linux-based 2U-high rack-mounted servers equipped with PCIe I/O cards and 10 GbE interfaces. This paper documents the system requirements for LHC Runs 2 and 3 and the design choices taken to meet them. The results of performance measurements and the re-use of ROS technology for the development of data sources, test platforms for other systems, and another ATLAS DAQ system component, namely the Region of Interest Builder (RoIB), are also discussed. Finally performance results for Run 2 operations are presented before looking at the upgrade for Run 3.

Design and Commissioning of the First 32-Tbit/s Event-Builder

The LHCb experiment is a forward spectrometer, designed to study beauty and charm quarks Physics at the LHC. To exploit of the higher luminosity that will be delivered during Run3, the full experiment needed a substantial upgrade, from the detector to the Data Acquisition (DAQ) and High Level Trigger (HLT). In this paper, we will focus on the new DAQ system for the LHCb experiment; that represents a substantial paradigm shift compared to the previous one, and to similar systems used by similar experiments in the past and present times. To overcome the inefficiencies introduced by a local selection implemented directly with the readout hardware, the Run3 system is designed to perform a full software reconstruction of all the produced events. To achieve this, both the DAQ and the HLT need to process the ~ 30 MHz full event-rate. In particular, this paper will introduce the final design of the system; it will provide a focus on the hardware and software design of the Event Building (EB) and how we integrated technologies designed for the High Performance Computing (HPC) world - like InfiniBand HDR (200 Gb/s)- into the DAQ system; we will present performance measurements of the full EB system under different operational conditions; and we will provide a feedback from EB operation during the beginning of the data-taking.

Upgrade of Online Storage and Express-Reconstruction System for the Belle II Experiment

The Belle II experiment started to take data of electron-positron collisions provided by the SuperKEKB accelerator, with all subdetectors from March 2019. The collected data are selected by a software-based high-level trigger system and stored on online storage. After storing the data, the online storage sends the data to an express-reconstruction system for semi-real-time data quality monitoring and event display. The high-level trigger system is built on the ZeroMQ networking library, but the online storage and express-reconstruction system are built on the ring buffer and TCP/IP socket-based framework. To improve the stability and future maintainability, we develop the new online storage and express-reconstruction system using the ZeroMQ library. The new systems have a few additional improvements. First, the online storage supports the ROOT output file format with compression. This reduces the online file size, necessary network bandwidth for the online-to-offline file transfer, and offline computing resource usage. Second, two types of express-reconstruction systems are prepared based on the high-level trigger results; for randomly sampled data and for physics-flagged data by the high-level trigger system. The separate express-reconstruction system can process a larger number of statistics of physics-flagged events.

Commissioning of the new muon-to-central-trigger-processor interface at ATLAS

The ATLAS trigger system includes a Level-1 (L1) trigger based on custom electronics and firmware, and a high-level software trigger running on off-the-shelf hardware. The L1 trigger system uses information from the forward detectors, the calorimeters and the muon trigger detectors. Once information from all muon trigger sectors has been received, trigger candidate multiplicities are calculated by the Muon-to-Central-Trigger-Processor Interface (MUCTPI). Muon multiplicity information is sent to the Central-Trigger-Processor (CTP) and trigger objects are sent to the L1 Topological Trigger Processor (L1Topo). The CTP combines the information received from the MUCTPI with the trigger information from the forward detectors, the calorimeters and the L1Topo, and takes the L1 trigger decision. As part of the ATLAS L1 trigger system upgrade for Run-3 of the Large Hadron Collider (LHC) a new MUCTPI has been designed and commissioned. We discuss the commissioning and operation of the new MUCTPI used in ATLAS from the beginning of Run-3. In particular, we describe the integration tests which have been carried out for the commissioning and operation of the new MUCTPI.

Online Event Selection and GPU-based Waveform Compression at the High Level Trigger of the KOTO Experiment

The Data Acquisition (DAQ) system of the KOTO experiment is currently undergoing a major upgrade, following an also ongoing accelerator’s beam intensity upgrade. The throughput of the current DAQ is limited by the hardware, and different prescalings have to be applied to reduce the data rate to a manageable level. The DAQ upgrade aims to remove the hardware bottleneck, increasing the amount of data that reaches the high level trigger system. The new high level trigger will make use of parallelization over CPU and GPU threads to perform event reconstruction, selection, and compression, to eventually allow for more variate physics searches in KOTO.

CMS High Level Trigger performance comparison on CPUs and GPUs

At the start of the upcoming LHC Run-3, CMS will deploy a heterogeneous High Level Trigger (HLT) farm composed of x86 CPUs and NVIDIA GPUs. In order to guarantee that the HLT can run on machines without any GPU accelerators - for example as part of the large scale Monte Carlo production running on the grid, or when individual developers need to optimise specific triggers - the HLT reconstruction has been implemented both for NVIDIA GPUs and for traditional CPUs. This contribution will describe how the CMS software used online and offline (CMSSW) can transparently switch between the two implementations, and compare their performance on GPUs and CPUs from different architectures, vendors and generations.

Performance portability for the CMS Reconstruction with Alpaka

For CMS, Heterogeneous Computing is a powerful tool to face the computational challenges posed by the upgrades of the LHC, and will be used in production at the High Level Trigger during Run 3. In principle, to offload the computational work on non-CPU resources, while retaining their performance, different implementations of the same code are required. This would introduce code-duplication which is not sustainable in terms of maintainability and testability of the software. Performance portability libraries allow to write code once and run it on different architectures with close-to-native performance. The CMS experiment is evaluating performance portability libraries for the near term future.

Commissioning LHCb’s GPU high level trigger

From 2022 onward, the upgraded LHCb experiment will use a triggerless readout system collecting data at an event rate of 30 MHz. A software-only High Level Trigger will enable unprecedented flexibility for trigger selections. During the first stage (HLT1), a sub-set of the full offline track reconstruction for charged particles is run to select particles of interest based on single or two-track selections. This includes decoding the raw data, clustering of hits, pattern recognition, as well as track fitting, ghost track rejection with machine learning techniques and finally event selections. HLT1 will be processed entirely on O(100) GPUs. We will present the integration of the GPU HLT1 into LHCb’s data acquisition system in preparation for data taking in 2022.

Developing GPU-compliant algorithms for CMS ECAL local reconstruction during LHC Run 3 and Phase 2

The higher LHC luminosities foreseen in Run 3 (2022+) and during the Phase 2 of the LHC (2029+), and the consequently larger number of simultaneous proton-proton collisions per event, pose significant challenges for the CMS event reconstruction. In particular maintaining the strict time budget for reconstruction algorithms at the CMS software high level trigger will not be possible considering only the expected increase of processing power from conventional CPUs. Therefore, CMS is investigating the use of heterogeneous architectures, including GPU accelerators, to satisfy the needs of the Phase 2 reconstruction. The local reconstruction algorithm of the CMS electromagnetic calorimeter (ECAL) is among the first to be implemented on heterogeneous platforms for LHC Run 3. This article discusses the necessary changes to the CMS software framework that support the execution of code on accelerators. The current development status of the ECAL local reconstruction algorithm is described and first performance results of GPU enabled code are presented.