ATLAS Data Acquisition System Research Articles

The ATLAS Readout System for LHC Runs 2 and 3

The ReadOut System (ROS) is a central part of the data acquisition (DAQ) system of the ATLAS Experiment at the CERN Large Hadron Collider (LHC). The system is responsible for receiving and buffering event data from all detector subsystems and serving these to the High Level Trigger (HLT) system via a 10 GbE network, discarding or transporting data onward once the trigger has completed its selection process. The ATLAS ROS was completely replaced during the 2013–2014 experimental shutdown in order to meet the demanding conditions expected during LHC Run 2 and Run 3 (2015–2025). The ROS consists of roughly one hundred Linux-based 2U-high rack-mounted servers equipped with PCIe I/O cards and 10 GbE interfaces. This paper documents the system requirements for LHC Runs 2 and 3 and the design choices taken to meet them. The results of performance measurements and the re-use of ROS technology for the development of data sources, test platforms for other systems, and another ATLAS DAQ system component, namely the Region of Interest Builder (RoIB), are also discussed. Finally performance results for Run 2 operations are presented before looking at the upgrade for Run 3.

Tile Computer-on-Module for the ATLAS Tile Calorimeter Phase-II upgrades

The Tile PreProcessor (TilePPr) is the core element of the Tile Calorimeter (TileCal) off-detector electronics for High-luminosity Large Hadron Collider (HL-LHC). The TilePPr comprises FPGA-based boards to operate and read out the TileCal on-detector electronics. The Tile Computer on Module (TileCoM) mezzanine is embedded within TilePPr to carry out three main functionalities. These include remote configuration of on-detector electronics and TilePPr FPGAs, interface the TilePPr with the ATLAS Trigger and Data Acquisition (TDAQ) system, and interfacing the TilePPr with the ATLAS Detector Control System (DCS) by providing monitoring data. The TileCoM is a 10-layer board with a Zynq UltraScale+ ZU2CG for processing data, interface components to integrate with TilePPr and the power supply to be connected to the Advanced Telecommunication Computing Architecture carrier. A CentOS embedded Linux is deployed on the TileCoM to implement the required functionalities for the HL-LHC. In this paper we present the hardware and firmware developments of the TileCoM system in terms of remote programming, interface with ATLAS TDAQ system and DCS system.

The Controls and Configuration Software of the ATLAS Data Acquisition System: evolution towards LHC Run 3

The ATLAS experiment at the Large Hadron Collider (LHC) operated very successfully in the years 2008 to 2018, in two periods identified as Run 1 and Run 2. ATLAS achieved an overall data-taking efficiency of 94%, largely constrained by the irreducible dead-time introduced to accommodate the limitations of the detector read-out electronics. Out of the 6% dead-time only about 15% could be attributed to the central trigger and DAQ system, and out of these, a negligible fraction was due to the Control and Configuration subsystem. Despite these achievements, and in order to improve even more the already excellent efficiency of the whole DAQ system in the coming Run 3, a new campaign of software updates was launched for the second long LHC shutdown (LS2). This paper presents, using a few selected examples, how the work was approached and which new technologies were introduced into the ATLAS Control and Configuration software. Despite these being specific to this system, many solutions can be considered and adapted to different distributed DAQ systems.

Data-acquisition system developments for ATLAS pixel QA and QC test toward High-Luminosity LHC

In preparation for the High-luminosity LHC upgrade, the whole ATLAS inner tracker will be replaced by a new silicon detector tracker. The innermost region will be covered by silicon pixel detectors as a high density of produced particles is expected. To operate in such an environment, high-resolution sensors and a high-speed readout system are required. At the moment, the front-end readout chip prototype RD53A and a data acquisition system (the YARR system) based on a commercial FPGA board and dedicated software for quality assurance and quality control test have been developed. Due to the high density of sensor channels, the output speed from RD53A is at maximum 1.28 Gbps per line, sixteen times faster than the readout front-end chip currently used in the ATLAS pixel system. The data acquisition system needs to establish communication with 1.28 Gbps speed during the quality control tests, to validate the data acquisition path, and to optimize the procedure for data taking at high speed. From the quality control perspective this optimization allows to test large numbers of modules simultaneously exploiting the data acquisition speed reducing the time needed for the tests. Another challenging point is the novel concept of readout structure planned for the operation after installation. In High-luminosity LHC, large parts of the ATLAS data acquisition system infrastructure are going to be shared among all sub-detectors, using FELIX systems, while current ATLAS data acquisition systems are dedicated for each sub-detector. This means that all processes done in the present data acquisition system hardware need to be overhauled into software running on the new data acquisition system. To minimize the differences between the data acquisition system for operation and quality control test, we introduced the prototype FELIX system into the data acquisition path of the YARR system. In this proceeding, an established data acquisition structure for quality assurance and quality control test of the new pixel detector is introduced, and results from basic quality control tests of the pixel detector with the new readout chip are presented.

Electronics performance of the ATLAS New Small Wheel Micromegas wedges at CERN

A series of upgrades are planned for the LHC accelerator to increase its instantaneous luminosity to 7.5 × 1034 cm−2s−1. The luminosity increase drastically impacts the ATLAS trigger and readout data rates. The present ATLAS small wheel muon detector will be replaced with a New Small Wheel (NSW) detector which is expected to be installed in the ATLAS underground cavern by the end of the Long Shutdown 2 of the LHC . With the final micromegas (MM) quadruplets (modules) already produced the activities concerning the integration of the modules into the final, fully equipped MM wedges, that will then be installed on the wheel structure on surface, are currently in full swing at CERN. One crucial part of the integration procedure concerns the installation, testing and validation of the on-detector electronics and readout chain for a very large system with more than 2.1 millions electronic channels in total. These include ∼4 thousands MM Front-End Boards (MMFE8), custom printed circuit boards each one housing eight 64-channel VMM Application Specific Integrated Circuits (ASICs) that interface with the ATLAS Trigger and Data Acquisition (TDAQ) system through ∼1 thousands data-driver cards (ADDC & L1DDC, respectively). The readout chain is based on optical link technology (GigaBit Transceiver links) connecting the back-end to the front-end electronics via the Front-End LInk eXchange (FELIX), a newly developed system that will serve as the next generation readout driver for ATLAS. Experience and performance results from the first large-scale electronics integration tests performed at CERN on final MM wedges, including system validation with cosmic-rays, are presented.

LHC-ATLAS Phase-1 upgrade: firmware validation for real time digital processing for new trigger readout system of the Liquid Argon calorimeter

The third run (Run 3) of the ATLAS experiment at CERN's Large Hadron Collider will begin in 2021 in a high luminosity and high energy environment. In this circumstance, an effective trigger system is required. ATLAS uses a two-stage trigger system. The first stage trigger system (Level 1 trigger) is based on hardware, and the second stage is a software-based high-level trigger on a PC farm. With this system, an overall trigger rate is required to be suppressed to 1 kHz or less from a bunch-crossing rate of 40 MHz. In the liquid argon calorimeter level 1 trigger system, new trigger readouts with a factor of ten higher granularity compared to the Run 2 trigger readout, are introduced to suppress backgrounds. By measuring shower shape with the higher granularity, the electron and photon trigger rate can be suppressed sufficiently. To process all data from the supercells, new readout boards are prepared. One of the boards, dubbed LATOME, converts ADC data to transverse energy with fixed latency and sends three types of energies to the Level 1 trigger system. The trigger system uses computed energies with LATOME firmware, and therefore the development and validation of the LATOME firmware is important for the ATLAS data acquisition system. One of the LATOME firmware module, User Code, calculates the transverse energy for all supercells. To ensure all User Code blocks are functional, firmware validation is required. Simulation based firmware verifications and validations with the LATOME are presented with the firmware design.

ATLAS High Level Trigger within the multi-threaded software framework AthenaMT

Athena is the software framework used in the ATLAS experiment throughout the data processing path, from the software trigger system through offline event reconstruction to physics analysis. The shift from high-power single-core CPUs to multi-core systems in the computing market means that the throughput capabilities of the framework have become limited by the available memory per process. For Run 2 of the Large Hadron Collider (LHC), ATLAS has exploited a multi-process forking approach with the copy-on-write mechanism to reduce the memory use. To better match the increasing CPU core count and, therefore, the decreasing available memory per core, a multi-threaded framework, AthenaMT, has been designed and is now being implemented. The ATLAS High Level Trigger (HLT) system has been remodelled to fit the new framework and to rely on common solutions between online and offline software to a greater extent than in Run 2. We present the implementation of the new HLT system within the AthenaMT framework, which is going to be used in ATLAS data-taking during Run 3 (2021 onwards) of the LHC. We also report on interfacing the new framework to the current ATLAS Trigger and Data Acquisition (TDAQ) system, which aims to bring increased flexibility whilst needing minimal modifications to the current system. In addition, we show some details of architectural choices which were made to run the HLT selection inside the ATLAS online dataflow, such as the handling of the event loop, returning of the trigger decision and handling of errors.

The Controls and Configuration Software of the Atlas Data Acquisition System for LHC Run 2

The ATLAS experiment at the Large Hadron Collider (LHC) operated very successfully in the years 2008 to 2013, a period identified as Run 1. ATLAS achieved an overall data-taking efficiency of 94%, largely constrained by the irreducible dead-time introduced to accommodate the limitations of the detector read-out electronics. Out of the 6% dead-time only about 15% could be attributed to the central trigger and DAQ system, and out of these, a negligible fraction was due to the Control and Configuration sub-system. Despite these achievements, and in order to improve the efficiency of the whole DAQ system in Run 2 (2015-2018), the first long LHC shutdown (2013-2014) was used to carry out a complete revision of the control and configuration software. The goals were three-fold: properly accommodate additional requirements that could not be seamlessly included during steady operation of the system; re-factor software that had been repeatedly modified to include new features, thus becoming less maintainable; and seize the opportunity of modernizing software written even before Run 1, thus profiting from the rapid evolution in IT technologies. This upgrade was carried out retaining the important constraint of minimally impacting the mode of operation of the system and public APIs, in order to maximize the acceptance of the changes by the large user community. This paper presents, using a few selected examples, how the work was approached and which new technologies were introduced into the ATLAS DAQ system, and how they were performing in course of Run 2. Despite these being specific to this system, many solutions can be considered and adapted to different distributed DAQ systems.

Designing a New Infrastructure for ATLAS Online Web Services

Within the ATLAS detector, the Trigger and Data Acquisition system is responsible for the online processing of data streamed from the detector during collisions at the Large Hadron Collider (LHC) at CERN. The online farm is composed of ~4000 servers processing the data read out from ~100 million detector channels through multiple trigger levels. The capability to monitor the ongoing data taking and all the involved applications is essential to debug and intervene promptly to ensure efficient data taking. The base of the current web service architecture was designed a few years ago, at the beginning of the ATLAS operation (Run 1). It was intended to serve primarily static content from a Network-attached Storage, and privileging strict security, using separate web servers for internal (ATLAS Technical and Control Network - ATCN) and external (CERN General Purpose Network and public internet) access. During these years, it has become necessary to add to the static content an increasing number of dynamic web-based User Interfaces, as they provided new functionalities and replaced legacy desktop UIs. These are typically served by applications on VMs inside ATCN and made accessible externally via chained reverse HTTP proxies. As the trend towards Web UIs continues, the current design has shown its limits, and its increasing complexity became an issue for maintenance and growth. It is, therefore, necessary to review the overall web services architecture for ATLAS, taking into account the current and future needs of the upcoming LHC Run 3. In this paper, we present our investigation and roadmap to re-design the web services system to better operate and monitor the ATLAS detector, while maintaining the security of critical services, such as Detector Control System, and maintaining the separation of remote monitoring and on-site control according to ATLAS policies.

The ATLAS Access Manager Policy Browser: state-of-the-art web technologies for a rich and interactive data visualization experience

The ATLAS experiment is operated daily by many users and experts working concurrently on several aspects of the detector. The safe and optimal access to the various software and hardware resources of the experiment is guaranteed by a role-based access control system (RBAC) provided by the ATLAS Trigger and Data Acquisition (TDAQ) system. The roles are defined by an inheritance hierarchy. Depending on duties, every ATLAS user has a well-defined set of access privileges (rules) corresponding to a specific set of assigned roles. In total, there are several hundred roles and several thousand users. Over the years, the system grew up in terms of users and roles, motivating the deployment of a visualization tool named “Policy Browser”. Currently, it is the primary tool for role administrators to manage all the aspects of the Access Management via a rich web-based interface. This paper presents the requirements, design and implementation of the “Policy Browser”. The tool is able to aggregate and correlate all the information provided by the RBAC system and offers a visual representation of the interrelations occurring among roles, users, hosts and rules. Additionally, the “Policy Browser” implements a powerful and flexible query mechanism facilitating the browsing of all the authorizations granted by the system. As an example of the available visual representations, the “Policy Browser” is capable of dynamically generating graphs to quickly display the role giving a user some defined privileges. A graph explorer is also provided in order to browse the role&s inheritance hierarchy. The “Policy Browser” is implemented using robust JavaScript frameworks: AngularJS, Bootstrap, D3.js for the front-end, and Django a python framework for the back-end. The use cases and the results based on an informal evaluation provided by the roles administrators are also presented.

Design and Evaluation of the LAr Trigger Digitizer Board in the ATLAS Phase-I Upgrade

The Phase-I upgrade of the trigger readout electronics for the ATLAS liquid argon (LAr) calorimeters will be installed during the second long shutdown of the Large Hadron Collider (LHC) in 2019-2020, which will enable enhanced instantaneous luminosities during LHC Run 3 from 2021 through 2023. In this trigger upgrade, the so-called supercells are introduced to provide higher granularity, higher resolution, and longitudinal shower shape information from the LAr calorimeters to the Level-1 trigger processors. A new LAr trigger digitizer board (LTDB) will process and digitize up to 320 channels of supercell signals and transmit them via 40 fiber optical links to the back-end where the data are further processed and transmitted to the trigger processors. Five pairs of bidirectional GigaBit Transceiver links are used for slow control and monitoring between LTDB and the front-end link eXchange (FELIX) in the ATLAS trigger and data acquisition (TDAQ) system. The LTDB also outputs 64 summed analog legacy signals to the current tower builder board via a new baseplane. A test system for the production of 124 LTDBs has been developed, supporting configuration and calibration of all application-specified integrated circuits (ASICs) on the LTDB, as well as evaluation of full LTDB functionality.

ATLAS TDAQ System Administration: Master of Puppets

Within the ATLAS detector, the Trigger and Data Acquisition system is responsible for the online processing of data streamed from the detector during collisions at the Large Hadron Collider at CERN. The online farm is comprised of ∼4000 servers processing the data read out from ∼100 million detector channels through multiple trigger levels. The configurtion of these servers is not an easy task, especially since the detector itself is made up of multiple different sub-detectors, each with their own particular requirements. The previous method of configuring these servers, using Quattor and a hierarchical scripts system was cumbersome and restrictive. A better, unified system was therefore required to simplify the tasks of the TDAQ Systems Administrators, for both the local and net-booted systems, and to be able to fulfil the requirements of TDAQ, Detector Control Systems and the sub-detectors groups. Various configuration management systems were evaluated, though in the end, Puppet was chosen as the application of choice and was the first such implementation at CERN.

A web-based solution to visualize operational monitoring data in the Trigger and Data Acquisition system of the ATLAS experiment at the LHC

The Trigger and Data Acquisition (TDAQ) system of the ATLAS detector at the Large Hadron Collider at CERN is composed of a large number of distributed hardware and software components (about 3000 computers and more than 25000 applications) which, in a coordinated manner, provide the data-taking functionality of the overall system. During data taking runs, a huge flow of operational data is produced in order to constantly monitor the system and allow proper detection of anomalies or misbehaviours. In the ATLAS trigger and data acquisition system, operational data are archived and made available to applications by the P-BEAST (Persistent Back-End for the Atlas Information System of TDAQ) service, implementing a custom time-series database. The possibility to efficiently visualize both realtime and historical operational data is a great asset facilitating both online identification of problems and post-mortem analysis. This paper will present a web-based solution developed to achieve such a goal: the solution leverages the flexibility of the P-BEAST archiver to retrieve data, and exploits the versatility of the Grafana dashboard builder to offer a very rich user experience. Additionally, particular attention will be given to the way some technical challenges (like the efficient visualization of a huge amount of data and the integration of the P-BEAST data source in Grafana) have been faced and solved.

The Resource Manager the ATLAS Trigger and Data Acquisition System

The Resource Manager is one of the core components of the Data Acquisition system of the ATLAS experiment at the LHC. The Resource Manager marshals the right for applications to access resources which may exist in multiple but limited copies, in order to avoid conflicts due to program faults or operator errors. The access to resources is managed in a manner similar to what a lock manager would do in other software systems. All the available resources and their association to software processes are described in the Data Acquisition configuration database. The Resource Manager is queried about the availability of resources every time an application needs to be started. The Resource Manager’s design is based on a client-server model, hence it consists of two components: the Resource Manager “server” application and the “client” shared library. The Resource Manager server implements all the needed functionalities, while the Resource Manager client library provides remote access to the “server” (i.e., to allocate and free resources, to query about the status of resources). During the LHC’s Long Shutdown period, the Resource Manager’s requirements have been reviewed at the light of the experience gained during the LHC’s Run 1. As a consequence, the Resource Manager has undergone a full re-design and re-implementation cycle with the result of a reduction of the code base by 40% with respect to the previous implementation. This contribution will focus on the way the design and the implementation of the Resource Manager could leverage the new features available in the C++11 standard, and how the introduction of external libraries (like Boost multi-container) led to a more maintainable system. Additionally, particular attention will be given to the technical solutions adopted to ensure the Resource Manager could effort the typical requests rates of the Data Acquisition system, which is about 30000 requests in a time window of few seconds coming from more than 1000 clients.

Tests with beam setup of the TileCal phase-II upgrade electronics

The LHC has planned a series of upgrades culminating in the High Luminosity LHC which will have an average luminosity 5-7 times larger than the nominal Run-2 value. The ATLAS Tile calorimeter plans to introduce a new readout architecture by completely replacing the back-end and front-end electronics for the High Luminosity LHC. The photomultiplier signals will be fully digitized and transferred for every bunch crossing to the off-detector Tile PreProcessor. The Tile PreProcessor will further provide preprocessed digital data to the first level of trigger with improved spatial granularity and energy resolution in contrast to the current analog trigger signals. A single super-drawer module commissioned with the phase-II upgrade electronics is to be inserted into the real detector to evaluate and qualify the new readout and trigger concepts in the overall ATLAS data acquisition system. This new super-drawer, so-called hybrid Demonstrator, must provide analog trigger signals for backward compatibility with the current system. This Demonstrator drawer has been inserted into a Tile calorimeter module prototype to evaluate the performance in the lab. In parallel, one more module has been instrumented with two other front-end electronics options based on custom ASICs (QIE and FATALIC) which are under evaluation. These two modules together with three other modules composed of the current system electronics were exposed to different particles and energies in three test-beam campaigns during 2015 and 2016.

Optimizing the data-collection time of a large-scale data-acquisition system through a simulation framework

The ATLAS detector at CERN records particle collision “events” delivered by the Large Hadron Collider. Its data-acquisition system identifies, selects, and stores interesting events in near real-time, with an aggregate throughput of several 10 GB/s. It is a distributed software system executed on a farm of roughly 2000 commodity worker nodes communicating via TCP/IP on an Ethernet network. Event data fragments are received from the many detector readout channels and are buffered, collected together, analyzed and either stored permanently or discarded. This system, and data-acquisition systems in general, are sensitive to the latency of the data transfer from the readout buffers to the worker nodes. Challenges affecting this transfer include the many-to-one communication pattern and the inherently bursty nature of the traffic. The main performance issues brought about by this workload are addressed in this paper, focusing in particular on the so-called TCP incast pathology. Since performing systematic studies of these issues is often impeded by operational constraints related to the mission-critical nature of these systems, we developed a simulation model of the ATLAS data-acquisition system. The resulting simulation tool is based on the well-established, widely-used OMNeT++ framework. This tool was successfully validated by comparing the obtained simulation results with existing measurements of the system’s behavior. Furthermore, the simulation tool enables the study of the theoretical behavior of the system in numerous what-if scenarios and with modifications that are not immediately applicable to the real system. In this paper, we take advantage of this to analyze the behavior of the system using different traffic shaping and scheduling policies, and with network hardware modifications. This analysis leads to conclusions that could be used to devise future system enhancements.

The ATLAS Data Acquisition and High Level Trigger system

This paper describes the data acquisition and high level trigger system of the ATLAS experiment at the Large Hadron Collider at CERN, as deployed during Run 1. Data flow as well as control, configuration and monitoring aspects are addressed. An overview of the functionality of the system and of its performance is presented and design choices are discussed.

The ATLAS Data Acquisition System: from Run 1 to Run 2

The experience gained during the first period of very successful data taking of the ATLAS experiment (Run 1) has inspired a number of ideas for improvement of the Data Acquisition (DAQ) system that are being put in place during the so-called Long Shutdown 1 of the Large Hadron Collider (LHC), in 2013/14. We have updated the data-flow architecture, rewritten an important fraction of the software and replaced hardware, profiting from state of the art technologies. This paper summarizes the main changes that have been applied to the ATLAS DAQ system and highlights the expected performance and functional improvements that will be available for the LHC Run 2. Particular emphasis will be put on explaining the reasons for our architectural and technical choices, as well as on the simulation and testing approach used to validate this system.

ATLAS TDAQ System Administration: evolution and re-design

The ATLAS Trigger and Data Acquisition system is responsible for the online processing of live data, streaming from the ATLAS experiment at the Large Hadron Collider at CERN. The online farm is composed of ∼3000 servers, processing the data read out from ∼100 million detector channels through multiple trigger levels. During the two years of the first Long Shutdown there has been a tremendous amount of work done by the ATLAS Trigger and Data Acquisition System Administrators, implementing numerous new software applications, upgrading the OS and the hardware, changing some design philosophies and exploiting the High- Level Trigger farm with different purposes. The OS version has been upgraded to SLC6; for the largest part of the farm, which is composed of net booted nodes, this required a completely new design of the net booting system. In parallel, the migration to Puppet of the Configuration Management systems has been completed for both net booted and local booted hosts; the Post-Boot Scripts system and Quattor have been consequently dismissed. Virtual Machine usage has been investigated and tested and many of the core servers are now running on Virtual Machines. Virtualisation has also been used to adapt the High-Level Trigger farm as a batch system, which has been used for running Monte Carlo production jobs that are mostly CPU and not I/O bound. Finally, monitoring the health and the status of ∼3000 machines in the experimental area is obviously of the utmost importance, so the obsolete Nagios v2 has been replaced with Icinga, complemented by Ganglia as a performance data provider. This paper serves for reporting of the actions taken by the Systems Administrators in order to improve and produce a system capable of performing for the next three years of ATLAS data taking.

Development and test of the DAQ system for a Micromegas prototype to be installed in the ATLAS experiment

A Micromegas (MM) quadruplet prototype with an active area of 0.5 m2 that adopts the general design foreseen for the upgrade of the innermost forward muon tracking systems (Small Wheels) of the ATLAS detector in 2018-2019, has been built at CERN and is going to be tested in the ATLAS cavern environment during the LHC RUN-II period 2015-2017.The integration of this prototype detector into the ATLAS data acquisition system using custom ATCA equipment is presented. An ATLAS compatible Read Out Driver (ROD) based on the Scalable Readout System (SRS), the Scalable Readout Unit (SRU), will be used in order to transmit the data after generating valid event fragments to the high-level Read Out System (ROS). The SRU will be synchronized with the LHC bunch crossing clock (40.08 MHz) and will receive the Level-1 trigger signals from the Central Trigger Processor (CTP) through the TTCrx receiver ASIC. The configuration of the system will be driven directly from the ATLAS Run Control System. By using the ATLAS TDAQ Software, a dedicated Micromegas segment has been implemented, in order to include the detector inside the main ATLAS DAQ partition. A full set of tests, on the hardware and software aspects, is presented.