Advanced Telecommunications Computing Architecture Research Articles

Lessons learned while developing the Serenity-S1 ATCA card

The Serenity-S1 is a production-optimised Advanced Telecommunications Computing Architecture (ATCA) processing blade based on the AMD Xilinx Virtex Ultrascale+ device. It incorporates many developments from the Serenity-A and Serenity-Z prototype cards and, where possible, adopts solutions being used across CERN. Due to the shortage of components during the recent semiconductor crisis, commonly used components in the prototypes had to be replaced by new ones after qualification. In this work, we discuss various improvements to simplify manufacturing, the performance of new components, some of the more difficult aspects of procurement, the performance of production-grade Samtec 25 Gb/s optical firefly parts, and concerns regarding the rack cooling infrastructure.

The PreProcessor module for the ATLAS Tile calorimeter at the HL-LHC

The Large Hadron Collider (LHC) has played a crucial role in advancing our understanding of fundamental physics. With the discovery of the Higgs boson in 2012, the ATLAS and CMS experiments have made significant progess in studying its properties and searching for new physics beyond the Standard Model. To maintain and expand the LHC’s discovery potential, the High-Luminosity LHC (HL-LHC) project is planned as a major upgrade. As part of this upgrade, the ATLAS experiment has developed a comprehensive roadmap for upgrades, including the installation of new detector components and advancements in data acquisition and processing systems. A crucial component of these enhancements involves the complete replacement of the ATLAS central Tile hadronic calorimeter readout electronics. This upgrade aims to optimize the system’s ability to handle higher data rates and improve its resilience to radiation. Through these comprehensive improvements, the ATLAS experiment aims to meet the requirements and seize the opportunities presented by the HL-LHC era. This paper presents the design and architecture of the TilePPr (Tile PreProcessor) module, which serves as a crucial interface between the on-detector electronics and the central systems of the ATLAS experiment, including Data Acquisition, Detector Control, and Trigger systems. The TilePPr module is based on the Advanced Telecommunications Computing Architecture (ATCA) and incorporates high-speed optical links, communication interfaces, and data processing capabilities. Through a series of certification tests, the module has demonstrated its compliance with industry standards and functional requirements, confirming its suitability for seamless integration into the ATLAS experiment during the HL-LHC phase.

Hardware design and testing of the generic rear transition module for the global trigger subsystem of ATLAS Phase-II Upgrade

In the framework of the ATLAS experiment’s Phase-II Upgrade at the High-Luminosity Large Hadron Collider (HL-LHC), new and improved trigger hardware and algorithms will be implemented onto a single-level, 10 μs-latency architecture. The Global Trigger is a new subsystem which will bring event-filter capabilities by performing offline-like algorithms on full-granularity calorimeter data. The implementation of the functionality is firmware-focused and composed of several processing nodes, which are hosted on identical hardware, made up of an Advanced Telecommunications Computing Architecture (ATCA) front board, called Global Common Module (GCM), and a rear transition module (RTM), called Generic RTM (GRM). The GRM, which was developed to mitigate the risks deriving from the complex design and power management of the GCM, features an advanced Xilinx Versal Prime system-on-chip and can handle communication with the GCM and Front-End Link eXchange (FELIX) subsystem and trigger processors through 124 25.8 Gb/s transceiver links, for readout and control. Additionally, the GRM mounts a Low-Power GigaBit Transceiver (lpGBT) chip which enables emulation of the detector front-ends for integration tests. This paper presents the GRM hardware design and its testing.

Tile Computer-on-Module for the ATLAS Tile Calorimeter Phase-II upgrades

The Tile PreProcessor (TilePPr) is the core element of the Tile Calorimeter (TileCal) off-detector electronics for High-luminosity Large Hadron Collider (HL-LHC). The TilePPr comprises FPGA-based boards to operate and read out the TileCal on-detector electronics. The Tile Computer on Module (TileCoM) mezzanine is embedded within TilePPr to carry out three main functionalities. These include remote configuration of on-detector electronics and TilePPr FPGAs, interface the TilePPr with the ATLAS Trigger and Data Acquisition (TDAQ) system, and interfacing the TilePPr with the ATLAS Detector Control System (DCS) by providing monitoring data. The TileCoM is a 10-layer board with a Zynq UltraScale+ ZU2CG for processing data, interface components to integrate with TilePPr and the power supply to be connected to the Advanced Telecommunication Computing Architecture carrier. A CentOS embedded Linux is deployed on the TileCoM to implement the required functionalities for the HL-LHC. In this paper we present the hardware and firmware developments of the TileCoM system in terms of remote programming, interface with ATLAS TDAQ system and DCS system.

CMS phase-2 DAQ and timing hub prototyping results and perspectives

This paper describes recent progress on the design of the DAQ and Timing Hub, or DTH, an ATCA (Advanced Telecommunications Computing Architecture) hub board intended for the phase-2 upgrade of the CMS experiment. Prototyping was originally divided into multiple feature lines, spanning all different aspects of the DTH functionality. The second DTH prototype merges all R&D and prototyping lines into a single board, which is intended to be the production candidate. Emphasis is on the process and experience in going from the first to the second DTH prototype, which included a change of the chosen FPGA as well as the integration of a commercial networking solution.

ZynqMP-based board-management mezzanines for Serenity ATCA-blades

In the context of the CMS Phase-2 tracker back-end processing system, two mezzanines based on the Zynq Ultrascale+ Multi-Processor System-on-Chip (MPSoC) device have been developed to serve as centralized slow control and board management solution for the Serenity-family Advanced Telecommunications Computing Architecture (ATCA) blades. This paper presents the developments of the MPSoC mezzanines to execute the Intelligent Platform Management Controller (IPMC) software in the real-time capable processors of the MPSoC. In coordination with the Shelf Manager, once full-power is enabled, a CentOS-based Linux distribution is executed in the application processors of the MPSoC, on which EMPButler and the Serenity Management Shell (SMASH) are running.

A flexible and low-cost open-source IPMC mezzanine for ATCA boards based on OpenIPMC

This work presents the development of an Intelligent Platform Management Controller mezzanine in a Mini DIMM form factor for use in electronic boards compliant to the PICMG Advanced Telecommunication Computing Architecture (ATCA) standard. The module is based on an STMicroelectronics STM32H745 microcontroller running the OpenIPMC open-source software. The mezzanine has been successfully tested on a variety of ATCA boards being proposed for the upgrade of the experiments at the HL-LHC, with its design and firmware being distributed under open-source hardware license.

Hardware design of the generic rear transition module for the global trigger system of the ATLAS phase II upgrade

The High-Luminosity Large Hadron Collider is expected to start operations in the middle of 2027, to deliver more than ten times the integrated luminosity of the Large Hadron Collider Runs 1–3 combined (up to 4000 fb−1). Meeting these requirements poses significant challenges to the hardware design of the trigger and data acquisition system. Global trigger is a new subsystem in the ATLAS phase-II upgrade, which will bring event filter-like capability to the level-0 trigger system. A common hardware platform in Advanced Telecommunications Computing Architecture form factor named Global Common Module is proposed to be configured as processor nodes in the global trigger. To mitigate the risk and simplify the Global Common Module hardware design, a Generic Rear Transition Module is being developed. The Generic Rear Transition Module, which has been implemented with a Xilinx Versal Prime Field Programmable Gate Array and sufficient multi-gigabit transceivers, cannot only achieve system control and communication with the Front-End Link eXchange, but also provide additional processing or readout capacity.

OpenIPMC: A Free and Open-Source Intelligent Platform Management Controller Software

OpenIPMC is a free and open source firmware designed to operate as an Intelligent Platform Management Controller (IPMC). An IPMC is a fundamental component of electronic boards conformant to the Advanced Telecommunications Computing Architecture (ATCA) standard, currently being adopted by a number of high energy physics experiments, and is responsible for monitoring the health parameters of the board, managing its power states, and providing board control, debug and recovery functions to remote clients. OpenIPMC is based on the FreeRTOS real-time operating system and is designed to be architecture-independent, allowing it to be built for a variety of different Microcontrollers. Having a fully free and open source code is an innovative aspect for this kind of firmware, allowing full customization by the user. In this work we present the features and structure of OpenIPMC and its example implementations on Xilinx Zynq UltraScale+ (ZynqUS+), Espressif ESP32, and ST Microelectronics STM32 architectures.

Advanced high-performance processing tools for diagnostics and control in fusion devices

The ITER project demanding operating conditions, as well as other enabling experiments in relevant fusion projects, present new challenges to the diagnostics, control and instrumentation. The most critical requirements relate to the high acquisition rates (up to some gigasample per second), high physical event rate (up to several megaevents per second), need to handle enormous quantities of data in long experimental pulses (up to 30 minutes) and highly complex control and diagnostic algorithms. Radiation hardness of all equipment involved must also be taken into account for high magnetic fields as well as neutron irradiation in the fusion devices port cells.The implementation of state-of-the-art designs were tested both with in-house built prototypes and complete systems installed in the major international fusion experiments. The main technologies used include Advanced Telecommunications Computing Architecture (ATCA), high speed electronic devices for signal digitization, complex real-time algorithms for physics analysis, control and data compression implemented in high-performance computing using field-programmable gate arrays and multicore processor architectures.The control and data acquisition group at Instituto de Plasmas e Fusão Nuclear (Instituto Superior Técnico - Lisboa) has conducted several projects and studies for the development of reliable high-performance processing tools and instrumentation for the diagnostics and control of fusion devices capable of complying with the most demanding requirements.

An implementation of module management controller for MicroTCA data processing system

On the basis of full compatibility with advanced telecommunications computing architecture (ATCA) standard, micro telecommunications computing architecture (MicroTCA) standard has a evident advantage over ATCA in terms of space, cost and scalability. The MicroTCA standard has become a popular platform in high energy physics. For the advanced mezzanine card (AMC) installed in the MicroTCA chassis, module management controller (MMC) is a prerequisite to fulfill its function. Although there are many MMC solutions, most of their firmware work in polling mode and rarely use the real-time operating system (RTOS). In order to overcome the lack of real-time performance in polling MMC solution, the novel firmware is developed based on FreeRTOS, which is an open source real-time operating system. This firmware implements the basic functions of MMC, such as hot swap, power management, firmware upgrade, and adds the feature of serial command debugging. To verify the difference in real-time performance between the FreeRTOS and the polling solutions, the interrupt reaction time test is carried out on both solutions. The test shows that the average interrupt reaction time of the FreeRTOS solution is about 300 ms shorter than that of the polling method. This result proves that the FreeRTOS-based solution can significantly improve the real-time performance of transaction processing.

Firmware developments on the TileCoM for the Phase-II Upgrade of the ATLAS Tile Calorimeter

The Tile Computer on Module (TileCoM) mezzanine board is one of the auxiliary boards of the Tile PreProcessor (TilePPr) for the Phase-II Upgrade of the readout electronics of the ATLAS Tile Calorimeter (TileCal). This board will be responsible for system monitoring and configuration by interfacing the Trigger Data Acquisition (TDAQ) system and the TilePPr. Features include configuration and monitoring of the Advanced Telecommunications Computing Architecture (ATCA) carrier and Compact Processing Module (CPM) onboard sensors through I2C and Gigabit Ethernet. This contribution presents firmware developments on an embedded Linux for the ZYNQ System-on-Chip (SoC) targeting an Avnet Ultra96-V2 ZYNQ UltraScale+ MPSoC evaluation board. This test bench will serve as a basis for the development of the main functionalities of the TileCoM mezzanine board to interface the TilePPr with the Detector Control System (DCS) and the TDAQ-I system of the Tile Calorimeter.

Development of the ATLAS Liquid Argon (LAr) Calorimeter readout electronics for the HL-LHC

To meet new trigger and data acquisition (TDAQ) buffering requirements and withstand the high expected radiation doses at the High-Luminosity LHC, the ATLAS Liquid Argon Calorimeter readout electronics will be upgraded. The triangular calorimeter signals are amplified and shaped by analog electronics over a dynamic range of 16 bits, with low noise and excellent linearity. Developments of low-power preamplifiers and shapers in the 130 nm CMOS technology are ongoing to meet these requirements. In order to digitize the analog signals on two gains after shaping, a radiation-hard, low-power 40 MHz 14-bit Analog-to-Digital Converter (ADC) is developed using a pipeline + Successive Approximation Register (SAR) architecture in the 65 nm CMOS technology. Characterization of the prototypes of the front-end components show good promise to fulfill all the requirements. The signals will be sent at 40 MHz to the off detector electronics, where Field-Programmable Gate Arrays (FPGAs) connected through high-speed links will perform energy and time reconstruction through the application of corrections and digital filtering. Reduced data will be sent with low latency to the first level trigger, while the full data will be buffered until the reception of trigger accept signals. The data-processing, control and timing functions will be realized by dedicated boards connected through Advanced Telecommunications Computing Architecture (ATCA) crates.

MTCA control and data acquisition platform for Plasma Diagnostics

The Advanced Telecommunications Computing Architecture (ATCA) instrumentation platform for fusion control and diagnostics developed at IST, is being extended with a Micro Telecommunications Computing Architecture (MTCA) control and data acquisition platform, still able to maintain performance characteristics in speed, channel density and availability of ATCA, at a cost suitable for smaller systems. Software and firmware have been ported to MTCA using mostly in-house developed hardware, preserving many of ATCA's benefits in performance in a smaller form-factor. The first instance of the proposed platform comprises a MTCA 1U shelf with 6 Advanced Mezzanine Card (AMC) slots, MTCA Carrier Hub (MCH) module, AMC to Field-programmable Gate Array (FPGA) Mezzanine Card (FMC) carrier modules, 500 MSPS to 2 GSPS ADC FMC modules, timing and synchronization input/output (IO) module, or other Commercial Off-The-Shelf (COTS) modules. Data is handled by the MCH, featuring PCIe Gen 3 switching to a host computer, as well as performing clock distribution and Intelligent Platform Management Interface (IPMI) based hardware management for high availability. This compact, flexible and autonomous or scalable platform may either integrate a large-experiment system infrastructure, as well as being configured as a lower-cost, self-contained platform for smaller experiments or development and testbench purposes. The focus of this paper is the development of the MCH MTCA.0 module, which is key to introduce the platform architecture.

PCIe Hot-Plug Support Standardization Challenges in ATCA

Throughout the last decade, the Advanced Telecommunications Computing Architecture (ATCA) solidified its position as one of the main switched-based crate standards for advanced Physics instrumentation, offering not only highly performant characteristics in data throughput, channel density or power supply/dissipation capabilities, but also special features for high availability (HA), required for latest and upcoming large-scale endeavours, as is the case of ITER. Hot Swap is one of the main HA features in ATCA, allowing for Boards to be replaced in a crate (Shelf), without powering off the whole system. Platforms using the Peripheral Component Interconnect Express (PCIe) protocol on the Fabric Interface must be complemented, at the software level, with the PCIe Hot Plug native feature, currently not specified for the ATCA form-factor. From a customised Hot Plug support implementation for ATCA Node Boards, the paper presents an implementation extension for Hub Boards, allowing Hot Plug of PCIe switching devices, without causing bus enumeration problems. This paper further addresses the main issues concerning an eventual standardization of PCIe Hot Plug support in ATCA, such as the implementability of Hot Plug Elements and the generation and management of Hot Plug Events, aiming to stimulate the discussion within the PICMG community towards a long overdue standardized solution for Hot Plug in ATCA.

Design and Simulation of ISTTOK Real-Time Magnetic Multiple-Input Multiple-Output Control

Control of the plasma current and the position of its centroid and its shape is vital for obtaining advanced performance regimes in tokamak operation. For that aim, the use of poloidal magnetic coils for feedback control is essential. In this paper, a multiple-input multiple-output (MIMO) optimal control algorithm for the vertical and horizontal control position of the plasma current centroid at IST tokamak (ISTTOK) is presented. Using the ISTTOK magnetic system, the real-time control is made possible by the execution of generic application modules (GAMs), developed to be executed on the Multithreaded Application Real-Time executor ISTTOK controller. The design of the proposed MIMO controller is based on a state-space model of the ISTTOK magnetic response that has been identified from experimental data. Both the control design and the identification procedure rely on the reconstruction of the plasma centroid position computed by the magnetic probes (Mirnov coils) installed at ISTTOK. The centroid position is reconstructed in the Position GAM by numerical integration of the magnetic probes signals in the recently upgraded hardware based on the Advanced Telecommunications Computing Architecture. The effectiveness of the proposed control approach is presented comparing simulation results with experimental ones obtained using single-input single-output PID controllers. Moreover, due to the modularity of the ISTTOK control system, the proposed real-time reconstruction algorithm has been easily deployed on the plant, and the same approach will be adopted to test the MIMO controller in the future.

New FPGA based hardware implementation for JET gamma-ray camera upgrade

The Gamma-Ray Camera Upgrade (GCU) project aims at installing a new set of 19 scintillators with multi-pixel photon counter (MPPC) embedded, capable to meet the high fluxes expected during deuterium-tritium plasmas while improving the diagnostic spectroscopic capabilities. GCU will benefit from the Advanced Telecommunications Computing Architecture (ATCA)-based Data Acquisition System (DAQ), successfully installed and commissioned during the JET-EP2 enhancement. However, to cope with the new GCU detector signals, the DAQ Field Programmable Gate Array (FPGA) codes need to be rebuilt. This work presents the FPGA code upgrade for Gamma Camera (GC) DAQ, capable to sustain the expected fast response of new detectors, while exploiting the full capabilities of the DAQ. Dedicated codes were designed, capable to acquire the new signals at full rate, and simultaneously processing them in real-time through suitable algorithms, fitted to the new signals shape. First results of real-time processing codes applied to data from detector prototypes are presented.

Control and data acquisition software upgrade for JET gamma-ray diagnostics

The Joint European Torus (JET), the largest magnetic confinement plasma physics experiment in operation, has a large amount of key diagnostics for physics exploration and machine operation, which include several Gamma-Ray Diagnostics. The Gamma-Ray Spectrometer (GRS), Gamma Camera (GC) and Gamma-Ray Spectrometer Upgrade (GSU) diagnostics have similar Control and Data Acquisition Systems (CDAQ) based on the Advanced Telecommunication Computing Architecture standard, featuring Field Programmable Gate Arrays for data processing and management.During past JET-EP2 enhancements, the GRS and GC diagnostics were successfully installed and commissioned. However, the installed CDAQ software that interfaces these diagnostics to JET Control and Data Acquisition System is different, requiring higher maintenance costs.Benefiting from the Gamma Camera Upgrade (GCU) and new GSU installation and commissioning, the upgrading of the software and controller hardware used in the GRS and GC was evaluated, aiming at software standardization between all three diagnostics for easier maintenance.This paper describes the software standardization process between the diagnostics towards the usage of the same CDAQ software as well as the same Operating System (OS) for the controllers, which allows the operator to minimize the maintenance time, avoiding the need for system specific expertise. The rationale behind the choice of MARTe framework as CDAQ software and Scientific Linux as OS will also be presented.

PCIe hot-plug event handling tasks using PICMG standard interrupt mechanism for ATCA based instrumentation

The Advanced Telecommunications Computing Architecture (AdvancedTCA®) specification implements important key features such as high reliability, high availability, redundancy and serviceability for control and data acquisition instrumentation fault condition, hardware malfunction, firmware updates and hardware reconfiguration. Red Hat Enterprise Linux and corresponding kernels already have built-in mechanisms and embedded software for Peripheral Component Interconnect Express (PCIe) hot-plug support that allows automatic removal of PCIe device nodes and associated device files from the system providing a fast replacement strategy for damaged cards without requiring an entire system shutdown. This paper describes handling of PCIe hot-plug events at a software middle level using the PCI Industrial Computer Manufacturers Group (PICMG) standard interrupt mechanism. The handling tasks can be accomplished by AdvancedTCA® cards chipsets with support for PCIe hot-plug features, Linux hot-plug embedded controller and Red Hat built-in device manager module. The goal is to implement a fast hardware replacement solution without system shutdown providing high availability capabilities to AdvancedTCA® control and data acquisition instrumentation specially directed for large fusion experiments such as International Thermonuclear Experiment Reactor (ITER).

The Belle II Pixel Detector Data Acquisition and Background Suppression System

The Belle II experiment at the future SuperKEKB collider in Tsukuba, Japan, features a design luminosity of 8 · 1035 cm−2s−1, which is a factor of 40 larger than that of its predecessor Belle. The pixel detector (PXD) with about 8 million pixels is based on the DEPFET technology and will improve the vertex resolution in beam direction by a factor of 2. With an estimated trigger rate of 30 kHz, the PXD is expected to generate a data rate of 20 GBytes/s, which is about 10 times larger than the amount of data generated by all other Belle II subdetectors. Due to the large beam-related background, the PXD requires a data acquisition system with high-bandwidth data links and realtime background reduction by a factor of 30. To achieve this, the Belle II pixel DAQ uses an FPGA-based computing platform with high speed serial links implemented in the ATCA (Advanced Telecommunications Computing Architecture) standard. The architecture and performance of the data acquisition system and data reduction of the PXD will be presented. In April 2016 and February 2017 a prototype PXD-DAQ system operated in a test beam campaign delivered data with the whole readout chain under realistic high rate conditions. Final results from the beam test will be presented.