Event Building Research Articles

The 40 MHz trigger-less DAQ for the LHCb Upgrade

The LHCb experiment will undergo a major upgrade during the second long shutdown (2018–2019), aiming to let LHCb collect an order of magnitude more data with respect to Run 1 and Run 2. The maximum readout rate of 1MHz is the main limitation of the present LHCb trigger. The upgraded detector, apart from major detector upgrades, foresees a full read-out, running at the LHC bunch crossing frequency of 40MHz, using an entirely software based trigger.A new high-throughput PCIe Generation 3 based read-out board, named PCIe40, has been designed for this purpose. The read-out board will allow an efficient and cost-effective implementation of the DAQ system by means of high-speed PC networks. The network-based DAQ system reads data fragments, performs the event building, and transports events to the High-Level Trigger at an estimated aggregate rate of about 32Tbit/s. Different architecture for the DAQ can be implemented, such as push, pull and traffic shaping with barrel-shifter. Possible technology candidates for the foreseen event-builder under study are InfiniBand and Gigabit Ethernet. In order to define the best implementation of the event-builder we are performing tests of the event-builder on different platforms with different technologies. For testing we are using an event-builder evaluator, which consists of a flexible software implementation, to be used on small size test beds as well as on HPC scale facilities. The architecture of DAQ system and up to date performance results will be presented.

The data acquisition system for a fixed target experiment at NICA complex at JINR and its connection to the ATLAS TileCal readout electronics

Today's large-scale science projects have always encountered challenges in processing large data flow from the experiments, the ATLAS detector records proton-proton collisions provided by the Large Hadron Collider (LHC) at CERN every 50 ns which results in a total data flow of 10 Pb/s. These data must be reduced to the science data product for further analysis, thus a very fast decisions need to be executed, to modify this large amounts of data at high rates. The capabilities required to support this scale of data movement is development and improvement of high-throughput electronics. The upgraded LHC will provide collisions at rates that will be at least 10 times higher than those of today due to it's luminosity by 2022. This will require a complete redesign of the read-out electronics and Processing Units (PU) in the Tile-calorimeter (TileCal) of the ATLAS experiment. A general purpose, high-throughput PU has been developed for the TileCal at CERN, by using several ARM-processors in cluster configuration. The PU is capable of handling large data throughput and apply advanced operations at high rates. This system has been proposed for the fixed target experiment at NICA complex to handle the first level processes and event building. The aim of this work is to have a look at the architecture of the data acquisition system (DAQ) of the fixed target experiment at the NICA complex at JINR, by compiling the data-flow requirements of all the subcomponents. Furthermore, the VME DAQ modules characteristics to control, triggering and data acquisition will be described in order to define the DAQ with maximum readout efficiency, no dead time and data selection and compression.

Protocol-Independent Event Building Evaluator for the LHCb DAQ System

The Data Acquisition (DAQ) system of LHCb is a complex real-time system. It will be upgraded to provide LHCb with an all-software, trigger-free readout starting from 2020. Consequently, more CPU power in the form of servers will be needed and the DAQ network will grow to a capacity of 40 Tbps. A PC-based readout system would receive data incoming from the detector, which would then be scattered across builder nodes, and further distributed to a computing farm for data filtering. The design bandwidth of such a DAQ system requires rates as high as 400 Gbps single-duplex per node. These builder nodes will be connected with cost-effective, high-bandwidth data-centre switches in order to minimize the system cost. The behaviour of such an Event Building network can of course be studied in simulation but experience tells us that it is crucial to test, in particular to find out limitations in the switches themselves and to which extent various Event Building protocols can mitigate these limitations. We present a protocol, topology and transport independent emulation software named DAQ Protocol-Independent Performance Evaluator (DAQPIPE). It allows us to test different communication architectures, such as push or pull, with regards to the initiator of the communication. Different topologies and transport protocols can also be tested. We present throughput and stress tests on an InfiniBand FDR multi-rail based LAN network setup, with a focus on the network performance. Large tests on the current system LHCb DAQ are shown to demonstrate the scalability of DAQPIPE itself and its capability to be deployed on any kind of large, tightly interconnected network to test its suitability for Event Building applications.

New DAQ System for the CANDLES Experiment

A new data acquisition (DAQ) system for the CANDLES experiment was developed using the SpaceWire protocol and the DAQ-Middleware framework. The CANDLES experiment uses a trigger board and Flash Analog-to-Digital Converters (FADCs). The SpaceWire helps us construct the DAQ system with a flexible and multi-path access to FADCs and the trigger board. FADCs have a ring buffer with three buffers to detect sequential decays from backgrounds. We developed the DAQ system with parallel read-out to reduce the dead time at high trigger rates using three read-out personal computers (PCs) that are connected to the three buffers in FADCs one by one. Each PC connects all FADCs and the trigger board and gathers a complete data set of one event without any event builder. The maximum DAQ speed of parallel read-out by three PCs was 2.4 times higher than that of single read-out. In order to collect event data sets from PCs, we built the network distributed DAQ system through Ethernet, which is naturally introduced with the DAQ-Middleware. To realize remote monitoring and histogram modification while the DAQ running, we also developed a dynamic online monitor system independent with the DAQ-Middleware.

The ONSEN Data Reduction System for the Belle II Pixel Detector

We present an FPGA-based online data reduction system for the pixel detector of the future Belle II experiment. The occupancy of the pixel detector is estimated at 3 %. This corresponds to a data output rate of more than 20 GB/s after zero suppression, dominated by background. The Online Selection Nodes (ONSEN) system aims to reduce the background data by a factor of 30. It consists of 33 MicroTCA cards, each equipped with a Xilinx Virtex-5 FPGA and 4 GiB DDR2 RAM. These cards are hosted by 9 AdvancedTCA carrier boards. The ONSEN system buffers the entire output data from the pixel detector for up to 5 seconds. During this time, the Belle II high-level trigger PC farm performs an online event reconstruction, using data from the other Belle II subdetectors. It extrapolates reconstructed tracks to the layers of the pixel detector and defines regions of interest around the intercepts. Based on this information, the ONSEN system discards all pixels not inside a region of interest before sending the remaining hits to the event builder system. During a beam test with one layer of the pixel detector and four layers of the surrounding silicon strip detector, including a scaled-down version of the high-level trigger and data acquisition system, the pixel data reduction using regions of interest was exercised. We investigated the data produced in more than 20 million events and verified that the ONSEN system behaved correctly, forwarding all pixels inside regions of interest and discarding the rest.

The Three-Level Event Building System for the Belle II Experiment

Belle II, an experiment designed to search for physics beyond the standard model in the decay of B mesons, will soon start operation at the Super KEKB e <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">+</sup> e <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-</sup> collider. The trigger rate is expected to be 30 kHz with a margin of 10 kHz, and the raw data rate will exceed 30 GB/s. Since the system will be unable to record all data on its storage system, online reduction is required. We have developed a three-level event building system using inexpensive switches and software for traffic smoothing, enabling online event processing at a reasonable cost. The data will be recorded on the storage system about 3 GB/s after rate reduction. This paper describes the architecture and the verification result in actual beam tests.

Achieving High Performance With TCP Over 40 GbE on NUMA Architectures for CMS Data Acquisition

TCP and the socket abstraction have barely changed over the last two decades, but at the network layer there has been a giant leap from a few megabits to 100 gigabits in bandwidth. At the same time, CPU architectures have evolved into the multi-core era and applications are expected to make full use of all available resources. Applications in the data acquisition domain based on the standard socket library running in a Non-Uniform Memory Access (NUMA) architecture are unable to reach full efficiency and scalability without the software being adequately aware about the IRQ (Interrupt Request), CPU and memory affinities. During the first long shutdown of LHC, the CMS DAQ system is going to be upgraded for operation from 2015 onwards and a new software component has been designed and developed in the CMS online framework for transferring data with sockets. This software attempts to wrap the low-level socket library to ease higher-level programming with an API based on an asynchronous event driven model similar to the DAT uDAPL API. It is an event-based application with NUMA optimizations, that allows for a high throughput of data across a large distributed system. This paper describes the architecture, the technologies involved and the performance measurements of the software in the context of the CMS distributed event building.

Time-based Cellular Automaton track finder for the CBM experiment

The future heavy-ion experiment CBM (FAIR/GSI, Darmstadt, Germany) will focus on the measurement of rare probes at interaction rates up to 10 MHz with data flow of up to 1 TB/s. The beam will provide free stream of particles without bunch structure. That requires full online event reconstruction and selection not only in space, but also in time, so- called 4D event building and selection. This is a task of the First-Level Event Selection (FLES) package. The FLES reconstruction and selection package consists of several modules: track finding, track fitting, short-lived particles finding, event building and event selection. The input data are distributed within the FLES farm in a form of so-called time-slices, in which time length is proportional to a compute power of a processing node. A time-slice is reconstructed in parallel between cores within a CPU, thus minimising communication between CPUs. After all tracks of the whole time-slice are found and fitted, they are collected into clusters of tracks originated from common primary vertices. After that short-lived particles are found and the full event building process is finished.

The readout chain for the P̄ANDA MVD strip detector

The P̄ANDA (antiProton ANnihilation at DArmstadt) experiment will study the strong interaction in annihilation reactions between an antiproton beam and a stationary gas jet target. The detector will comprise different sub-detectors for tracking, particle identification and calorimetry. The Micro-Vertex Detector (MVD) as the innermost part of the tracking system will allow precise tracking and detection of secondary vertices. For the readout of the double-sided silicon strip sensors a custom-made ASIC is being developed, employing the Time-over-Threshold (ToT) technique for digitization and utilize time-to-digital converters (TDC) to provide a high-precision time stamp of the hit. A custom-made Module Data Concentrator ASIC (MDC) will multiplex the data of all front-ends of one sensor towards the CERN-developed GBT chip set (GigaBit Transceiver). The MicroTCA-based MVD Multiplexer Board (MMB) at the off-detector site will receive and concentrate the data from the GBT links and transfer it to FPGA-based compute nodes for global event building.

Event building and reconstruction at 30 MHz using a CPU farm

The LHCb experiment will be upgraded between 2018 and 2019 in order to reach unprecedented precision on the measurements of the main observables of the beauty and charm quarks. This paper describes the trigger-less readout system foreseen for the upgrade.

The New CMS DAQ System for Run-2 of the LHC

The data acquisition (DAQ) system of the CMS experiment at the CERN Large Hadron Collider assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of 100 GB/s to the high level trigger (HLT) farm. The HLT farm selects interesting events for storage and offline analysis at a rate of around 1 kHz. The DAQ system has been redesigned during the accelerator shutdown in 2013/14. The motivation is twofold: Firstly, the current compute nodes, networking, and storage infrastructure will have reached the end of their lifetime by the time the LHC restarts. Secondly, in order to handle higher LHC luminosities and event pileup, a number of sub-detectors will be upgraded, increasing the number of readout channels and replacing the off-detector readout electronics with a <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\mu {\hbox {TCA}}$</tex></formula> implementation. The new DAQ architecture will take advantage of the latest developments in the computing industry. For data concentration, 10/40 Gb/s Ethernet technologies will be used, as well as an implementation of a reduced TCP/IP in FPGA for a reliable transport between custom electronics and commercial computing hardware. A Clos network based on 56 Gb/s FDR Infiniband has been chosen for the event builder with a throughput of <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\sim 4~\hbox{Tb/s}$</tex> </formula> . The HLT processing is entirely file based. This allows the DAQ and HLT systems to be independent, and to use the HLT software in the same way as for the offline processing. The fully built events are sent to the HLT with 1/10/40 Gb/s Ethernet via network file systems. Hierarchical collection of HLT accepted events and monitoring meta-data are stored into a global file system. This paper presents the requirements, technical choices, and performance of the new system.

A FPGA-based Cluster Finder for CMOS Monolithic Active Pixel Sensors of the MIMOSA-26 Family

CMOS Monolithic Active Pixel Sensors (MAPS) demonstrated excellent performances in the field of charged particle tracking. Among their strong points are an single point resolution few μm, a light material budget of 0.05% X0 in combination with a good radiation tolerance and high rate capability. Those features make the sensors a valuable technology for vertex detectors of various experiments in heavy ion and particle physics. To reduce the load on the event builders and future mass storage systems, we have developed algorithms suited for preprocessing and reducing the data streams generated by the MAPS. This real-time processing employs remaining free resources of the FPGAs of the readout controllers of the detector and complements the on-chip data reduction circuits of the MAPS.

An Event Building scenario in the trigger-less PANDA experiment

The PANDA experiment will be running up to 2 × 107 antiproton-proton collisions per second at energies reaching 15 GeV. The lack of simple features distinguishing the interesting events from background, as well as strong pileup of events' data streams make the use of a hardware trigger impossible. As a consequence the whole data stream of about 300 GB/s has to be analyzed online, i.e: tracking, vertex finding, particle identification and event building.The GEM Tracker covers polar angles from 4 to 20 degrees in the forward direction, and can be used for the event building in a large fraction of events. In this work the analysis chain and the implemented algorithms will be presented. Moreover, the event builder prototype based on GEM Tracker data will be presented.

FPGA based data acquisition system for COMPASS experiment

This paper presents the current data acquisition system (DAQ) of the COMPASS experiment at CERN and discusses development of a new DAQ. The aim of the new DAQ is to substitute software event building by structure composed of special FPGA cards that will do the event building. The software part of the new DAQ is robust multinode system with high emphasis on reliability. It is based on state machines and implemented in C++ with usage of the QT framework, the DIM library, and the IPBus technology. A prototype of the system has been developed and tested. The new DAQ software fulfills given requirements.

Time Structure Analysis of the LHCb DAQ Network

The LHCb DAQ Network is a real time high performance network, in which 350 data sources send data over a Gigabit Ethernet LAN to more than 1500 receiving nodes. The aggregated throughput of the application, called Event Building, is more than 60 Gbps. The protocol employed by LHCb makes the sending nodes transmit simultaneously portions of events to one receiving node at a time, which is selected using a credit-token scheme. The resulting traffic is very bursty and sensitive to irregularities in the temporal distribution of packet-bursts to the same destination or region of the network.In order to study the relevant properties of such a dataflow, a non-disruptive monitoring setup based on a networking capable FPGA (Netfpga) has been deployed. The Netfpga allows order of hundred nano-second precise time-stamping of packets. We study in detail the timing structure of the Event Building communication, and we identify potential effects of micro-bursts like buffer packet drops or jitter.

The new CMS DAQ system for LHC operation after 2014 (DAQ2)

The Data Acquisition system of the Compact Muon Solenoid experiment at CERN assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of 100 GByte/s. We are presenting the design of the 2nd generation DAQ system, including studies of the event builder based on advanced networking technologies such as 10 and 40 Gbit/s Ethernet and 56 Gbit/s FDR Infiniband and exploitation of multicore CPU architectures. By the time the LHC restarts after the 2013/14 shutdown, the current compute nodes, networking, and storage infrastructure will have reached the end of their lifetime. In order to handle higher LHC luminosities and event pileup, a number of sub-detectors will be upgraded, increase the number of readout channels and replace the off-detector readout electronics with a μTCA implementation. The second generation DAQ system, foreseen for 2014, will need to accommodate the readout of both existing and new off-detector electronics and provide an increased throughput capacity. Advances in storage technology could make it feasible to write the output of the event builder to (RAM or SSD) disks and implement the HLT processing entirely file based.

SND data acquisition system upgrade

The data acquisition (DAQ) system of the SND detector successfully operated during four data-taking seasons (2010–2013) at the e+e− collider VEPP-2000. Currently the collider is shut down for planned reconstruction, which is expected to increase the VEPP-2000 luminosity and data flow from the SND detector electronics by up to 10 times.Since current DAQ system implementation (electronics and computer part) does not have enough reserve for selection of events in the new environment without compromising quality, there arose the need for the system upgrade.Here we report on the major SND data acquisition system upgrade which includes developing new electronics for digitization and data transfer, complete redesign of the data network, increasing of the DAQ computer farm processing capacity and making the event building process concurrent. These measures will allow us to collect data flow from the most congested detector subsystems in parallel in contrast to the current situation. We would like to discuss also the possibility to implement full software trigger solution in the future.

Development of a read out driver for ATLAS micromegas based on the Scalable Readout System

With future LHC luminosity upgrades, part of the ATLAS muon spectrometer has to be changed, to cope with the increased flux of uncorrelated neutron and gamma particles. Micromegas detectors were chosen as precision tracker for the New Small Wheels, that will replace the current Small Wheel muon detector stations during the LHC shutdown foreseen for 2018. To read out these detectors together with all other ATLAS subsystems, a readout driver was developed to integrate these micromegas detectors into the ATLAS data acquisition infrastructure.The readout driver is based on the Scalable Readout System, and its tasks include trigger handling, slow control, event building and data transmission to the high-level readout systems. This article describes the layout and functionalities of this readout driver and its components, as well as a test of its functionalities in the cosmic ray facility of Ludwig-Maximilians University Munich.

A Comprehensive Zero-Copy Architecture for High Performance Distributed Data Acquisition Over Advanced Network Technologies for the CMS Experiment

This paper outlines a software architecture where zero-copy operations are used comprehensively at every processing point from the Application layer to the Physical layer. The proposed architecture is being used during feasibility studies on advanced networking technologies for the CMS experiment at CERN. The design relies on a homogeneous peer-to-peer message passing system, which is built around memory pool caches allowing efficient and deterministic latency handling of messages of any size through the different software layers. In this scheme portable distributed applications can be programmed to process input to output operations by mere pointer arithmetic and DMA operations only. The approach combined with the open fabric protocol stack (OFED) allows one to attain near wire-speed message transfer at application level. The architecture supports full portability of user applications by encapsulating the protocol details and network into modular peer transport services whereas a transparent replacement of the underlying protocol facilitates deployment of several network technologies like Gigabit Ethernet, Myrinet, Infiniband, etc. Therefore, this solution provides a protocol-independent communication framework and prevents having to deal with potentially difficult couplings when the underlying communication infrastructure is changed. We demonstrate the feasibility of this approach by giving efficiency and performance measurements of the software in the context of the CMS distributed event building studies.

The Belle II Pixel Detector Data Acquisition and Reduction System

The upcoming Belle II experiment is designed to work at a luminosity of 8×10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">35</sup> cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-2</sup> s <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-1</sup> , 40 times higher than its predecessor. The pixel detector of Belle II with its ~ 8 million channels will deliver ten times more data than all other sub-detectors together. A data rate of 22 Gbytes/s is expected for a trigger rate of 30 kHz and an estimated pixel detector occupancy of 3%, which is by far exceeding the specifications of the Belle II event builder system. Therefore a realtime data reduction of a factor > 30 is needed. A hardware platform capable of processing this amount of data is the ATCA based Compute Node (CN). Each CN consists of an xTCA carrier board and four AMC/xTCA daughter boards. The carrier board supplies the high bandwidth connectivity to the other CNs via Rocket-IO links. In the current prototype design, each AMC board is equipped with a Xilinx Virtex-5 FX70T FPGA, 4 GB of memory, Gbit Ethernet and two bi-directional optical links allowing for a bandwidth of up to 12.5 Gbits/s. IPMI control of mother and daughter board is foreseen. One ATCA shelf containing 10 mother boards/40 daughter boards is sufficient to process the data from the 40 front end boards. The data reduction on the CN is done in two steps. First, the event data delivered by the front end electronics via optical links is stored in memory until the high level trigger (HLT) decision has been made. The HLT rejects >2/3 of these events. In a second step, pixel data of positively triggered events is reduced with the help of regions of interest (ROI), calculated by the HLT from projecting trajectories back to the pixel detector plane. The design allows additional ROI inputs computed from hit cluster properties or tracklets from the surrounding silicon strip detector. The final data reduction is achieved by sending only data within these ROIs to the main event builder.