Event Building System Research Articles

Design and Commissioning of the First 32-Tbit/s Event-Builder

The LHCb experiment is a forward spectrometer, designed to study beauty and charm quarks Physics at the LHC. To exploit of the higher luminosity that will be delivered during Run3, the full experiment needed a substantial upgrade, from the detector to the Data Acquisition (DAQ) and High Level Trigger (HLT). In this paper, we will focus on the new DAQ system for the LHCb experiment; that represents a substantial paradigm shift compared to the previous one, and to similar systems used by similar experiments in the past and present times. To overcome the inefficiencies introduced by a local selection implemented directly with the readout hardware, the Run3 system is designed to perform a full software reconstruction of all the produced events. To achieve this, both the DAQ and the HLT need to process the ~ 30 MHz full event-rate. In particular, this paper will introduce the final design of the system; it will provide a focus on the hardware and software design of the Event Building (EB) and how we integrated technologies designed for the High Performance Computing (HPC) world - like InfiniBand HDR (200 Gb/s)- into the DAQ system; we will present performance measurements of the full EB system under different operational conditions; and we will provide a feedback from EB operation during the beginning of the data-taking.

A DAQ Upgrade Solution for Belle II Experiment

While the current Belle II data acquisition (DAQ) system has proven to be reliable in the first collisions delivered since May 2018, an upgrade to increase the bandwidth of the readout system is foreseen to be deployed over the next few years for higher target luminosity SuperKEKB running. Currently, the front-end electronics (FEE) of individual subdetectors are connected to the online event building system via a unified optical link called Belle2Link, offering unidirectional data readout as well as bidirectional slow control access. The FEE data are received by high-speed link boards (HSLB) and common pipelined platform for electronics readout (COPPER) boards collecting data from up to four links, forwarding to readout PCs (ROPC), which further aggregate the data and forward the data to the event builder. It is planned to keep the Belle2Link and subsystems FEE as they are, but to replace the HSLB/COPPER (including the back-end side of Belle2link) and ROPC with an upgraded high-density system with increased throughput and processing capabilities, to enable the operation at higher rates. This article presents a solution for this upgrade based on the xTCA-for-Physics (MicroTCA)-compatible concentration preprocessing and fan-out plus (CPPF+) board featuring significant data concentration and processing power, providing up to 40-Gb Ethernet output bandwidth. Belle2Link back-end firmware and subevent building are implemented in the CPPF+ board to deliver equivalent functionality as the COPPER/ROPC. The MicroTCA platform provides scalability, reconfigurability, and reliable operations, even for a future possible increase in bandwidth and potential FEE upgrades. A fully functional demonstration readout system has been constructed and verified with the time-of-propagation (TOP) detector front-end test bench set up at KEK in terms of functionality as well as performance. Results show that this solution fulfills the requirements of the DAQ upgrade and works stably and reliably in long-term tests.

Network simulation of a 40 MHz event building system for the LHCb experiment

The LHCb experiment will be upgraded in 2021, and a new triggerless readout system will be implemented. In the upgraded system, both Event Building (EB) and event selection will be performed in software for every collision produced in every bunch-crossing of the LHC. In order to transport the full data rate of 32 Tb/s, we will use state of the art off-the-shelf network technologies, e.g. InfiniBand EDR. The full event building system will require around 500 nodes interconnected together via a non-blocking topology. Because of the size of the system it is challenging to test at production scale, before the actual procurement. Therefore, we resort to network simulations as a powerful tool for finding the optimal configuration. We developed an accurate low-level description of an InfiniBand based network with event building like traffic. We will present a full-scale simulation of a possible implementation of the LHCb EB network.

Flit-Level InfiniBand Network Simulations of the DAQ System of the LHCb Experiment for Run-3

The LHCb (Large Hadron Collider beauty) experiment is designed to study differences between particles and anti-particles as well as very rare decays in the charm and beauty sector at the LHC (Large Hadron Collider). The detector will be upgraded in 2019 and a new trigger-less readout system will be implemented in order to significantly increase its efficiency and take advantage of the increased machine luminosity. In the upgraded system, both event building and event filtering will be performed in software for all the data produced in every bunch-crossing of the LHC. In order to transport the full data rate of 32 Tb/s we will use custom FPGA readout boards (PCIe40) and state of the art off-the-shelf network technologies. The full event building system will require around 500 nodes interconnected together. From a networking point of view, event building traffic has an all-to-all pattern, therefore it tends to create high network congestion. In order to maximize the link utilization different techniques can be adopted in various areas like traffic shaping, network topology and routing optimization. The size of the system makes it very difficult to test at production scale, before the actual procurement. We resort therefore to network simulations as a powerful tool for finding the optimal configuration. We will present an accurate low level description of an InfiniBand based network with event building like traffic. We will show comparison between simulated and real systems and how changes in the input parameters affect performances.

Performance of the CMS Event Builder

The data acquisition system (DAQ) of the CMS experiment at the CERN Large Hadron Collider assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of to the high-level trigger farm. The DAQ architecture is based on state-of-the-art network technologies for the event building. For the data concentration, 10/40 Gbit/s Ethernet technologies are used together with a reduced TCP/IP protocol implemented in FPGA for a reliable transport between custom electronics and commercial computing hardware. A 56 Gbit/s Infiniband FDR Clos network has been chosen for the event builder. This paper presents the implementation and performance of the event-building system.

Leveraging multimodal information for event summarization and concept-level sentiment analysis

The rapid growth in the amount of user-generated content (UGCs) online necessitates for social media companies to automatically extract knowledge structures (concepts) from photos and videos to provide diverse multimedia-related services. However, real-world photos and videos are complex and noisy, and extracting semantics and sentics from the multimedia content alone is a very difficult task because suitable concepts may be exhibited in different representations. Hence, it is desirable to analyze UGCs from multiple modalities for a better understanding. To this end, we first present the EventBuilder system that deals with semantics understanding and automatically generates a multimedia summary for a given event in real-time by leveraging different social media such as Wikipedia and Flickr. Subsequently, we present the EventSensor system that aims to address sentics understanding and produces a multimedia summary for a given mood. It extracts concepts and mood tags from visual content and textual metadata of UGCs, and exploits them in supporting several significant multimedia-related services such as a musical multimedia summary. Moreover, EventSensor supports sentics-based event summarization by leveraging EventBuilder as its semantics engine component. Experimental results confirm that both EventBuilder and EventSensor outperform their baselines and efficiently summarize knowledge structures on the YFCC100M dataset.

The ONSEN Data Reduction System for the Belle II Pixel Detector

We present an FPGA-based online data reduction system for the pixel detector of the future Belle II experiment. The occupancy of the pixel detector is estimated at 3 %. This corresponds to a data output rate of more than 20 GB/s after zero suppression, dominated by background. The Online Selection Nodes (ONSEN) system aims to reduce the background data by a factor of 30. It consists of 33 MicroTCA cards, each equipped with a Xilinx Virtex-5 FPGA and 4 GiB DDR2 RAM. These cards are hosted by 9 AdvancedTCA carrier boards. The ONSEN system buffers the entire output data from the pixel detector for up to 5 seconds. During this time, the Belle II high-level trigger PC farm performs an online event reconstruction, using data from the other Belle II subdetectors. It extrapolates reconstructed tracks to the layers of the pixel detector and defines regions of interest around the intercepts. Based on this information, the ONSEN system discards all pixels not inside a region of interest before sending the remaining hits to the event builder system. During a beam test with one layer of the pixel detector and four layers of the surrounding silicon strip detector, including a scaled-down version of the high-level trigger and data acquisition system, the pixel data reduction using regions of interest was exercised. We investigated the data produced in more than 20 million events and verified that the ONSEN system behaved correctly, forwarding all pixels inside regions of interest and discarding the rest.

The Three-Level Event Building System for the Belle II Experiment

Belle II, an experiment designed to search for physics beyond the standard model in the decay of B mesons, will soon start operation at the Super KEKB e <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">+</sup> e <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-</sup> collider. The trigger rate is expected to be 30 kHz with a margin of 10 kHz, and the raw data rate will exceed 30 GB/s. Since the system will be unable to record all data on its storage system, online reduction is required. We have developed a three-level event building system using inexpensive switches and software for traffic smoothing, enabling online event processing at a reasonable cost. The data will be recorded on the storage system about 3 GB/s after rate reduction. This paper describes the architecture and the verification result in actual beam tests.

The Belle II Pixel Detector Data Acquisition and Reduction System

The upcoming Belle II experiment is designed to work at a luminosity of 8×10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">35</sup> cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-2</sup> s <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-1</sup> , 40 times higher than its predecessor. The pixel detector of Belle II with its ~ 8 million channels will deliver ten times more data than all other sub-detectors together. A data rate of 22 Gbytes/s is expected for a trigger rate of 30 kHz and an estimated pixel detector occupancy of 3%, which is by far exceeding the specifications of the Belle II event builder system. Therefore a realtime data reduction of a factor > 30 is needed. A hardware platform capable of processing this amount of data is the ATCA based Compute Node (CN). Each CN consists of an xTCA carrier board and four AMC/xTCA daughter boards. The carrier board supplies the high bandwidth connectivity to the other CNs via Rocket-IO links. In the current prototype design, each AMC board is equipped with a Xilinx Virtex-5 FX70T FPGA, 4 GB of memory, Gbit Ethernet and two bi-directional optical links allowing for a bandwidth of up to 12.5 Gbits/s. IPMI control of mother and daughter board is foreseen. One ATCA shelf containing 10 mother boards/40 daughter boards is sufficient to process the data from the 40 front end boards. The data reduction on the CN is done in two steps. First, the event data delivered by the front end electronics via optical links is stored in memory until the high level trigger (HLT) decision has been made. The HLT rejects >2/3 of these events. In a second step, pixel data of positively triggered events is reduced with the help of regions of interest (ROI), calculated by the HLT from projecting trajectories back to the pixel detector plane. The design allows additional ROI inputs computed from hit cluster properties or tracklets from the surrounding silicon strip detector. The final data reduction is achieved by sending only data within these ROIs to the main event builder.

New Backend Processor Card for the Pipelined Readout System at Belle II

The Belle II experiment at the Super KEKB storage ring at KEK will search New Physics beyond the Standard Model, with 40 times higher luminosity than at KEKB. Much higher event rates are accounted for by moving all front-end digitizers inside or near the detector, and the resulting data received by a common pipeline readout platform developed in KEK. There are about 200 such platforms in the Belle II experiment. We have developed a new Intel Atom based processor card to handle the data and send it to the event building system in three steps. We fully utilize the PCI bus bandwidth and meet the high data rate requirement. In this report, we describe this processor card and the results from testing it in a realistic setup.

The ATLAS ReadOut System—Performance with first data and perspective for the future

The ATLAS ReadOut System (ROS) receives data fragments from ∼1600 detector readout links, buffers them and provides them on demand to the second-level trigger or to the event building system. The ROS is implemented with ∼150PCs. Each PC houses a few, typically 4, custom-built PCI boards (ROBIN) and a 4-port PCIe Gigabit Ethernet NIC. The PCs run a multi-threaded object-oriented application managing the requests for data retrieval and for data deletion coming through the NIC, and the collection and output of data from the ROBINs. At a nominal event fragment arrival rate of 75kHz the ROS has to concurrently service up to approximately 20kHz of data requests from the second-level trigger and up to 3.5kHz of requests from event building nodes. The full system has been commissioned in 2007. Performance of the system in terms of stability and reliability, results of laboratory rate capability measurements and upgrade scenarios are discussed in this paper.

Effects of Adaptive Wormhole Routing in Event Builder Networks

The data acquisition system of the CMS experiment at the Large Hadron Collider features a two-stage event builder, which combines data from about 500 sources into full events at an aggregate throughput of 100 GB/s. To meet the requirements, several architectures and interconnect technologies have been quantitatively evaluated. Myrinet will be used for the communication from the underground frontend devices to the surface event building system. Gigabit Ethernet is deployed in the surface event building system. Nearly full bi-section throughput can be obtained using a custom software driver for Myrinet based on barrel shifter traffic shaping. This paper discusses the use of Myrinet dual-port network interface cards supporting channel bonding to achieve virtual 5 GBit/s links with adaptive routing to alleviate the throughput limitations associated with wormhole routing. Adaptive routing is not expected to be suitable for high-throughput event builder applications in high-energy physics. To corroborate this claim, results from the CMS event builder preseries installation at CERN are presented and the problems of wormhole routing networks are discussed.

The COMPASS data acquisition system

A fully pipelined and massively parallel data acquisition system has been developed for the COMPASS experiment at CERN. The main requirements are to read 250000 detector channels at a trigger rate of up to 100 kHz. Such high rates are only possible when using a hit selection mechanism on the front-end combined with dead-time free readout. For this purpose, a time-to-digital converter (TDC) chip has been developed and is used for all time measurement applications in COMPASS. Distributed, field programmable gate array (FPGA)-based readout-driver modules handle parallel front-end initialization, synchronous trigger and control-signal distribution, and local event building at a processing speed of 160 Mbyte/s. Each of the 160 readout-driver modules connects to 16 front-end boards through independent twisted pair cables (CAT 7, 600 MHz) or optical fibers using an industrial (ESCON), self synchronizing link at 40 Mbyte/s. Automatic configuration through unique module and link identification ensures the flexibility and scalability to very large detector systems. The preprocessed data are transmitted through optical fibers at 160 Mbyte/s to the master event building system. Here global event building is realized on high-performance personal computers (PCs) and Gigabit ethernet network components. The complete events are sent to the central data recording at the CERN main site at an average rate of 40 Mbyte/s and stored in an object oriented database. A reduced system was set up for the commissioning run of COMPASS in 2000. Operation of the full system starts in July 2001.

Process and Data Flow Control in KLOE

The core of the KLOE distributed event building system is a switched network. The online processes are distributed over a large set of processors in this network. All processes have to change coherently their state of activity as a consequence of local or remote commands. A fast and reliable message system based on the SNMP protocol has been developed. A command server has been implemented as a non privileged daemon able to respond to “set” and “get” queries on private SNMP variables. This process is able to convert remote set operations into local commands and to map automatically an SNMP subtree on a user-defined set of process variables. Process activity can be continuously monitored by remotely accessing their variables by means of the command server. Only the command server is involved in these operations, without disturbing the process flow. Subevents coming from subdetectors are sent to different nodes of a computing farm for the last stage of event building. Based on features of the SNMP protocol and of the KLOE message system, the Data Flow Control System (DFC) is able to rapidly redirect network traffic, keeping in account the dynamics of the whole DAQ system in order to assure coherent subevent addressing in an asynchronous “push” architecture, without introducing dead time. The KLOE DFC is currently working in the KLOE DAQ system. Its main characteristics and performance are discussed.

The Belle event building system

We have built a barrel-shifter type event builder for the Belle experiment and operated it successfully. This is the first experience applying the barrel-shifter type event builder to the real experiment. The type of our event builder is global-traffic control-based. The Belle event builder is required to be scalable up to 11 source nodes (detector readout subsystems) and six destination nodes (online computer farm subsystems). The average event size is estimated to be 30 kbyte and the maximum trigger rate is 500 Hz. This event builder has been operated with full detectors since Dec. 1998. We have shown that the barrel-shifter type event builder is applicable for this size of experiment.

Self-routing event building on sorting network

Event building system based on sorting network has been developed and discussed. The event builder comprises a comparator with subsidiary functions as a network node, FIFOs as a time slot interchanger at the input stage and data buffers at the output stage. The event building system gathers data without any complex controls by the software. The throughput of the data and scalability were examined for various speeds of the system clocks. The system has a scalability of the speed for the network-size. Though the throughput is limited by the speed of the device we chose, the speed can be increased by choosing a faster device.

Global traffic control system for unbalanced data flow on nonsquare event builder in BELLE experiment

It has been stated that an unbalanced data flow cannot be handled by blocking switches such as a barrel shifter. It is not true for a square switch, which has the same number of destination nodes as source nodes. However, if the switch is not square, a simple barrel shifter is not adequate. We have studied the minimal switch configuration for the event builder system in a BELLE experiment in KEK. The performance in the actual experiment was measured.