Event Building Research Articles

Effects of Adaptive Wormhole Routing in Event Builder Networks

The data acquisition system of the CMS experiment at the Large Hadron Collider features a two-stage event builder, which combines data from about 500 sources into full events at an aggregate throughput of 100 GB/s. To meet the requirements, several architectures and interconnect technologies have been quantitatively evaluated. Myrinet will be used for the communication from the underground frontend devices to the surface event building system. Gigabit Ethernet is deployed in the surface event building system. Nearly full bi-section throughput can be obtained using a custom software driver for Myrinet based on barrel shifter traffic shaping. This paper discusses the use of Myrinet dual-port network interface cards supporting channel bonding to achieve virtual 5 GBit/s links with adaptive routing to alleviate the throughput limitations associated with wormhole routing. Adaptive routing is not expected to be suitable for high-throughput event builder applications in high-energy physics. To corroborate this claim, results from the CMS event builder preseries installation at CERN are presented and the problems of wormhole routing networks are discussed.

Data Acquisition Backbone Core DABC

For the new experiments at FAIR new concepts of data acquisition systems have to be developed like the distribution of self-triggered, time stamped data streams over high performance networks for event building. The Data Acquisition Backbone Core (DABC) is a software package designed for FAIR detector tests, readout components test, and data flow investigations. All kinds of data channels (front-end systems) are connected by program plug-ins into functional components of DABC like data input, combiner, scheduler, event builder, analysis and storage components. After detailed simulations real tests of event building over a switched network (InfiniBand clusters with up to 23 nodes) have been performed. With the DABC software more than 900 MByte/s input and output per node can be achieved meeting the most demanding requirements. The software is ready for the implementation of various test beds needed for the final design of data acquisition systems at FAIR.

Performance of the Final Event Builder for the ATLAS Experiment

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> Event data from proton-proton collisions at the LHC will be selected by the ATLAS experiment by a three level trigger system, which reduces the initial bunch crossing rate of 40 MHz at its first two trigger levels (LVL1+LVL2) to <formula formulatype="inline"><tex>$\sim$</tex></formula>3 kHz. At this rate the Event-Builder collects the data from all Read-Out system PCs (ROSs) and provides fully assembled events to the the Event-Filter (EF), which is the third level trigger, to achieve a further rate reduction to <formula formulatype="inline"> <tex>$\sim$</tex></formula>200 Hz for permanent storage. The Event-Builder is based on a farm of <formula formulatype="inline"><tex>$O(100)$</tex></formula> PCs, interconnected via Gigabit Ethernet to <formula formulatype="inline"><tex>$O(150)$</tex> </formula> ROSs. These PCs run Linux and multi-threaded software applications implemented in C++. All the ROSs and one third of the Event-Builder PCs are already installed and commissioned. Performance measurements have been exercised on this initial system, which show promising results that the required final data rates and bandwidth for the ATLAS event builder are in reach. </para>

CMS DAQ Event Builder Based on Gigabit Ethernet

The CMS data acquisition system is designed to build and filter events originating from 476 detector data sources at a maximum trigger rate of 100 kHz. Different architectures and switch technologies have been evaluated to accomplish this purpose. Events will be built in two stages: the first stage will be a set of event builders called front-end driver (FED) builders. These will be based on Myrinet technology and will pre-assemble groups of about eight data sources. The second stage will be a set of event builders called readout builders. These will perform the building of full events. A single readout builder will build events from about 60 sources of 16 kB fragments at a rate of 12.5 kHz. In this paper, we present the design of a readout builder based on TCP/IP over Gigabit Ethernet and the refinement that was required to achieve the design throughput. This refinement includes architecture of the readout builder, the setup of TCP/IP, and hardware selection.

The Read-Out Driver for the RPC of the ATLAS Muon Spectrometer

The spectrometer of the ATLAS experiment has been designed to identify muon tracks with transverse momenta up to 1 TeV/c. Its barrel section is made of monitored drift tubes and resistive plate chambers (RPCs) arranged into 32 sectors, to form two wheels surrounding the interaction point. The RPC subsystem provides the level-1 trigger in the barrel and it is read out by a specific DAQ system. On-detector electronics pack the RPC data in frames with an event number assigned by the trigger logic and transmit them to the counting room optically. Data from each sector are then routed together to a read-out driver (ROD) board. This is a custom processor that parses the frames, checks their coherence and builds a data structure for all the RPCs of one of the 32 sectors of the spectrometer. Each ROD sends the event fragments to a read-out buffer for further event building and analysis. The ROD is a VME64x board, designed around two Xilinx Virtex-II FPGAs and an ARM7 microcontroller. In this paper we describe the system architecture, the event binding algorithms and the monitoring features embedded into the design. The board is in production and will be deployed in the first ATLAS runs.

Integration of the Trigger and Data Acquisition Systems in ATLAS

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> During 2006 and spring 2007, integration and commissioning of trigger and data acquisition (TDAQ) equipment in the ATLAS experimental area has progressed. Much of the work has focused on a final prototype setup consisting of around eighty computers representing a subset of the full TDAQ system. There have been a series of technical runs using this setup. Various tests have been run including those where around 6 k Level-1 preselected simulated proton–proton events have been processed in a loop mode through the trigger and dataflow chains. The system included the readout buffers containing the events, event building, second level and third level trigger processors. Aspects critical for the final system, such as event processing times, have been studied using different trigger algorithms as well as the different dataflow components. </para>

The CMS High Level Trigger System

The CMS data acquisition (DAQ) system relies on a purely software driven high level trigger (HLT) to reduce the full Level 1 accept rate of 100 kHz to approximately 100 Hz for archiving and later offline analysis. The HLT operates on the full information of events assembled by an event builder collecting detector data from the CMS front-end systems. The HLT software consists of a sequence of reconstruction and filtering modules executed on a farm of O(1000) CPUs built from commodity hardware. This paper presents the architecture of the CMS HLT, which integrates the CMS reconstruction framework in the online environment. The mechanisms to configure, control, and monitor the filter farm and the procedures to validate the filtering code within the DAQ environment are described.

Modeling of the Architectural Studies for the PANDA DAT System

We present design studies of the DAQ and trigger system (DAT) for the PANDA detector proposed for the new FAIR facility at GSI. The broad physics program of PANDA requires a novel DAT system able to cope with high interaction rates (up to Hz) and to trigger on various event topologies simultaneously. We used SystemC as a modeling platform to investigate candidate architecture for the PANDA DAT system. We simulated the behavior of the complete system with simplified models of all the components. The model covers detector buffers connected via Ethernet to farms of computing nodes constituting two filtering levels and an event building level. We present results from modeling illustrating the impact of the key architectural choices and parameters on the overall performance.

Implementation of on-line data reduction algorithms in the CMS Endcap Preshower Data Concentrator Cards

The CMS Endcap Preshower (ES) sub-detector comprises 4288 silicon sensors, each containing 32 strips. The data are transferred from the detector to the counting room via 1208 optical fibres running at 800Mbps. Each fibre carries data from two, three or four sensors. For the readout of the Preshower, a VME-based system, the Endcap Preshower Data Concentrator Card (ES-DCC), is currently under development. The main objective of each readout board is to acquire on-detector data from up to 36 optical links, perform on-line data reduction via zero suppression and pass the concentrated data to the CMS event builder. This document presents the conceptual design of the Reduction Algorithms as well as their implementation in the ES-DCC FPGAs. These algorithms, as implemented in the ES-DCC, result in a data-reduction factor of 20.

Verification of the GREAT Total Data Readout System Using Pseudo Random Pattern Generator

Total Data Readout (TDR) is a powerful data acquisition tool developed for use with Recoil Decay Tagging (RDT) experiments on the GREAT (Gamma, Recoil, Electron, Alpha Tagging) spectrometer. RDT experiments correlate prompt radioactive decays detected at the target position with subsequent decays of recoiling reaction products implanted in GREAT, located at the focal plane of the RITU recoil separator, several microseconds later. TDR uses independent channels associated in a software event builder based on timestamps to form a single time ordered data stream. It is crucial to maintain the accuracy of the timestamps over long periods (microseconds to hours) for the study of Exotic Nuclei. The TDR system cannot be tested fully by use of calibrated sources with GREAT and a novel method of verifying the system is required. Our solution is to use the predictable randomness of a Pseudo Random Number generator based on Linear Feedback Shift Registers (LFSR) implemented in a 16 channel VME module. The random value defines the delay between pulses into the TDR system. The recorded results are sorted and checked for correspondence with the LFSR sequence. We show the method, design of the module, and discuss the results and benefits of the tests

ATLAS DataFlow: the read-out subsystem, results from trigger and data-acquisition system testbed studies and from modeling

In the ATLAS experiment at the LHC, the output of read-out hardware specific to each subdetector will be transmitted to buffers, located on custom made PCI cards (ROBINs). The data consist of fragments of events accepted by the first-level trigger at a maximum rate of 100 kHz. Groups of four ROBINs will be hosted in about 150 Read-Out Subsystem (ROS) PCs. Event data are forwarded on request via Gigabit Ethernet links and switches to the second-level trigger or to the Event builder. In this paper a discussion of the functionality and real-time properties of the ROS is combined with a presentation of measurement and modelling results for a testbed with a size of about 20% of the final DAQ system. Experimental results on strategies for optimizing the system performance, such as utilization of different network architectures and network transfer protocols, are presented for the testbed, together with extrapolations to the full system.

FutureDAQ for CBM: on-line event selection

At the upcoming new Facility for Antiproton and Ion Research FAIR at GSI the Compressed Baryonic Matter experiment CBM requires a new architecture of front-end electronics, data acquisition, and event processing. The detector systems of CBM are a Silicon Tracker System (STS), RICH detectors, a TRD, RPCs, and an electromagnetic calorimeter. The envisioned interaction rate of 10 MHz will produce a data rate of up to 1 TByte/s. Because of the complexity and variability of trigger decisions no common trigger will be applied. Instead, the front-end electronics of all detectors will be self-triggered and marked by time stamps. The full data rate must be switched through a high speed network fabric into a computational network with configurable processing resources for event building and filtering. The decision for selecting candidate events requires tracking, primary vertex reconstruction, and secondary vertex finding in the STS at the full interaction rate. The essential performance factor is now computational throughput rather than decision latency, which results in a much better utilization of the processing resources especially in the case of heavy ion collisions with strongly varying multiplicities. The development of key components is supported by the FutureDAQ project of the European Union (FP6 I3HP JRA1).

The ROD crate DAQ software framework of the ATLAS data acquisition system

In the ATLAS experiment at the LHC, the ROD Crate DAQ provides a complete software framework to implement data acquisition functionality at the boundary between the detector specific electronics and the common part of the data acquisition system. Based on a plugin mechanism, it allows selecting and using common services (like data output and data monitoring channels) and developing software to control and acquire data from detector specific modules providing the infrastructure for control, monitoring and calibration. Including also event building functionality, the ROD Crate DAQ is intended to be the main data acquisition tool for the first phase of detector commissioning. This paper presents the design, functionality and performance of the ROD Crate DAQ and its usage in the ATLAS data acquisition system and during detector tests.

Sub Farm Interface of the ATLAS Dataflow System

Sub Farm Interface is the event builder of the ATLAS(A Toroidal LHC ApparatuS) Dataflow System. It receives event fragments from the Read Out System, builds full events and sends complete events to the Event Filter for high level event selection. This paper describes the implementation of the Sub Farm Interface. Furthermore, this paper introduces some issues on SFI(Sub Farm Interface) optimization and the monitoring service inside SFI.

New communication network protocol for a data acquisition system

An event builder based on communication networks has been used in high-energy physics experiments, and various networks have been adopted, for example, IEEE 802.3 (Ethernet), asynchronous transfer mode (ATM), and so on. In particular, Ethernet is widely used because its infrastructure is very cost effective. Many systems adopt standard protocols that are designed for a general network. However, in the case of an event builder, the communication pattern between stations is different from that in a general network. The unique communication pattern causes congestion, and thus makes it difficulty to quantitatively design the network. To solve this problem, we have developed a simple network protocol for a data acquisition (DAQ) system. The protocol is designed to keep the sequence of senders so that no congestion occurs. We implemented the protocol on a small hardware component [a field programmable gate array (FPGA)] and measured the performance, so that it will be ready for a generic DAQ system

Development of event building farm for SuperKEKB

For the purpose of new physics search in rare B meson decays, an upgrade of the KEKB accelerator is being planned where the luminosity becomes more than ten times higher. The trigger rate with the increased luminosity is expected to reach up to 30 kHz with the event size of ~300 kB. A new event builder is designed to cope with the condition by modularizing the system currently used in the Belle experiment and by operating multiple units in parallel. A test bench is built to study the new design and results of performance tests are reported

Design, deployment and functional tests of the online event filter for the ATLAS experiment at LHC

The Event Filter (EF) selection stage is a fundamental component of the ATLAS Trigger and Data Acquisition architecture. Its primary function is the reduction of data flow and rate to values acceptable by the mass storage operations and by the subsequent offline data reconstruction and analysis steps. The computing instrument of the EF is organized as a set of independent subfarms, each connected to one output of the Event Builder (EB) switch fabric. Each subfarm comprises a number of processors analyzing several complete events in parallel. This paper describes the design of the ATLAS EF system, its deployment in the 2004 ATLAS combined test beam together with some examples of integrating selection and monitoring algorithms. Since the processing algorithms are not explicitly designed for EF but are adapted from the offline ones, special emphasis is reserved to system reliability and data security, in particular for the case of failures in the processing algorithms. Other key design elements have been system modularity and scalability. The EF shall be able to follow technology evolution and should allow for using additional processing resources possibly remotely located

Readout, first- and second-level triggers of the new Belle silicon vertex detector

Abstract A major upgrade of the Silicon Vertex Detector (SVD 2.0) of the Belle experiment at the KEKB factory was installed along with new front-end and back-end electronics systems during the summer shutdown period in 2003 to cope with higher particle rates, improve the track resolution and meet the increasing requirements of radiation tolerance. The SVD 2.0 detector modules are read out by VA1TA chips which provide “fast or” (hit) signals that are combined by the back-end FADCTF modules to coarse, but immediate level 0 track trigger signals at rates of several tens of a kHz. Moreover, the digitized detector signals are compared to threshold lookup tables in the FADCTFs to pass on hit information on a single strip basis to the subsequent level 1.5 trigger system, which reduces the rate below the kHz range. Both FADCTF and level 1.5 electronics make use of parallel real-time processing in Field Programmable Gate Arrays (FPGAs), while further data acquisition and event building is done by PC farms running Linux. The new readout system hardware is described and the first results obtained with cosmics are shown.

Readout, first- and second-level triggers of the new Belle silicon vertex detector

A major upgrade of the Silicon Vertex Detector (SVD 2.0) of the Belle experiment at the KEKB factory was installed along with new front-end and back-end electronics systems during the summer shutdown period in 2003 to cope with higher particle rates, improve the track resolution and meet the increasing requirements of radiation tolerance. The SVD 2.0 detector modules are read out by VA1TA chips which provide “fast or” (hit) signals that are combined by the back-end FADCTF modules to coarse, but immediate level 0 track trigger signals at rates of several tens of a kHz. Moreover, the digitized detector signals are compared to threshold lookup tables in the FADCTFs to pass on hit information on a single strip basis to the subsequent level 1.5 trigger system, which reduces the rate below the kHz range. Both FADCTF and level 1.5 electronics make use of parallel real-time processing in Field Programmable Gate Arrays (FPGAs), while further data acquisition and event building is done by PC farms running Linux. The new readout system hardware is described and the first results obtained with cosmics are shown.

Event Builder and Level 3 at the CDF experiment

The Event Builder and Level 3 systems constitute critical components of the DAQ in the Collider Detector at Fermilab experiment. These systems are responsible for collecting data fragments from the Front End electronics, assembling the data into complete event records, reconstructing the events and forming the final trigger decision. With Tevatron Run IIa in progress, the systems have been running successfully at high throughput rates, the design utilizing scalable architecture and distributed event processing to meet the requirements. Brief descriptions of current performance in Run IIa and possible upgrade for Run IIb are presented.