Time Structure Analysis of the LHCb DAQ Network

G Antichi,G Liu,N Neufeld,P Owezarski,M Bruyere,D H Câmpora Pérez,S Giordano,A W Moore

doi:10.1088/1742-6596/513/6/062009

Abstract

The LHCb DAQ Network is a real time high performance network, in which 350 data sources send data over a Gigabit Ethernet LAN to more than 1500 receiving nodes. The aggregated throughput of the application, called Event Building, is more than 60 Gbps. The protocol employed by LHCb makes the sending nodes transmit simultaneously portions of events to one receiving node at a time, which is selected using a credit-token scheme. The resulting traffic is very bursty and sensitive to irregularities in the temporal distribution of packet-bursts to the same destination or region of the network.In order to study the relevant properties of such a dataflow, a non-disruptive monitoring setup based on a networking capable FPGA (Netfpga) has been deployed. The Netfpga allows order of hundred nano-second precise time-stamping of packets. We study in detail the timing structure of the Event Building communication, and we identify potential effects of micro-bursts like buffer packet drops or jitter.

Highlights

In the LHCb experiment [1], particle collisions produce data at a rate of 40 MHz
We study in detail the timing structure of the Event Building communication, and we identify potential effects of micro-bursts like buffer packet drops or jitter
This data is gathered by FPGA-based readout unit boards, known as TELL1s [2], which apply a filter based on selection algorithms

Summary

Introduction

This data is gathered by FPGA-based readout unit boards, known as TELL1s [2], which apply a filter based on selection algorithms This hardware level trigger, discards uninteresting events and reduces the data rate to 1.1 MHz. The TELL1 boards pack the data and inject it into the Data Acquisition (DAQ) network, where the data is subsequently processed in software, in stages known as Event Building and High Level Trigger. The traffic flows unidirectionally through the network following a many-to-one pattern, from the TELL1s to the farm nodes, using a custom format named Multi-Event Packets (MEP). All sources send MEP datagrams to receiving farm nodes, one at a time, which are selected using a credit-token scheme. The card contains a Xilinx Virtex-II Pro Field-Programmable Gate Array

F10 S60 TOR switch x 1G

Conclusion