Effects of Adaptive Wormhole Routing in Event Builder Networks

G Bauer ,Andrea Petrucci,Vincent Boyer,S Cittolin ,R Moser ,G Maron ,R Gómez-Reino ,J A Lopez Perez ,E Cano ,H Sakulin ,L Orsini ,C Schwick ,Jin Cheol Kim ,A Oh ,L Pollet ,M Pieri ,C Paus ,A Rácz ,E Meschi ,Andrea Carboni ,M Klute ,F Meijers ,I Suzuki ,J G Branson ,J Gutleber ,F Glege ,E.g Mlot ,C Jacobs ,M Ciganek ,A M Brett ,S Erhan ,D Tsirigkas ,M Gulmini ,M Sani ,K Sumorok ,S Murray ,D Gigi ,E Lipeles ,P Schieferdecker ,J Varela

doi:10.1109/tns.2008.915925

Abstract

The data acquisition system of the CMS experiment at the Large Hadron Collider features a two-stage event builder, which combines data from about 500 sources into full events at an aggregate throughput of 100 GB/s. To meet the requirements, several architectures and interconnect technologies have been quantitatively evaluated. Myrinet will be used for the communication from the underground frontend devices to the surface event building system. Gigabit Ethernet is deployed in the surface event building system. Nearly full bi-section throughput can be obtained using a custom software driver for Myrinet based on barrel shifter traffic shaping. This paper discusses the use of Myrinet dual-port network interface cards supporting channel bonding to achieve virtual 5 GBit/s links with adaptive routing to alleviate the throughput limitations associated with wormhole routing. Adaptive routing is not expected to be suitable for high-throughput event builder applications in high-energy physics. To corroborate this claim, results from the CMS event builder preseries installation at CERN are presented and the problems of wormhole routing networks are discussed.

Highlights

This paper discusses the use of Myrinet dual-port network interface cards supporting channel bonding to achieve virtual 5GBit/s links with adaptive routing to alleviate the throughput limitations associated with wormhole routing
One aspect inherent to adaptive wormhole routing becomes observable in the presence of unidirectional N!N traffic, as it is the case for event building
We have implemented a Myrinet Express based peer transport for the CMS on-line software framework (XDAQ) and we have determined the values for various event building traffic related parameters

Summary

Introduction

The RUs receive event data fragments from detector elements at a first level trigger peak rate of 100 kHz and buffer the fragments for up to one second. The expected average event size is 1 MByte (log-normal distributed), corresponding to event fragment sizes of 2 kBytes. With events of this size, the 512 ! The event manager controls the event flow It broadcasts the first level trigger information to all RUs and assigns the destination BU to each event. This message flow is routed through the event-building network

Methods

Results

Discussion

Conclusion