NaNet: a configurable NIC bridging the gap between HPC and real-time HEP GPU computing

A Lonardo,Ilaria Neri ,Elena Pastorelli ,Laura Tosoratto ,F Simeone,Francesco Simula ,P Paolucci ,Francesca Lo Cicero ,M Fiorini,M Sozzi,Roberto Ammendola ,A Biagioni,P Vicini,Ottorino Frezza ,G Lamanna,L Pontisso,A Cotta Ramusino,M Martinelli,Davide Rossetti ,F Ameli

doi:10.1088/1748-0221/10/04/c04011

Abstract

NaNet is a FPGA-based PCIe Network Interface Card (NIC) design with GPUDirect and Remote Direct Memory Access (RDMA) capabilities featuring a configurable and extensible set of network channels. The design currently supports both standard—Gbe (1000BASE-T) and 10GbE (10Base-R)—and custom—34 Gbps APElink and 2.5 Gbps deterministic latency KM3link—channels, but its modularity allows for straightforward inclusion of other link technologies. The GPUDirect feature combined with a transport layer offload module and a data stream processing stage makes NaNet a low-latency NIC suitable for real-time GPU processing. In this paper we describe the NaNet architecture and its performances, exhibiting two of its use cases: the GPU-based low-level trigger for the RICH detector in the NA62 experiment at CERN and the on-/off-shore data transport system for the KM3NeT-IT underwater neutrino telescope.

Highlights

On Board Memory and 2.5 Gbps deterministic latency optical KM3link [5]
All data produced by Optical Modules (OM), hydrophones and instruments is collected by an electronic board contained in a vessel at the floor centre; this board called Floor Control Module (FCM) manages the communication between the on-shore laboratory and the underwater devices, distributing the timing information and signals
All floor data produced by the OMs, the hydrophones and other devices that monitor the apparatus status and environmental conditions is collected by the Floor Control Module (FCM) board, packed together and transmitted along the optical link

Summary

Introduction

On Board Memory and 2.5 Gbps deterministic latency optical KM3link [5]. NaNet includes a network stack protocol offload engine yielding a very stable communication latency, a feature making it suitable for use in real-time contexts. Paper we describe the NaNet architecture and its performances, exhibiting two of its use cases: the GPU-based low-level trigger for the RICH detector in the NA62 experiment at CERN and the on-/off-shore data transport system for the KM3NeT-IT underwater neutrino telescope.

Results

Conclusion