High Performance FPGA-Based DMA Interface for PCIe

Abstract

We present a data communication suite developed for use in the Track Engine Trigger for the IceCube Neutrino Observatory (South Pole). It is a PCIe-based system implemented in Xilinx FPGAs with a bus master DMA on a 4-lane, generation 2 link. The suite contains DMA controller hardware IPs, test benches, Linux driver and user application for DMA and PIO transfers into memory modules and FIFOs. The Linux driver uses streaming mapping, vector write functionality, race condition controllers, page-wise memory allocation, wait queues and Message Signaled Interrupt (MSI) to facilitate high performance and throughput. The DMA which is based on the Xilinx' bus master DMA, produces measured transfer speeds up to 748MB/s (read) and 784MB/s (write) using the Xilinx VC707 Virtex-7 board. The hardware has been verified on different platforms with different FPGAs. Besides the original IceCube application, the suite has also been used for the development of readout electronics for particle physics experiments. Other applications are also considered.