Communication-Efficient Distributed Deep Learning with GPU-FPGA Heterogeneous Computing

Abstract

Distributed deep learning has an inevitable bottleneck in communication between GPUs. This bottleneck is due to the difference in the data transfer latency of internode and intra-node communication. Here, we propose a GPU-FPGA (field programmable gate array) heterogeneous computing system and a hardware-specific communication module with parameter-based computation/communication overlap. In our system, we configure a network interface card (NIC) with a data aggregation function. To reduce the load on CPUs, we developed a device driver for remote direct memory access between GPUs and FPGAs. The hardware and software can be run with TensorFlow and Horovod. For comparison, we compared our system with the conventional GPUDirect RDMA system. Results of the measurement show Allreduce latency is reduced to 1/4 by offloading Allreduce to FPGA NICs. Our hardware/software co-design can also conceal about 90% of the communication overhead and improve scalability by 20%. The end-to-end time consumed for training in distributed deep learning with ResNet-50 and ImageNet is reduced to 87.3% without any degradation in validation accuracy.