Efficient Large-Scale Deep Learning Framework for Heterogeneous Multi-GPU Cluster

Abstract

This paper presents a novel Distributed Deep Learning Framework for heterogeneous multi-GPU cluster that can effectively improve overall resource utilization without sacrificing training accuracy. Specifically, we employ a hybrid aggregation approach of parameter-server and All-reduce schemes in order to address potential performance degradation problems in running deep learning applications on a heterogeneous computing system. In addition, we designed and implemented an asynchronous large mini-batch training mechanism to maintain training accuracy for asynchronous data-paralleled deep learning processing with enhanced collective communication capability based on MPI. We have successfully implemented our proposed framework on Tensorflow, and performed extensive experiments in both of homogeneous and heterogeneous computing systems. Evaluation results show that our proposed framework can improve computing performance by decreasing I/O bottlenecks, and effectively increase the resource utilization in the heterogeneous multi-GPU cluster.