Hylo: Hybrid Layer-Based Optimization to Reduce Communication in Distributed Deep Learning

Abstract

In distributed deep learning training, the synchronization of gradients usually brings huge network communication overhead. Although many methods have been proposed to solve the problem, limited effectiveness has been obtained, since these methods do not fully consider the differences of diverse layers. We propose a novel hybrid layer-based optimization approach named Hylo to reduce the communication overhead. Two different strategies are designed for gradient compression of two types of layers (convolution layer and fully-connected layer). For convolution layers, only some important convolution kernels are chosen for gradient transmission. For fully-connected layers, all gradients are quantized to 2 bits with an adaptive gradient threshold. The experimental results show that Hylo brings obvious accelerations for distributed deep learning systems, while with little accuracy loss. It achieves training speedups up to 1.31\(\times \) compared to state-of-the-art works.