An Adaptive Distributed Source Coding Design for Distributed Learning

Abstract

A major bottleneck in distributed learning is the communication overhead of exchanging intermediate model update parameters between the worker nodes and the parameter server. Existing works reconstruct each model update from worker nodes and implicitly assume that the local model updates are independent over worker nodes. In distributed learning, however, the model update is an indirect multi-terminal source coding problem, also called as the CEO problem. The existing works do not leverage the redundancy in the information transmitted by different worker nodes. This paper studies the practical distributed source coding (DSC) scheme for distributed learning. Our goal is to reduce the communication cost by leveraging the correlation in the local gradients. We first propose a DSC framework, named successive Wyner-Ziv coding, for distributed learning based on quantization and Slepian-Wolf (SW) coding. The practical SW coding is implemented by low density parity check (LDPC) code when gradients statistics are known. Finally, we propose an adaptive SW coding scheme that estimates the gradient statistics based on the observed quantized gradients at the parameter server and then dynamically adjusts the LDPC codes in each iteration.