Communication‐Efficient Distributed Gradient Descent via Random Projection

Abstract

ABSTRACTThe gradient descent algorithm is one type of communication‐efficient algorithm in distributed computing as it only requires transmitting the gradient vectors to update parameters. However, as the dimensionality of model parameters increases, even transmitting the gradients alone can result in a significant amount of communication costs. On the other hand, many local computers in the distributed system may suffer from communication bandwidth limitations, so the transmission of the full gradients are practically prohibited. Therefore, reducing the communication cost of gradient descent algorithms, particularly the communication bandwidth, becomes an important issue. To address this problem, we propose a randomly projected gradient descent (RPGD) algorithm. The proposed algorithm consists of three main steps. First, the gradient computed by the local computers is compressed into a low‐dimensional vector using a random matrix. Then, these projected gradients are transmitted to the central computer, aggregated and broadcasted back to the local ones. Finally, the local computers recover the gradients to their original dimensions and update the parameters. By employing random projection, we can reduce the communication bandwidth requirements in distributed computing while we can also provide better privacy protection for local computers. We have provided a theoretical convergence analysis of the RPGD algorithm. Extensive numerical studies have been conducted to demonstrate the finite sample performance of the proposed method.