Accelerated Primal-Dual Algorithm for Distributed Non-convex Optimization

Abstract

This paper investigates accelerating the convergence of distributed optimization algorithms on non-convex problems. We propose a distributed stochastic gradient primal-dual with fixed parameters and “powerball” (DSGPA-F-PH) method to accelerate. We show that the proposed algorithm achieves the linear speedup convergence rate <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{O}(1/\sqrt{nT})$</tex> for general smooth (possi-bly non-convex) cost functions. We demonstrate the efficiency of the algorithm through numerical experiments by training two-layer fully connected neural networks and convolutional neural networks on the MNIST dataset to compare with state-of-the-art distributed stochastic gradient descent (SGD) algorithms and centralized SGD algorithms.