Achieving Globally Superlinear Convergence for Distributed Optimization with Adaptive Newton Method

Abstract

In this paper, we study the distributed optimization problem over a peer-to-peer network, where nodes optimize the sum of local objective functions via local computation and communicating with neighbors. Most existing algorithms cannot achieve globally superlinear convergence since they rely on either asymptotic consensus methods with linear convergence rates that bottleneck the global rate, or the pure Newton method that converges only locally. To this end, we introduce a finite-time set-consensus method, and then incorporate it into Polyak's adaptive Newton method, leading to our distributed adaptive Newton algorithm (DAN). Then, we propose a communication-efficient version of DAN called DAN-LA, which adopts a low-rank approximation idea to compress the Hessian and reduce the size of transmitted messages from O(p <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) to O(p), where p is the dimension of decision vectors. We show that DAN and DAN-LA can globally achieve quadratic and superlinear convergence rates, respectively. Numerical experiments are conducted to show the advantages over existing methods.