Newton-like Method with Diagonal Correction for Distributed Optimization

Abstract

We consider distributed optimization problems where networked nodes cooperatively minimize the sum of their locally known convex costs. A popular class of methods to solve these problems are the distributed gradient methods, which are attractive due to their inexpensive iterations, but have a drawback of slow convergence rates. This motivates the incorporation of second order information in the distributed methods, but this task is challenging: although the Hessians which arise in the algorithm design respect the sparsity of the network, their inverses are dense, hence rendering distributed implementations difficult. We overcome this challenge and propose a class of distributed Newton-like methods, which we refer to as distributed quasi-newton (DQN). The DQN family approximates the Hessian inverse by (1) splitting the Hessian into its diagonal and off-diagonal parts, (2) inverting the diagonal part, and (3) approximating the inverse of the off-diagonal part through a weighted linear function. The approxim...