Linear convergence of decentralized estimation for statistical estimation using gradient method

Abstract

There has been a growing interest in solving consensus optimization problems in a multi-agent system. It is known that there is an exactness-speed dilemma for decentralized optimization for the naive gradient method, where either we have fast linear convergence to a neighborhood of size O(η) using a fixed constant step size η, or slow convergence to the exact optimizer with a decreasing step size. We reinvestigate a special case of this problem from a statistical point of view, where each agent’s local function is a sample estimate of a common population function. It is shown that when considering the convergence to the population target instead of to the exact consensual solution, the convergence is linear up to the statistical accuracy of the problem, using a step size only depending on the network characteristics instead of the sample size N. The objective function is not required to be strongly convex (but their statistical limit is), which allows for example the absolute deviation function which is used in median regression. Such results are in stark contrast to the optimization literature, where strong convexity of the objective function is necessary for linear convergence.