Finite-Time Analysis of Decentralized Stochastic Approximation with Applications in Multi-Agent and Multi-Task Learning

Abstract

Stochastic approximation, a data-driven approach for finding the root of an unknown operator, provides a unified framework for solving many problems in stochastic optimization and reinforcement learning. Motivated by a growing interest in multi-agent and multi-task learning, we study a decentralized variant of stochastic approximation over a network of agents, where the goal is to find the root of the aggregate of the local operators at the agents. In this method, each agent implements a local stochastic approximation using noisy samples from its operator while averaging its iterates with the ones received from its neighbors. Our main contribution is to provide a finite-time analysis of the decentralized stochastic approximation method and to characterize the impacts of the underlying communication topology between agents. Our model for the data observed at each agent is that it is sampled from a Markov process; this lack of independence makes the iterates biased and (potentially) unbounded. Under mild assumptions we show that the convergence rate of the proposed method is essentially the same as if the samples were independent, differing only by a log factor that represents the mixing time of the Markov process. Finally, we present applications of the proposed method on a number of interesting learning problems in multi-agent systems, including distributed robust system identification and decentralized Q-learning for solving multitask reinforcement learning.