Online distributed dual averaging algorithm for multi-agent bandit optimization over time-varying general directed networks

Abstract

The paper deals with online distributed convex optimization over a multi-agent network, where a swarm of agents attempts to minimize coordinately a sequence of time-changing global loss functions. At per time slot, the global loss function is decomposed into a sum of several local loss functions, each of which is available sequentially and held privately by one agent over the network, and individual agent does not possess prior knowledge of the future global loss function. We are interested in a bandit setting, where only the values of local loss functions at queried points are disclosed to individual agent. Meanwhile, we consider a general multi-agent network where the agents’ communication is represented as a sequence of time-changing unbalanced digraphs and the corresponding weight matrices are only row stochastic. We investigate two bandit scenarios including one- and two-point bandit feedback, and then design two corresponding online distributed bandit algorithms for such a class of problems by exploiting the classical dual averaging method. We show that both algorithms can achieve the sub-linear expected dynamic regret provided that the accumulative variation of the benchmark sequence grows sub-linearly with time. In particular, the bounds of the expected static regret obtained in this paper can reduce the relevant results when restricted to the centralized bandit online convex optimization. In contrast to existing algorithms with gradient feedback, numerical tests verify the competitive performances of the proposed algorithms.