Distributed Markov Chain Redesign for Multiagent Decision-Making Problems

Abstract

In this article, we consider the problem of modifying in a distributed way the transition probabilities of a Markov chain over an undirected graph in order to achieve a desired limiting distribution, while minimizing the variation from the current weights. This problem setting could be used to model a (graph-based) distributed decision-making process where static agents, e.g., elements of Internet of Things (IoT) networks, are required to achieve a common objective while adapting to different operational conditions, e.g., monitoring of time-varying and spatial-varying phenomena. This could be effectively used to describe several applications settings, ranging from sensor-network-based border patrolling to IoT-based environmental precision farming. In this context, our contribution is threefold: 1) we show that, under the assumption that a global optimal solution exists, then such a solution can be computed by solving a relaxed problem, where the irreducibility and aperiodicity constraints are lifted; 2) we derive an algebraic optimality condition for the relaxed problem; and 3) we design a distributed algorithm that provably converges towards this optimality condition.