Weakly Coupled Constrained Markov Decision Processes in Borel Spaces

Abstract

Consider a multi-agent stochastic control problem where the agents have decoupled system dynamics. Each agent has an associated cost function and a constraint function. The agents want to find a control strategy which minimizes their long term average cumulative cost function while keeping the long term average cumulative constraint function below a certain threshold. This problem is referred to as weakly coupled constrained Markov decision process (MDP). In this paper, we consider the problem of weakly coupled constrained MDP with Borel state and action spaces. We use the linear programming (LP) based approach of [1] to derive an occupation measure based LP to find the optimal decentralized control strategies for our problem. We show that randomized stationary policies are optimal for each agent under some assumptions on the transition kernels, cost and the constraint functions. We further consider the special case of multi-agent Linear Quadratic Gaussian (LQG) systems and show that the optimal control strategy could be obtained by solving a semi-definite program (SDP). We illustrate our results through numerical experiments.