On exploiting spectral properties for solving MDP with large state space

Abstract

A large number of systems are well-modeled by Markov Decision Processes (MDPs). In particular, certain wireless communication networks and biological networks admit such models. Herein, moderate complexity strategies are proposed for computing the optimal policy for a large state space with long run discounted cost MDP, by exploiting spectral properties of the probability transition matrices (PTM). Methods such as value iteration and policy iteration for such problems are computationally prohibitive for large state spaces. Reduced dimensional policy iteration can be achieved by projecting the value function on a proper subspace. However there is no clear method for determining the optimal subspace. To this end, Graph signal processing methods have the potential to provide a solution. In order to use spectral techniques, an appropriate positive semi-definite (PSD) matrix is generated from the PTM and the single stage cost vector. Low complexity computation of the value function is enabled by the bases of this dominant subspace. The proposed projections are combined with policy iteration to find the optimal policy. Finally, numerical results on a wireless system are provided to highlight the performances and trade-offs of these various algorithms, and we found that direct spectral decomposition of outer product of PTM gives us best performance in general.