An application of simulation for large-scale Markov decision processes to a problem in telephone network routing

Abstract

This paper presents a new approach to solving large-scale Markov decision processes (MDP) which uses simulation to generate empirical estimates of the parameters for an approximating MDP model. We begin with a brief description of this simulation for model generation algorithm and discuss some theoretical concerns regarding its implementation. We then discuss the application of the approach to a problem of choosing the optimal routes for calls within a telephone network. Although an MDP approach to this problem has been shown to be effective on small problems, no optimal procedure has yet been developed for solving this problem when the network has a nontrivial number of nodes. The ability of the algorithm to handle the complexity of such larger networks is demonstrated through some experimental results, and a comparison of its relative performance with respect to several existing solution methodologies is provided.