NDP Methods for Multi-chain MDPs

Abstract

Simulation optimization techniques are discussed for multichain Markov decision processes (MDPs) by the learning of performance potentials. Different from ergodic or unichain models, where a single sample path suffices to be used for the learning of potentials, under a multichain case, there are more than one recurrent classes for the underlying Markov chain, therefore the sample path has to be restarted often so as not to circulate only in one recurrent class. Similar to unichain models, temporal difference (TD) learning algorithms can also be developed for learning potentials. In addition, by representing the estimates of potentials via a neural network, one neuro-dynamic programming (NDP) method, i.e., the critic algorithm, is derived as what has been supposed for unichain models. The obtained results are also applicable for general multichain semi-Markov decision processes (SMDPs), and we use a numerical example to illustrate the extension.