Computationally efficient algorithms for on-line optimization of markov decision processes

Abstract

Iterative algorithms are proposed for on-line optimization of finite state Markov decision processes. The optimization criterion is the minimization of long-run average cost. The transition probabilities of the underlying Markov chain are assumed to depend on certain unknown parameters. The algorithms estimate the unknown para-meters using a strongly consistent estimator, and use this estimate in place of the true parameter to generate a sequence of control policies which converge to an optimal policy. The main feature of the algorithms is that the updating step does not have to be carried out in all states at each iteration. At each iteration, the updating step may be performed in any number of states—as few as one—as long as all states are updated “often enough” as the number of iterations increases; in this sense the algorithms are asynchronous. The asynchronous nature of our algorithms allows us to update the costly states more often. As will be illustrated through simulation results, the asynchronism of our algorithms makes them feasible for on-line optimization.