The mirror descent control algorithm for weakly regular homogeneous finite Markov chains with unknown mean losses

Abstract

We address the adaptive stochastic control problem for a discrete time system described by controlled Markov chain with finite number of states. The mirror descent randomized control algorithm on the class of controlled homogeneous finite Markov chains with unknown mean losses has been proposed and studied. Here we develop the approach represented in Nazin and Miller (2011). The main assumptions are the following: processes are independent and stationary, nonnegative random losses are almost surely bounded by a given constant, and the connectivity assumption for the controlled Markov chain holds. The uncertainty is that the mean loss matrix is unknown. The novelty of the approach is in extension of the class of controlled homogeneous finite Markov chains to the chains with connectivity assumption. The main result consists in demonstration of the asymptotical upper bound (that is asymptotic by time) and in determining the explicit constant which is weakly depending on the logarithm of the number of states.