A reinforcement learning based algorithm for markov decision processes

Abstract

A variant of a recently proposed two-timescale reinforcement learning based actor-critic algorithm for infinite horizon discounted cost Markov decision processes with finite state and compact action spaces is proposed. On the faster timescale, the value function corresponding to a given stationary deterministic policy is updated and averaged while the policy itself is updated on the slower scale. The latter recursion uses the sign of the gradient estimate instead of the estimate itself. A potential advantage in the use of sign function lies in significantly reduced computation and communication overheads in applications such as congestion control in communication networks and distributed computation. Convergence analysis of the algorithm is briefly sketched and numerical experiments for a problem of congestion control are presented.