The vanishing discount approach in Markov chains with risk-sensitive criteria

Abstract

In this paper stochastic dynamic systems are studied, modeled by a countable state space Markov cost/reward chain, satisfying a Lyapunov-type stability condition. For an infinite planning horizon, risk-sensitive (exponential) discounted and average cost criteria are considered. The main contribution is the development of a vanishing discount approach to relate the discounted criterion problem with the average criterion one, as the discount factor increases to one, i.e., no discounting. In comparison to the well-established risk-neutral case, our results are novel and reveal several fundamental and surprising differences. Other contributions made include the use of convex analytic arguments to obtain appropriately convergent sequences and a verification theorem for the case of unbounded solutions to the average cost Poisson equation arising in the risk-sensitive case. Also of importance is the fact that our developments are very much self-contained and employ only basic probabilistic and analysis principles.