The Role of Discount Factor in Risk Sensitive Markov Decision Processes

Abstract

Markov Decision Processes (MDPs) have long been the framework for modeling optimal decisions in stochastic environments. Although less known, Risk Sensitive Markov Decision Processes (RSMDP) extend MDPs by allowing arbitrary risk attitude. However, not every environment is well-defined in MDPs and RSMDPs and both versions make use of discount in costs to turn every problem well-defined, because of the exponential grow in RSMDPs, the problem of a well-defined problem is even harder. Here, we show that the use of discount in costs: (i) in MDPs induces a risk-prone attitude in MDPs, and (ii) in MDPs, hinders risk-averse attitude for some scenarios.