Constant Risk Sensitivity Research Articles

Discounted approximations in risk-sensitive average Markov cost chains with finite state space

This work concerns with Markov chains on a finite state space. It is supposed that a state-dependent cost is associated with each transition, and that the evolution of the system is watched by an agent with positive and constant risk-sensitivity. For a general transition matrix, the problem of approximating the risk-sensitive average criterion in terms of the risk-sensitive discounted index is studied. It is proved that, as the discount factor increases to 1, an appropriate normalization of the discounted value functions converges to the average cost, extending recent results derived under the assumption that the state space is communicating.

Discounted approximations to the risk-sensitive average cost in finite Markov chains

This work concerns with Markov chains on a finite state space, which is endowed with a cost function. The evolution of the chain is observed by an agent with constant risk-sensitivity and, assuming that the state space is a communicating class, the relation between the risk-sensitive discounted and average performance criteria is studied. It is proved that, as the discount factor increases to 1, an appropriate normalization of the discounted value function converges to the average cost. Also, it is shown that if the classical normalization used in the risk-neutral case is applied in the risk-sensitive context, then the normalized discounted value function converges to an arithmetic mean of the average cost.

A poisson equation for the risk-sensitive average cost in semi-markov chains

This work concerns semi-Markov chains evolving on a finite state space. The system development generates a cost when a transition is announced, as well as a holding cost which is incurred continuously during each sojourn time. It is assumed that these costs are paid by an observer with positive and constant risk-sensitivity, and the overall performance of the system is measured by the corresponding (long-run) risk-sensitive average cost criterion. In this framework, conditions are provided under which the average index does not depend on the initial state and is characterized in terms of a single Poisson equation.

Solutions of the average cost optimality equation for finite Markov decision chains: risk-sensitive and risk-neutral criteria

This work is concerned with controlled Markov chains with finite state and action spaces. It is assumed that the decision maker has an arbitrary but constant risk sensitivity coefficient, and that the performance of a control policy is measured by the long-run average cost criterion. Within this framework, the existence of solutions of the corresponding risk-sensitive optimality equation for arbitrary cost function is characterized in terms of communication properties of the transition law.

Necessary and sufficient conditions for a solution to the risk-sensitive Poisson equation on a finite state space

A Markov chain with finite state space endowed with a cost function is considered. The transition mechanism is stationary, the observer has a constant risk-sensitivity, and the overall performance of the chain is measured by the risk-sensitive long-run average cost. In this context, the existence of solutions of the corresponding Poisson equation for arbitrary cost function is characterized in terms of the communication properties of the transition matrix. The result in this direction establishes that the Poisson equation has a solution for each cost function if, and only if, the transition matrix has a unique recurrent class and a strong form of the Doeblin condition holds.

The Value Iteration Algorithm in Risk-Sensitive Average Markov Decision Chains with Finite State Space

This work concerns discrete-time Markov decision chains with finite state space and bounded costs. The controller has constant risk sensitivity λ, and the performance of a control policy is measured by the corresponding risk-sensitive average cost criterion. Assuming that the optimality equation has a solution, it is shown that the value iteration scheme can be implemented to obtain, in a finite number of steps, (1) an approximation to the optimal λ-sensitive average cost with an error less than a given tolerance, and (2) a stationary policy whose performance index is arbitrarily close to the optimal value. The argument used to establish these results is based on a modification of the original model, which is an extension of a transformation introduced by Schweitzer (1971) to analyze the the risk-neutral case.

Solution to the risk-sensitive average cost optimality equation in a class of Markov decision processes with finite state space

This work concerns discrete-time Markov decision processes with finite state space and bounded costs per stage. The decision maker ranks random costs via the expectation of the utility function associated to a constant risk sensitivity coefficient, and the performance of a control policy is measured by the corresponding (long-run) risk-sensitive average cost criterion. The main structural restriction on the system is the following communication assumption: For every pair of states x and y, there exists a policy π, possibly depending on x and y, such that when the system evolves under π starting at x, the probability of reaching y is positive. Within this framework, the paper establishes the existence of solutions to the optimality equation whenever the constant risk sensitivity coefficient does not exceed certain positive value.

Stationary optimal policies in a class of multichain positive dynamic programs with finite state space and risk-sensitive criterion

This work concerns Markov decision processes with finite state space and compact action sets. The decision maker is supposed to have a constant-risk sensitivity coefficient, and a control policy is graded via the risk-sensitive expected total-reward crite

Nearly optimal policies in risk-sensitive positive dynamic programming on discrete spaces

This note concerns Markov decision processes on a discrete state space. It is supposed that the reward function is nonnegative, and that the decision maker has a nonnull constant risk-sensitivity, which leads to grade random rewards via the expectation of an exponential utility function. The perfomance index is the risk-sensitive expected-total reward criterion, and the existence of approximately optimal stationary policies, in the absolute and relative senses, is studied. The main results, derived under mild conditions, extend classical theorems in risk-neutral positive dynamic programming and can be summarized as follows: Assuming that the optimal value function is finite, it is proved that (i) e-optimal stationary policies exist when the state and action spaces are both finite, and (ii) this conclusion is extended to the denumerable state space case whenever (a) the decision maker is risk-averse, and (b) the optimal value function is bounded. This latter result is a (weak) risk-sensitive version of a classical theorem formulated by Ornstein (1969).

Optimal stationary policies inrisk-sensitive dynamic programs with finite state spaceand nonnegative rewards

This work concerns controlled Markov chains with finite state space and nonnegative rewards; it is assumed that the controller has a constant risk-sensitivity, and that the performance ofa control policy is measured by a risk-sensitive expected total-rewa

Controlled Markov chains with risk-sensitive criteria: Average cost, optimality equations, and optimal solutions

We study controlled Markov chains with denumerable state space and bounded costs per stage. A (long-run) risk-sensitive average cost criterion, associated to an exponential utility function with a constant risk sensitivity coefficient, is used as a performance measure. The main assumption on the probabilistic structure of the model is that the transition law satisfies a simultaneous Doeblin condition. Working within this framework, the main results obtained can be summarized as follows: If the constant risk-sensitivity coefficient is small enough, then an associated optimality equation has a bounded solution with a constant value for the optimal risk-sensitive average cost; in addition, under further standard continuity-compactness assumptions, optimal stationary policies are obtained. However, it is also shown that the above conclusions fail to hold, in general, for large enough values of the risk-sensitivity coefficient. Our results therefore disprove previous claims on this topic. Also of importance is the fact that our developments are very much self-contained and employ only basic probabilistic and analysis principles.

Risk-Sensitive Markov Decision Processes

This paper considers the maximization of certain equivalent reward generated by a Markov decision process with constant risk sensitivity. First, value iteration is used to optimize possibly time-varying processes of finite duration. Then a policy iteration procedure is developed to find the stationary policy with highest certain equivalent gain for the infinite duration case. A simple example demonstrates both procedures.