Risk-sensitive Markov Decision Process Research Articles

Risk-Sensitive Markov Decision Processes of USV Trajectory Planning with Time-Limited Budget.

Trajectory planning plays a crucial role in ensuring the safe navigation of ships, as it involves complex decision making influenced by various factors. This paper presents a heuristic algorithm, named the Markov decision process Heuristic Algorithm (MHA), for time-optimized avoidance of Unmanned Surface Vehicles (USVs) based on a Risk-Sensitive Markov decision process model. The proposed method utilizes the Risk-Sensitive Markov decision process model to generate a set of states within the USV collision avoidance search space. These states are determined based on the reachable locations and directions considering the time cost associated with the set of actions. By incorporating an enhanced reward function and a constraint time-dependent cost function, the USV can effectively plan practical motion paths that align with its actual time constraints. Experimental results demonstrate that the MHA algorithm enables decision makers to evaluate the trade-off between the budget and the probability of achieving the goal within the given budget. Moreover, the local stochastic optimization criterion assists the agent in selecting collision avoidance paths without significantly increasing the risk of collision.

Dynamic reinsurance in discrete time minimizing the insurer's cost of capital

In the classical static optimal reinsurance problem, the cost of capital for the insurer's risk exposure determined by a monetary risk measure is minimized over the class of reinsurance treaties represented by increasing Lipschitz retained loss functions. In this paper, we consider a dynamic extension of this reinsurance problem in discrete time which can be viewed as a risk-sensitive Markov Decision Process. The model allows for both insurance claims and premium income to be stochastic and operates with general risk measures and premium principles. We derive the Bellman equation and show the existence of a Markovian optimal reinsurance policy. Under an infinite planning horizon, the model is shown to be contractive and the optimal reinsurance policy to be stationary. The results are illustrated with examples where the optimal policy can be determined explicitly.

Cautious Reinforcement Learning via Distributional Risk in the Dual Domain

We study the estimation of risk-sensitive policies in reinforcement learning (RL) problems defined by a Markov Decision Process (MDPs) whose state and action spaces are countably finite. Prior efforts are predominately afflicted by computational challenges associated with the fact that risk-sensitive MDPs are time-inconsistent, and hence modifications of Bellman's equations are required, which often do not permit efficient fixed point schemes. To ameliorate this issue, we propose a new definition of risk, which we call caution, as a penalty function added to the dual of the linear programming (LP) formulation of tabular RL. Caution quantifies the distributional risk of a policy, and is a function of the policy's long-term state occupancy measure. When the risk is a convex function of the occupancy measure, we propose to solve this problem in an online model-free manner with a stochastic variant of primal-dual method that uses Kullback-Lieber (KL) divergence as its proximal term. We establish that the sample complexity required to attain near-optimal solutions matches tight dependencies on the cardinality of the spaces, but also depends on the infinity norm of the risk measure's gradient. Moreover, when the risk is non-convex, we propose a block-coordinate augmentation of the aforementioned approach, and establish its convergence to KKT points. Experiments demonstrate that this approach improves the reliability of reward accumulation without additional computation as compared to risk-neutral LP solvers, both for convex and non-convex risks, respectively, for the cases of KL divergence to a prior, and the variance.

A Risk-Sensitive Approach for Packet Inter-Delivery Time Optimization in Networked Cyber-Physical Systems

In networked cyber-physical systems, the inter-delivery time of data packets becomes an important quantity of interest. However, providing a guarantee that the inter-delivery times of the packets are “small enough” becomes a difficult task in such systems due to the unreliable communication medium and limited network resources. We design scheduling policies that meet the inter-delivery time requirements of multiple clients connected over wireless channels. We formulate the problem as an infinite-state risk-sensitive Markov decision process, where large exceedances of inter-delivery times for different clients over their design thresholds are severely penalized. We reduce the infinite-state problem to an equivalent finite-state problem and establish the existence of a stationary optimal policy and an algorithm for computing it in a finite number of steps. However, its computational complexity makes it intractable when the number of clients is of the order of 100 or so that is found in applications such as in-vehicle networks. To design computationally efficient optimal policies, we, therefore, develop a theory based on the high reliability asymptotic scenario, in which the channel reliability probabilities are close to one. We thereby obtain an algorithm of relatively low computational complexity for determining an asymptotically optimal policy. To address the remaining case when the channels are not relatively reliable, we design index-based policies for the risk sensitive case, which extends key ideas for index policies in risk-neutral multi-armed bandit problems. Simulation results are provided to show the effectiveness of our policies.

A Turnpike Theorem For A Risk-Sensitive Markov Decision Process with Stopping

This paper concerns a Markov decision problem with an infinite planning horizon, stationary data, an exponential utility function, and no discounting, but also with the possibility of voluntary or involuntary termination (stopping). This paper establishes conditions under which the expected utility of the income received by repeated use of a well-chosen stationary control equals the limit, as n becomes large, of the optimal expected utility for the n-period problem. Tests for these (turnpike) conditions are provided, as is a linear program whose optimal solution determines such a control.

On the Optimality of Structured Policies in Countable Stage Decision Processes

Multi-stage decision processes are considered, in notation which is an outgrowth of that introduced by Denardo [Denardo, E. 1967. Contraction mappings in the theory underlying dynamic programming. SIAM Rev. 9 165–177.]. Certain Markov decision processes, stochastic games, and risk-sensitive Markov decision processes can be formulated in this notation. We identify conditions sufficient to prove that, in infinite horizon nonstationary processes, the optimal infinite horizon (present) value exists, is uniquely defined, is what is called “structured,” and can be found by solving Bellman's optimality equations: ϵ-optimal strategies exist: an optimal strategy can be found by applying Bellman's optimality criterion; and a specially identified kind of policy, called a “structured” policy is optimal in each stage. A link is thus drawn between (i) studies such as those of Blackwell [Blackwell, D. 1965. Discounted dynamic programming. Ann. Math. Stat. 36 226–235.] and Strauch [Strauch, R. 1966. Negative dynamic programming. Ann. Math. Stat. 37 871–890.], where general policies for general processes are considered, and (ii) other studies, such as those of Scarf [Scarf, H. 1963. The optimality of (S, s) policies in the dynamic inventory problem. H. Scarf, D. Gilford, M. Shelly, eds. Mathematical Methods in the Social Sciences . Stanford University Press, Stanford.] and Derman [Derman, C. 1963. On optimal replacement rules when changes of state are Markovian. R. Bellman, ed. Mathematical Optimization Techniques. University of California Press. Berkeley.] where structured policies for special processes are considered. Those familiar with dynamic programming models (e.g., inventory, queueing optimization, replacement, optimal stopping) will be well acquainted with the use of what we call structured policies and value functions. The infinite stage results are built on finite stage results. Results for the stationary infinite horizon case are also included. For an application, we provide conditions sufficient to prove that an optimal stationary strategy exists in a discounted stationary risk sensitive Markov decision process with constant risk aversion. In Porteus [Porteus, E. On the optimality of structured policies in countable stage decision processes. Research Paper No. 141, Graduate School of Business, Stanford University, 71 pp., 1973, 1974, unabridged version of present paper.], of which this is a condensation, we also (i) show how known conditions under which a Borel measurable policy is optimal in an infinite horizon, nonstationary Markov decision process, fit into our framework, and (ii) provide conditions under which a generalized (s, S) policy [Porteus, E. 1971. On the optimality of generalized (s, S) policies. Management Sci. 17 411–426.] is optimal in an infinite horizon nonstationary inventory process.

Contraction Mappings in the Theory Underlying Dynamic Programming

Contraction Mappings in the Theory Underlying Dynamic Programming