History-dependent Policies Research Articles

Risk-sensitive discounted Markov decision processes with unbounded reward functions and Borel spaces

This paper attempts to study the risk-sensitive discounted discrete-time Markov decision processes in Borel spaces, in which the reward functions are allowed to be unbounded from above and from below. We find mild conditions imposed on the primitive data of the decision processes, which not only ensure the existence of a solution to the optimality equation (OE in short), but also are the generalization of the bounded reward case. Furthermore, using the OE and a novel technique, we prove the existence of an optimal policy out of the class of randomized history-dependent policies. Finally, we illustrate our results with an inventory system.

Real-Time Pricing Method for Spot Cloud Services with Non-Stationary Excess Capacity

Cloud operators face massive unused excess computing capacity with a stochastic non-stationary nature due to time-varying resource utilization with peaks and troughs. Low-priority spot (pre-emptive) cloud services with real-time pricing have been launched by many cloud operators, which allow them to maximize excess capacity revenue while keeping the right to reclaim capacities when resource scarcity occurs. However, real-time spot pricing with the non-stationarity of excess capacity has two challenges: (1) it faces incomplete peak–trough and pattern shifts in excess capacity, and (2) it suffers time and space inefficiency in optimal spot pricing policy, which needs to search over the large space of history-dependent policies in a non-stationary state. Our objective was to develop a real-time pricing method with a spot pricing scheme to maximize expected cumulative revenue under a non-stationary state. We first formulated the real-time spot pricing problem as a non-stationary Markov decision process. We then developed an improved reinforcement learning algorithm to obtain the optimal solution for real-time pricing problems. Our simulation experiments demonstrate that the profitability of the proposed reinforcement learning algorithm outperforms that of existing solutions. Our study provides both efficient optimization algorithms and valuable insights into cloud operators’ excess capacity management practices.

Fiscal and monetary stabilization policy at the zero lower bound: Consequences of limited foresight

This paper reconsiders the degree to which macroeconomic stabilization is possible when the zero lower bound is a relevant constraint on the effectiveness of conventional monetary policy, under an assumption of bounded rationality. In particular, we reconsider the potential role of countercyclical fiscal transfers as a tool of stabilization policy. Because Ricardian Equivalence no longer holds when planning horizons are finite (even when relatively long), we find that fiscal transfers can be a powerful tool to reduce the contractionary impact of an increased financial wedge during a crisis, and can even make possible complete stabilization of both aggregate output and inflation under certain circumstances, despite the binding lower bound on interest rates. However, the power of such policies depends on the degree of monetary policy accommodation. We also show that a higher level of welfare is generally possible if both monetary and fiscal authorities commit themselves to history-dependent policies in the period after the financial disturbance that causes the lower bound to bind has dissipated.

Multiconstrained Finite-Horizon Piecewise Deterministic Markov Decision Processes with Unbounded Transition Rates

This paper studies a multiconstrained problem for piecewise deterministic Markov decision processes (PDMDPs) with unbounded cost and transition rates. The goal is to minimize one type of expected finite-horizon cost over history-dependent policies while keeping some other types of expected finite-horizon costs lower than some tolerable bounds. Using the Dynkin formula for the PDMDPs, we obtain an equivalent characterization of occupancy measures and express the expected finite-horizon costs in terms of occupancy measures. Under suitable assumptions, the existence of constrained-optimal policies is shown, the linear programming formulation and its dual program for the constrained problem are derived, and the strong duality between the two programs is established. An example is provided to demonstrate our results.

Fiscal and Monetary Stabilization Policy at the Zero Lower Bound: Consequences of Limited Foresight

This paper reconsiders the degree to which macroeconomic stabilization is possible when the zero lower bound is a relevant constraint on the effectiveness of conventional monetary policy, under an assumption of bounded rationality. In particular, we reconsider the potential role of countercyclical fiscal transfers as a tool of stabilization policy. Because Ricardian Equivalence no longer holds when planning horizons are finite (even when relatively long), we find that fiscal transfers can be a powerful tool to reduce the contractionary impact of an increased financial wedge during a crisis, and can even make possible complete stabilization of both aggregate output and inflation under certain circumstances, despite the binding lower bound on interest rates. However, the power of such policies depends on the degree of monetary policy accommodation. We also show that a higher level of welfare is generally possible if both monetary and fiscal authorities commit themselves to history-dependent policies in the period after the financial disturbance that causes the lower bound to bind has dissipated. Hence forward guidance continues to play an important role in increasing the effectiveness of stabilization policy. Institutional subscribers to the NBER working paper series, and residents of developing countries may download this paper without additional charge at www.nber.org.

Risk Probability Minimization Problems for Continuous-Time Markov Decision Processes on Finite Horizon

This article deals with a risk probability minimization problem for finite horizon continuous-time Markov decision processes with unbounded transition rates and history-dependent policies. Only using the assumption of nonexplosion of the controlled state process as well as the finiteness of actions available at each state, we not only establish the existence and uniqueness of a solution to the corresponding optimality equation, but also prove the existence of a risk probability optimal policy. Finally, we give two examples to illustrate our results: one example shows that the value iteration algorithm is provided for computing both the value function and an optimal risk probability policy, and the other shows the differences between the conditions in this article and those in the previous literature.

$N$-Person Nonzero-Sum Games for Continuous-Time Jump Processes With Varying Discount Factors

This paper is concerned with the first passage discounted payoff criterion for continuous-time $N$ -person nonzero-sum stochastic games in countable states and Borel action spaces, where the discount factor is nonconstant. For the game with unbounded reward functions and transition rates, the optimality is over the set of all randomized history-dependent policies. After showing the uniqueness of the solution to the Shapley equation, we give some suitable conditions imposed on the primitive data for the existence of Nash equilibria. Then, a financial system is introduced to illustrate the applications of the main result here.

Policies in Relational Contracts

We consider how a firm’s policies constrain its relational contracts. A policy is a sequence of decisions made by a principal; each decision determines how agents’ efforts affect their outputs. We consider surplus-maximizing policies in a flexible dynamic moral hazard problem between a principal and several agents with unrestricted vertical transfers and no commitment. If agents cannot coordinate to punish the principal following a deviation, then the principal might optimally implement dynamically inefficient, history-dependent policies to credibly reward high-performing agents. We develop conditions under which such backward-looking policies are surplus-maximizing and illustrate how they influence promotions, hiring, and performance. (JEL D21, D82, D86, M51)

Finite-horizon piecewise deterministic Markov decision processes with unbounded transition rates

ABSTRACTThis paper is concerned with the problem of minimizing the expected finite-horizon cost for piecewise deterministic Markov decision processes. The transition rates may be unbounded, and the cost functions are allowed to be unbounded from above and from below. The optimality is over the general history-dependent policies, where the control is continuously acting in time. The infinitesimal approach is employed to establish the associated Hamilton-Jacobi-Bellman equation, via which the existence of optimal policies is proved. An example is provided to verify all the assumptions proposed.

Minimizing Maximum Regret in Commitment Constrained Sequential Decision Making

In cooperative multiagent planning, it can often be beneficial for an agent to make commitments about aspects of its behavior to others, allowing them in turn to plan their own behaviors without taking the agent's detailed behavior into account. Extending previous work in the Bayesian setting, we consider instead a worst-case setting in which the agent has a set of possible environments (MDPs) it could be in, and develop a commitment semantics that allows for probabilistic guarantees on the agent's behavior in any of the environments it could end up facing. Crucially, an agent receives observations (of reward and state transitions) that allow it to potentially eliminate possible environments and thus obtain higher utility by adapting its policy to the history of observations. We develop algorithms and provide theory and some preliminary empirical results showing that they ensure an agent meets its commitments with history-dependent policies while minimizing maximum regret over the possible environments.

Finite-horizon optimality for continuous-time Markov decision processes with unbounded transition rates

In this paper we focus on the finite-horizon optimality for denumerable continuous-time Markov decision processes, in which the transition and reward/cost rates are allowed to be unbounded, and the optimality is over the class of all randomized history-dependent policies. Under mild reasonable conditions, we first establish the existence of a solution to the finite-horizon optimality equation by designing a technique of approximations from the bounded transition rates to unbounded ones. Then we prove the existence of ε (≥ 0)-optimal Markov policies and verify that the value function is the unique solution to the optimality equation by establishing the analog of the Itô-Dynkin formula. Finally, we provide an example in which the transition rates and the value function are all unbounded and, thus, obtain solutions to some of the unsolved problems by Yushkevich (1978).

Finite-horizon optimality for continuous-time Markov decision processes with unbounded transition rates

In this paper we focus on the finite-horizon optimality for denumerable continuous-time Markov decision processes, in which the transition and reward/cost rates are allowed to be unbounded, and the optimality is over the class of all randomized history-dependent policies. Under mild reasonable conditions, we first establish the existence of a solution to the finite-horizon optimality equation by designing a technique of approximations from the bounded transition rates to unbounded ones. Then we prove the existence of ε (≥ 0)-optimal Markov policies and verify that the value function is the unique solution to the optimality equation by establishing the analog of the Itô-Dynkin formula. Finally, we provide an example in which the transition rates and the value function are all unbounded and, thus, obtain solutions to some of the unsolved problems by Yushkevich (1978).

First Passage Optimality for Continuous-Time Markov Decision Processes With Varying Discount Factors and History-Dependent Policies

This paper is an attempt to study the first passage optimality criterion for continuous-time Markov decision processes with state-dependent discount factors and history-dependent policies. The state space is denumerable, the action space is a Borel space, and the transition and reward rates are unbounded. Under suitable conditions, we show the existence of a deterministic stationary optimal policy, establish the Bellman (optimality) equation, to which the value function is the unique solution, and give the value and policy iteration algorithms for solving (at least approximating) the value function and an optimal policy. Furthermore, we give examples about reliability and controlled birth processes with killing to illustrate the potential applications of the results obtained here, and also to show the difference between the main results in this paper and those in the previous literature.

Absorbing Continuous-Time Markov Decision Processes with Total Cost Criteria

In this paper we study absorbing continuous-time Markov decision processes in Polish state spaces with unbounded transition and cost rates, and history-dependent policies. The performance measure is the expected total undiscounted costs. For the unconstrained problem, we show the existence of a deterministic stationary optimal policy, whereas, for the constrained problems withNconstraints, we show the existence of a mixed stationary optimal policy, where the mixture is over no more thanN+1 deterministic stationary policies. Furthermore, the strong duality result is obtained for the associated linear programs.

Absorbing Continuous-Time Markov Decision Processes with Total Cost Criteria

In this paper we study absorbing continuous-time Markov decision processes in Polish state spaces with unbounded transition and cost rates, and history-dependent policies. The performance measure is the expected total undiscounted costs. For the unconstrained problem, we show the existence of a deterministic stationary optimal policy, whereas, for the constrained problems with N constraints, we show the existence of a mixed stationary optimal policy, where the mixture is over no more than N+1 deterministic stationary policies. Furthermore, the strong duality result is obtained for the associated linear programs.

Discounted continuous-time Markov decision processes with unbounded rates and randomized history-dependent policies: the dynamic programming approach

This paper deals with a continuous-time Markov decision process in Borel state and action spaces and with unbounded transition rates. Under history-dependent policies, the controlled process may not be Markov. The main contribution is that for such non-Markov processes we establish the Dynkin formula, which plays important roles in establishing optimality results for continuous-time Markov decision processes. We further illustrate this by showing, for a discounted continuous-time Markov decision process, the existence of a deterministic stationary optimal policy (out of the class of history-dependent policies) and characterizing the value function through the Bellman equation.

The Transformation Method for Continuous-Time Markov Decision Processes

In this paper, we show that a discounted continuous-time Markov decision process in Borel spaces with randomized history-dependent policies, arbitrarily unbounded transition rates and a non-negative reward rate is equivalent to a discrete-time Markov decision process. Based on a completely new proof, which does not involve Kolmogorov’s forward equation, it is shown that the value function for both models is given by the minimal non-negative solution to the same Bellman equation. A verifiable necessary and sufficient condition for the finiteness of this value function is given, which induces a new condition for the non-explosion of the underlying controlled process.

Linear Programming and Constrained Average Optimality for General Continuous-Time Markov Decision Processes in History-Dependent Policies

This paper attempts to study the constrained average optimality for continuous-time Markov decision processes in the class of randomized history-dependent policies. The states and actions are in general Polish spaces, and the transition rates are allowed to be unbounded. The optimality criterion to be optimized is expected average costs, multiple constraints are imposed on similar expected average costs, and all costs may be unbounded from above and from below. Under suitable conditions, we first show the existence of a constrained optimal policy by improving the concept of a stable policy in the previous literature and using the analogue of the forward Kolmogorov equation. Then, we develop a linear program (LP), which is equivalent to the constrained optimality problem and is used to obtain a constrained optimal policy. By introducing suitable operators and conditions, we further establish the dual program (DP) of the LP, show that the LP and DP are solvable, and show that there is no duality gap between them. Finally, we use a cash flow model and a controlled birth and death system to illustrate the applications of our main results.

Finite horizon semi-Markov decision processes with application to maintenance systems

This paper investigates finite horizon semi-Markov decision processes with denumerable states. The optimality is over the class of all randomized history-dependent policies which include states and also planning horizons, and the cost rate function is assumed to be bounded below. Under suitable conditions, we show that the value function is a minimum nonnegative solution to the optimality equation and there exists an optimal policy. Moreover, we develop an effective algorithm for computing optimal policies, derive some properties of optimal policies, and in addition, illustrate our main results with a maintenance system.

Discounted Continuous-Time Markov Decision Processes with Unbounded Rates: The Convex Analytic Approach

This paper deals with constrained discounted continuous-time Markov decision processes, also known as controlled jump Markov processes, with Borel state and action spaces. Under some conditions imposed on the primitives, allowing unbounded transition rates and unbounded (from both above and below) cost rates, first, we study the space of occupation measures. Then we reformulate the original problem as a linear program over the space of those measures and undertake the duality analysis. Finally, under some compactness-continuity conditions, we show the existence of a stationary optimal policy out of the class of randomized history-dependent policies.