Dynamic Programming Operator Research Articles

Optimization of Generalized Certainty Equivalents on the Finite Horizon

This paper addresses some open issues in optimization of generic certainty equivalents. Such equivalents have been modelled using increasing functionals of the discounted sums of the per-stage unbounded-above cost or reward functions defined on the paths of the underlying controlled Markov chain on general state spaces which models the random dynamics of the system. Examples of such functionals include logarithmic and power utilities as well as the robust Risk-Sensitive preferences among others. The critical results that were obtained were the solutions of this problem for generic unbounded-above per-stage cost minimization and for per stage reward maximization, both satisfying a w-growth (hence unbounded) condition in the nite horizon setup. In the process, we establish certain nontrivial closure properties of the dynamic programming operators. In addition, we provide a real-life example from Portfolio Consumption.

Optimal control of admission in service in a queue with impatience and setup costs

We consider a single server queue in continuous time, in which customers must be served before some limit sojourn time of exponential distribution. Customers who are not served before this limit leave the system: they are impatient. The fact of serving customers and the fact of losing them due to impatience induce costs. The fact of holding them in the queue also induces a constant cost per customer and per unit time. The purpose is to decide whether to serve customers or to keep the server idle, so as to minimize costs. We use a Markov Decision Process with infinite horizon and discounted cost. Since the standard uniformization approach is not applicable here, we introduce a family of approximated uniformizable models, for which we establish the structural properties of the stochastic dynamic programming operator, and we deduce that the optimal policy is of threshold type. The threshold is computed explicitly. We then pass to the limit to show that this threshold policy is also optimal in the original model and we characterize the optimal policy. A particular care is given to the completeness of the proof. We also illustrate the difficulties involved in the proof with numerical examples.

A Convex Optimization Approach to Dynamic Programming in Continuous State and Action Spaces

In this paper, a convex optimization-based method is proposed for numerically solving dynamic programs in continuous state and action spaces. The key idea is to approximate the output of the Bellman operator at a particular state by the optimal value of a convex program. The approximate Bellman operator has a computational advantage because it involves a convex optimization problem in the case of control-affine systems and convex costs. Using this feature, we propose a simple dynamic programming algorithm to evaluate the approximate value function at pre-specified grid points by solving convex optimization problems in each iteration. We show that the proposed method approximates the optimal value function with a uniform convergence property in the case of convex optimal value functions. We also propose an interpolation-free design method for a control policy, of which performance converges uniformly to the optimum as the grid resolution becomes finer. When a nonlinear control-affine system is considered, the convex optimization approach provides an approximate policy with a provable suboptimality bound. For general cases, the proposed convex formulation of dynamic programming operators can be modified as a nonconvex bi-level program, in which the inner problem is a linear program, without losing uniform convergence properties.

Optimal stopping for measure-valued piecewise deterministic Markov processes

AbstractThis paper investigates the random horizon optimal stopping problem for measure-valued piecewise deterministic Markov processes (PDMPs). This is motivated by population dynamics applications, when one wants to monitor some characteristics of the individuals in a small population. The population and its individual characteristics can be represented by a point measure. We first define a PDMP on a space of locally finite measures. Then we define a sequence of random horizon optimal stopping problems for such processes. We prove that the value function of the problems can be obtained by iterating some dynamic programming operator. Finally we prove via a simple counter-example that controlling the whole population is not equivalent to controlling a random lineage.

An accretive operator approach to ergodic zero-sum stochastic games

We study some ergodicity property of zero-sum stochastic games with a finite state space and possibly unbounded payoffs. We formulate this property in operator-theoretical terms, involving the solvability of an optimality equation for the Shapley operators (i.e., the dynamic programming operators) of a family of perturbed games. The solvability of this equation entails the existence of the uniform value, and its solutions yield uniform optimal stationary strategies. We first provide an analytical characterization of this ergodicity property, and address the generic uniqueness, up to an additive constant, of the solutions of the optimality equation. Our analysis relies on the theory of accretive mappings, which we apply to maps of the form $Id - T$ where $T$ is nonexpansive. Then, we use the results of a companion work to characterize the ergodicity of stochastic games by a geometrical condition imposed on the transition probabilities. This condition generalizes classical notion of ergodicity for finite Markov chains and Markov decision processes.

Risk-averse model predictive control

Risk-averse model predictive control (MPC) offers a control framework that allows one to account for ambiguity in the knowledge of the underlying probability distribution and unifies stochastic and worst-case MPC. In this paper we study risk-averse MPC problems for constrained nonlinear Markovian switching systems using generic cost functions, and derive Lyapunov-type risk-averse stability conditions by leveraging the properties of risk-averse dynamic programming operators. We propose a controller design procedure to design risk-averse stabilizing terminal conditions for constrained nonlinear Markovian switching systems. Lastly, we cast the resulting risk-averse optimal control problem in a favorable form which can be solved efficiently and thus deems risk-averse MPC suitable for applications.

Generic uniqueness of the bias vector of finite zero-sum stochastic games with perfect information

Mean-payoff zero-sum stochastic games can be studied by means of a nonlinear spectral problem. When the state space is finite, the latter consists in finding an eigenpair (u,λ) solution of T(u)=λe+u, where T:Rn→Rn is the Shapley (or dynamic programming) operator, λ is a scalar, e is the unit vector, and u∈Rn. The scalar λ yields the mean payoff per time unit, and the vector u, called bias, allows one to determine optimal stationary strategies in the mean-payoff game. The existence of the eigenpair (u,λ) is generally related to ergodicity conditions. A basic issue is to understand for which classes of games the bias vector is unique (up to an additive constant). In this paper, we consider perfect-information zero-sum stochastic games with finite state and action spaces, thinking of the transition payments as variable parameters, transition probabilities being fixed. We show that the bias vector, thought of as a function of the transition payments, is generically unique (up to an additive constant). The proof uses techniques of nonlinear Perron–Frobenius theory. As an application of our results, we obtain an explicit perturbation scheme allowing one to solve degenerate instances of stochastic games by policy iteration.

Effects of System Parameters on the Optimal Cost and Policy in a Class of Multidimensional Queueing Control Problems

We consider a class of Markov Decision Processes frequently employed to model queueing and inventory control problems. For these problems, we explore how changes in different system input parameters (transition rates, costs, discount rates etc.) affect the optimal cost and the optimal policy when the state space of the problem is multidimensional. To address a large class of problems, we introduce two generic dynamic programming operators to model different types of controlled events. For these operators, we derive sufficient conditions to propagate monotonicity and supermodularity properties of the value function. These properties allow to predict how changes in system input parameters affect the optimal cost and policy. Finally, we explore the case when several parameters are changed at the same time. The online appendix is available at https://doi.org/10.1287/opre.2017.1600 .

A dynamic programming operator for tour location problems applied to the covering tour problem

This paper presents an evaluation operator for single-trip vehicle routing problems where it is not necessary to visit all the nodes. Such problems are known as Tour Location Problems. The operator, called Selector, is a dynamic programming algorithm that converts a given sequence of nodes into a feasible tour. The operator returns a subsequence of this giant tour which is optimal in terms of length. The procedure is implemented in an adaptive large neighborhood search to solve a specific tour location problem: the Covering Tour Problem. This problem consists in finding a lowest-cost Hamiltonian cycle over a subset of nodes such that nodes outside the tour are within a given distance from a visited node. The metaheuristic proposed is competitive as shown by the quality of results evaluated using the output of a state-of-the-art exact algorithm.

Data-driven approximate value iteration with optimality error bound analysis

Features of the data-driven approximate value iteration (AVI) algorithm, proposed in Li et al. (2014) for dealing with the optimal stabilization problem, include that only process data is required and that the estimate of the domain of attraction for the closed-loop is enlarged. However, the controller generated by the data-driven AVI algorithm is an approximate solution for the optimal control problem. In this work, a quantitative analysis result on the error bound between the optimal cost and the cost under the designed controller is given. This error bound is determined by the approximation error of the estimation for the optimal cost and the approximation error of the controller function estimator. The first one is concretely determined by the approximation error of the data-driven dynamic programming (DP) operator to the DP operator and the approximation error of the value function estimator. These three approximation errors are zeros when the data set of the plant is sufficient and infinitely complete, and the number of samples in the interested state space is infinite. This means that the cost under the designed controller equals to the optimal cost when the number of iterations is infinite.

Stability and monotone convergence of generalised policy iteration for discrete-time linear quadratic regulations

ABSTRACTIn this paper, we analyse the convergence and stability properties of generalised policy iteration (GPI) applied to discrete-time linear quadratic regulation problems. GPI is one kind of the generalised adaptive dynamic programming methods used for solving optimal control problems, and is composed of policy evaluation and policy improvement steps. To analyse the convergence and stability of GPI, the dynamic programming (DP) operator is defined. Then, GPI and its equivalent formulas are presented based on the notation of DP operator. The convergence of the approximate value function to the exact one in policy evaluation is proven based on the equivalent formulas. Furthermore, the positive semi-definiteness, stability, and the monotone convergence (PI-mode and VI-mode convergence) of GPI are presented under certain conditions on the initial value function. The online least square method is also presented for the implementation of GPI. Finally, some numerical simulations are carried out to verify the effectiveness of GPI as well as to further investigate the convergence and stability properties.

Uniqueness of the fixed point of nonexpansive semidifferentiable maps

We consider semidifferentiable (possibly nonsmooth) maps, acting on a subset of a Banach space, that are nonexpansive either in the norm of the space or in Hilbert’s or Thompson’s metric inherited from a convex cone. We show that the global uniqueness of the fixed point of the map, as well as the geometric convergence of every orbit to this fixed point, can be inferred from the semidifferential of the map at this point. In particular, we show that the geometric convergence rate of the orbits to the fixed point can be bounded in terms of Bonsall’s nonlinear spectral radius of the semidifferential. We derive similar results concerning the uniqueness of the eigenline and the geometric convergence of the orbits to it, in the case of positively homogeneous maps acting on the interior of a cone, or of additively homogeneous maps acting on an AM-space with unit. This is motivated in particular by the analysis of dynamic programming operators (Shapley operators) of zero-sum stochastic games.

On the single-leg airline revenue management problem in continuous time

We consider the single-leg airline revenue management problem in continuous time with Poisson arrivals. Earlier work on this problem generally uses the Hamilton–Jacobi–Bellman equation to find an optimal policy whenever the value function is differentiable and is a solution to this equation. In this paper, we employ a different probabilistic approach, which does not rely on the smoothness of the value function. Instead, we use a continuous-time discrete-event dynamic programming operator to construct the value function and study its properties. A by-product of this approach is the analysis of the differentiability of the value function. We show that differentiability may break down for example with discontinuous arrival intensities. Therefore, one should exercise caution in using arguments based on the differentiability of the value function and the Hamilton–Jacobi–Bellman equation in general.

Scheduling Services in a Queuing System with Impatience and Setup Costs

We consider a single-server queue in discrete time, in which customers must be served before some limit sojourn time of geometrical distribution. A customer who is not served before this limit leaves the system: it is impatient. The service of customers, the loss due to impatience and the holding of customers in the queue induce costs. The purpose is to decide when to serve the customers so as to minimize them. We use a Markov decision process with infinite horizon and discounted cost. We establish the structural properties of the stochastic dynamic programming operator and we deduce that the optimal policy is of threshold type. In addition, we are able to compute explicitly the optimal value of this threshold in terms of the parameters of problem.

On optimal investment in a reinsurance context with a point process market model

We study an insurance model where the risk can be controlled by reinsurance and investment in the financial market. We consider a finite planning horizon where the timing of the events, namely the arrivals of a claim and the change of the price of the underlying asset(s), corresponds to a Poisson point process. The objective is the maximization of the expected total utility and this leads to a nonstandard stochastic control problem with a possibly unbounded number of discrete random time points over the given finite planning horizon. Exploiting the contraction property of an appropriate dynamic programming operator, we obtain a value-iteration type algorithm to compute the optimal value and strategy and derive its speed of convergence. Following Schäl (2004) we consider also the specific case of exponential utility functions whereby negative values of the risk process are penalized, thus combining features of ruin minimization and utility maximization. For this case we are able to derive an explicit solution. Results of numerical computations are also reported.

MDP algorithms for portfolio optimization problems in pure jump markets

We consider the problem of maximizing the expected utility of the terminal wealth of a portfolio in a continuous-time pure jump market with general utility function. This leads to an optimal control problem for piecewise deterministic Markov processes. Using an embedding procedure we solve the problem by looking at a discrete-time contracting Markov decision process. Our aim is to show that this point of view has a number of advantages, in particular as far as computational aspects are concerned. We characterize the value function as the unique fixed point of the dynamic programming operator and prove the existence of optimal portfolios. Moreover, we show that value iteration as well as Howard’s policy improvement algorithm works. Finally, we give error bounds when the utility function is approximated and when we discretize the state space. A numerical example is presented and our approach is compared to the approximating Markov chain method.

Stability and optimality of a multi-product production and storage system under demand uncertainty

This work develops a discrete event model for a multi-product multi-stage production and storage (P&S) problem subject to random demand. The intervention problem consists of three types of possible decisions made at the end of one stage, which depend on the observed demand (or lack of) for each item: (i) to proceed further with the production of the same product, (ii) to proceed with the production of another product or (iii) to halt the production. The intervention problem is formulated in terms of dynamic programming (DP) operators and each possible solution induces an homogeneous Markov chain that characterizes the dynamics. However, solving directly the DP problem is not a viable task in situations involving a moderately large number of products with many production stages, and the idea of the paper is to detach from strict optimality with monitored precision, and rely on stability. The notion of stochastic stability brought to bear requires a finite set of positive recurrent states and the paper derives necessary and sufficient conditions for a policy to induce such a set in the studied P&S problem. An approximate value iteration algorithm is proposed, which applies to the broader class of control problems described by homogeneous Markov chains that satisfy a structural condition pointed out in the paper. This procedure iterates in a finite subset of the state space, circumventing the computational burden of standard dynamic programming. To benchmark the approach, the proposed algorithm is applied to a simple two-product P&S system.

Dynamic Programming Equations for Discounted Constrained Stochastic Control

In this paper, the application of the dynamic programming approach to constrained stochastic control problems with expected value constraints is demonstrated. Specifically, two such problems are analyzed using this approach. The problems analyzed are the problem of minimizing a discounted cost infinite horizon expectation objective subject to an identically structured constraint, and the problem of minimizing a discounted cost infinite horizon minimax objective subject to a discounted expectation constraint. Using the dynamic programming approach, optimality equations, which are the chief contribution of this paper, are obtained for these problems. In particular, the dynamic programming operators for problems with expectation constraints differ significantly from those of standard dynamic programming and problems with worst-case constraints. For the discounted cost infinite horizon cases, existence and uniqueness of solutions to the dynamic programming equations are explicitly shown by using the Banach fixed point theorem to show that the corresponding dynamic programming operators are contractions. The theory developed is illustrated by numerically solving the constrained stochastic control dynamic programming equations derived for simple example problems. The example problems are based on a two-state Markov model that represents an error prone system that is to be maintained.

The optimal search for a Markovian target when the search path is constrained: the infinite-horizon case

A target moves among a finite number of cells according to a discrete-time homogeneous Markov chain. The searcher is subject to constraints on the search path, i.e., the cells available for search in the current epoch is a function of the cell searched in the previous epoch. The aim is to identify a search policy that maximizes the infinite-horizon total expected reward earned. We show the following structural results under the assumption that the target transition matrix is ergodic: 1) the optimal search policy is stationary; and 2) there exists /spl epsi/-optimal stationary policies which may be constructed by the standard value iteration algorithm in finite time. These results are obtained by showing that the dynamic programming operator associated with the search problem is an m-stage contraction mapping on a suitably defined space. An upper bound of m and the coefficient of contraction /spl alpha/ is given in terms of the transition matrix and other variables pertaining to the search problem. These bounds on m and /spl alpha/ may be used to derive bounds on suboptimal search polices constructed.

A Dynamic Programming Algorithm for the Optimal Control of Piecewise Deterministic Markov Processes

A piecewise deterministic Markov process (PDP) is a continuous time Markov process consisting of continuous, deterministic trajectories interrupted by random jumps. The trajectories may be controlled with the object of minimizing the expected costs associated with the process. A method of representing this controlled PDP as a discrete time decision process is presented, allowing the value function for the problem to be expressed as the fixed point of a dynamic programming operator. Decisions take the form of trajectory segments. The expected costs may then be minimized through a dynamic programming algorithm, rather than through the solution of the Bellman--Hamilton--Jacobi equation, assuming the trajectory segments are numerically tractable. The technique is applied to the optimal capacity expansion problem, that is, the problem of planning the construction of new production facilities to meet rising demand.