Infinite-horizon Problem Research Articles

Distributionally Robust Control of Constrained Stochastic Systems

We investigate the control of constrained stochastic linear systems when faced with limited information regarding the disturbance process, i.e., when only the first two moments of the disturbance distribution are known. We consider two types of distributionally robust constraints. In the first case, we require that the constraints hold with a given probability for all disturbance distributions sharing the known moments. These constraints are commonly referred to as distributionally robust chance constraints. In the second case, we impose conditional value-at-risk (CVaR) constraints to bound the expected constraint violation for all disturbance distributions consistent with the given moment information. Such constraints are referred to as distributionally robust CVaR constraints with second-order moment specifications. We propose a method for designing linear controllers for systems with such constraints that is both computationally tractable and practically meaningful for both finite and infinite horizon problems. We prove in the infinite horizon case that our design procedure produces the globally optimal linear output feedback controller for distributionally robust CVaR and chance constrained problems. The proposed methods are illustrated for a wind blade control design case study for which distributionally robust constraints constitute sensible design objectives.

Asymptotic problems in optimal control with a vanishing Lagrangian and unbounded data

In this paper we give a representation formula for the limit of the finite horizon problem as the horizon becomes infinite, with a nonnegative Lagrangian and unbounded data. It is related to the limit of the discounted infinite horizon problem, as the discount factor goes to zero. We give sufficient conditions to characterize the limit function as unique nonnegative solution of the associated HJB equation. We also briefly discuss the ergodic problem.

Improved Design of Nonlinear Model Predictive Controllers

One way to ensure recursive feasibility, stability and performance of Nonlinear Model Predictive Control is the combined use of a terminal region and a terminal cost. However, finding suitable combinations of the terminal cost and terminal region that guarantee closed-loop stability for nonlinear systems is in general challenging. Most existing methods are either based on the linearized system dynamics and a linear feedback, or assume that a control Lyapunov function for the system close to the origin is know. This paper proposes the use of higher order approximations of the optimal feedback and optimal cost of the infinite horizon problem via Al’brekht’Method to determine a suitable terminal region for polynomial systems. To do so, the stability conditions are reformulated in terms of a sumof-squares problem which is iteratively used to determine the terminal region. For a nonlinear chemical reactor example it is shown that the proposed approach leads to a larger terminal region and an improved performance compared to existing approaches.

Simultaneous pricing and inventory management of deteriorating perishable products

This study addresses the problem of combined pricing and inventory control in the context of perishable goods—specifically, when demand is uncertain and price-sensitive and the consumers are free to choose between new and old units, based on the relative affordability of prices. First, the problem is formulated as a periodic review problem for a product with finite arbitrary lifetime and a myopic policy, which is often followed in practice, is discussed. Analytically it is shown that the solution obtained by the policy is also the unique solution that maximizes the individual profits from the inventories at each age. Next, this study addresses a special case—that of products with two-period lifetime, which is a scenario of practical importance. It is proven that under the policy of discounts and sticky pricing, which is often practiced in the brick-and-mortar retail stores, the myopic policy is in fact the steady-state optimal policy of the infinite-horizon problem. The study concludes with a numerical example and directions for future research in the area of joint pricing and inventory management of deteriorating, perishable products.

Infinite horizon problems on stratifiable state-constraints sets

This paper deals with a state-constrained control problem. It is well known that, unless some compatibility condition between constraints and dynamics holds, the Value Function has not enough regularity, or can fail to be the unique constrained viscosity solution of a Hamilton–Jacobi–Bellman (HJB) equation. Here, we consider the case of a set of constraints having a stratified structure. Under this circumstance, the interior of this set may be empty or disconnected, and the admissible trajectories may have the only option to stay on the boundary without possible approximation in the interior of the constraints. In such situations, the classical pointing qualification hypothesis is not relevant. The discontinuous Value Function is then characterized by means of a system of HJB equations on each stratum that composes the state-constraints. This result is obtained under a local controllability assumption which is required only on the strata where some chattering phenomena could occur.

Necessity of limiting co-state arcs in Bolza-type infinite horizon problem

We investigate necessary conditions of optimality for the Bolza-type infinite horizon problem with free right end. The optimality is understood in the sense of weakly uniformly overtaking optimal control. No previous knowledge in the asymptotic behaviour of trajectories or adjoint variables is necessary. Following Seierstad’s idea, we obtain the necessary boundary condition at infinity in the form of a transversality condition for the maximum principle. Those transversality conditions may be expressed in the integral form through an Aseev–Kryazhimskii-type formulae for co-state arcs. The connection between these formulae and limiting gradients of pay-off function at infinity is identified; several conditions under which it is possible to explicitly specify the co-state arc through those Aseev–Kryazhimskii-type formulae are found. For infinite horizon problem of Bolza type, an example is given to clarify the use of the Aseev–Kryazhimskii formula as an explicit expression of the co-state arc.

NECESSARY AND SUFFICIENT CONDITIONS FOR DYNAMIC OPTIMIZATION

We characterize the necessary and sufficient conditions for optimality in discrete-time, infinite-horizon optimization problems with a state space of finite or infinite dimension. It is well known that the challenging task in this problem is to prove the necessity of the transversality condition. To do this, we follow a duality approach in an abstract linear space. Our proof resembles that of Kamihigashi (2003), but does not explicitly use results from real analysis. As an application, we formalize Sims's argument that the no-Ponzi constraint on the government budget follows from the necessity of the tranversality condition for optimal consumption.

Convergence of Solutions of Optimal Control Problems with Discounting on Large Intervals in the Regions Close to the Endpoints

In the present paper we analyze the convergence of approximate solutions of a discrete-time optimal control system with discounting. The states of this system belong to a compact metric space and no convexity (concavity) assumption is assumed. Recently it was shown that approximate solutions are determined mainly by the objective function, except in regions close to the endpoints of the time interval. In this paper we show that approximate solutions on large intervals converge to solutions of corresponding infinite horizon problems in regions close to the endpoints of the time intervals.

On Optimal Zero-Delay Coding of Vector Markov Sources

Optimal zero-delay coding (quantization) of a vector-valued Markov source driven by a noise process is considered. Using a stochastic control problem formulation, the existence and structure of optimal quantization policies are studied. For a finite-horizon problem with bounded per-stage distortion measure, the existence of an optimal zero-delay quantization policy is shown provided that the quantizers allowed are ones with convex codecells. The bounded distortion assumption is relaxed to cover cases that include the linear quadratic Gaussian problem. For the infinite horizon problem and a stationary Markov source the optimality of deterministic Markov coding policies is shown. The existence of optimal stationary Markov quantization policies is also shown provided randomization that is shared by the encoder and the decoder is allowed.

Analysis of a Simple Cost Allocation Rule for Joint Replenishment

We consider the infinite-horizon multiple retailer joint replenishment problem with first order interaction. In this model, the joint setup cost incurred by a group of retailers placing an order simultaneously consists of a group-independent major setup cost and retailer-specific minor setup costs. The goal is to determine an inventory replenishment policy that minimizes the long-run average system-wide cost. In this paper, we adopt a non-cooperative approach to study the joint replenishment game. We consider the allocation rule in which the major setup cost is split equally among the retailers who place an order together, and each retailer pays his own holding and minor setup costs. Given the preannounced allocation rule, each retailer determines his replenishment policy to minimize his own cost anticipating the other retailers' strategy. We show that a payoff dominant Nash equilibrium (N.E.) exists, and quantify the efficiency loss of the non-cooperative outcome relative to the social optimum. Although the worst-case ratio between the best decentralized outcome and the social optimum is O (sqrt{ln n}), where n is the number of retailers, numerical results suggest that the best equilibrium is near-optimal.

Transmission conditions on interfaces for Hamilton–Jacobi–Bellman equations

We establish a comparison principle for a Hamilton–Jacobi–Bellman equation, more appropriately a system, related to an infinite horizon problem in presence of an interface. Namely a low dimensional subset of the state variable space where discontinuities in controlled dynamics and costs take place. Since corresponding Hamiltonians, at least for the subsolution part, do not enjoy any semicontinuity property, the comparison argument is rather based on a separation principle of the controlled dynamics across the interface. For this, we essentially use the notion of ε-partition and minimal ε-partition for intervals of definition of an integral trajectory.

Infinite-Horizon Continuous-Time NMPC via Time Transformation

In this note, a solution method is presented for nonlinear model-predictive control of open-loop stable systems on an infinite horizon. The proposed method first reformulates the infinite-horizon continuous-time problem by a time-coordinate transformation as a finite horizon problem and computes the solution after discretization of the control variables. This method aims to ensure stability without imposing a terminal constraint set. The adaptive discretization algorithm allows an efficient and accurate solution of the infinite-horizon problem with a moderate number of discrete decision variables. The time transformation function is adapted such that the important dynamics of the system can be captured and the control variables can be discretized appropriately. An illustrative case study is presented.

Joint pricing and inventory control for additive demand models with reference effects

We study a periodic review joint inventory and pricing problem of a single item with stochastic demand subject to reference effects. The random demand is contingent on the current price and the reference price that acts a benchmark against which customers compare the price of a product. Randomness is introduced with an additive random term. The customers perceive the difference between the price and the reference price as a loss or a gain. Hence, they have different attitudes towards them, such as loss aversion, loss neutrality or loss seeking. We model the problem using safety stock as the decision variable and show that the problem can be decomposed into two subproblems for all demand models under mild conditions. Using the decomposition, we show that a steady state solution exists for the infinite horizon problem and we characterize the steady state solution. Defining the modified revenue as revenue less the production cost, we show that a state-dependent order-up-to policy is optimal for concave demand models with concave modified revenue functions and provide example demand models with absolute difference reference effects and loss-averse customers. We also show that the optimal inventory level increases with the reference price. All of our results hold for finite and infinite horizon problems.

A 4-Stated DICE: Quantitatively Addressing Uncertainty Effects in Climate Change

We introduce a version of the DICE-2007 model designed for uncertainty analysis. DICE is a wide-spread deterministic integrated assessment model of climate change. Climate change, long-term economic development, and their interactions are highly uncertain. The quantitative analysis of optimal mitigation policy under uncertainty requires a recursive dynamic programming implementation of integrated assessment models. Such implementations are subject to the curse of dimensionality. Every increase in the dimension of the state space is paid for by a combination of (exponentially) increasing processor time, lower quality of the value or policy function approximations, and reductions of the uncertainty domain. The paper promotes a state-reduced, recursive dynamic programming implementation of the DICE-2007 model. We achieve the reduction by simplifying the carbon cycle and the temperature delay equations. We compare our model’s performance and that of the DICE model to the scientific AOGCM models emulated by MAGICC 6.0 and find that our simplified model performs equally well as the original DICE model. Our implementation solves the infinite planning horizon problem in an arbitrary time step. The paper is the first to carefully analyze the quality of the value function approximation using two different types of basis functions and systematically varying the dimension of the basis. We present the closed form, continuous time approximation to the exogenous (discretely and inductively defined) processes in DICE, and we present a numerically more efficient re-normalized Bellman equation that, in addition, can disentangle risk attitude from the propensity to smooth consumption over time.

A 4-Stated Dice: Quantitatively Addressing Uncertainty Effects in Climate Change

We introduce a version of the DICE-2007 model designed for uncertainty analysis. DICE is a wide-spread deterministic integrated assessment model of climate change. Climate change, long-term economic development, and their interactions are highly uncertain. The quantitative analysis of optimal mitigation policy under uncertainty requires a recursive dynamic programming implementation of integrated assessment models. Such implementations are subject to the curse of dimensionality. Every increase in the dimension of the state space is paid for by a combination of (exponentially) increasing processor time, lower quality of the value or policy function approximations, and reductions of the uncertainty domain. The paper promotes a state reduced, recursive dynamic programming implementation of the DICE-2007 model. We achieve the reduction by simplifying the carbon cycle and the temperature delay equations. We compare our model's performance and that of the DICE model to the scientific AOGCM models emulated by MAGICC 6.0 and find that our simplified model performs equally well as the original DICE model. Our implementation solves the infinite planning horizon problem in an arbitrary time step. The paper is the first to carefully analyze the quality of the value function approximation using two different types of basis functions and systematically varying the dimension of the basis. We present the closed form, continuous time approximation to the exogenous (discretely and inductively defined) processes in DICE, and we present a numerically more efficient re-normalized Bellman equation that, in addition, can disentangle risk attitude from the propensity to smooth consumption over time.

NEAR-OPTIMAL MODIFIED BASE STOCK POLICIES FOR THE CAPACITATED INVENTORY PROBLEM WITH STOCHASTIC DEMAND AND FIXED COST

In this study, we investigate a single-item, periodic-review inventory problem where the production capacity is limited and unmet demand is backordered. We assume that customer demand in each period is a stationary, discrete random variable. Linear holding and backorder cost are charged per unit at the end of a period. In addition to the variable cost charged per unit ordered, a positive fixed ordering cost is incurred with each order given. The optimization criterion is the minimization of the expected cost per period over a planning horizon. We investigate the infinite horizon problem by modeling the problem as a discrete-time Markov chain. We propose a heuristic for the problem based on a particular solution of this stationary model, and conduct a computational study on a set of instances, providing insight on the performance of the heuristic.

Parameter Dependent Optimal Thresholds, Indifference Levels and Inverse Optimal Stopping Problems

Consider the classic infinite-horizon problem of stopping a one-dimensional diffusion to optimise between running and terminal rewards, and suppose that we are given a parametrised family of such problems. We provide a general theory of parameter dependence in infinite-horizon stopping problems for which threshold strategies are optimal. The crux of the approach is a supermodularity condition which guarantees that the family of problems is indexable by a set-valued map which we call the indifference map. This map is a natural generalisation of the allocation (Gittins) index, a classical quantity in the theory of dynamic allocation. Importantly, the notion of indexability leads to a framework for inverse optimal stopping problems.

Parameter Dependent Optimal Thresholds, Indifference Levels and Inverse Optimal Stopping Problems

Consider the classic infinite-horizon problem of stopping a one-dimensional diffusion to optimise between running and terminal rewards, and suppose that we are given a parametrised family of such problems. We provide a general theory of parameter dependence in infinite-horizon stopping problems for which threshold strategies are optimal. The crux of the approach is a supermodularity condition which guarantees that the family of problems is indexable by a set-valued map which we call the indifference map. This map is a natural generalisation of the allocation (Gittins) index, a classical quantity in the theory of dynamic allocation. Importantly, the notion of indexability leads to a framework for inverse optimal stopping problems.

The value function of an asymptotic exit-time optimal control problem

We consider a class of exit--time control problems for nonlinear systems with a nonnegative vanishing Lagrangian. In general, the associated PDE may have multiple solutions, and known regularity and stability properties do not hold. In this paper we obtain such properties and a uniqueness result under some explicit sufficient conditions. We briefly investigate also the infinite horizon problem.

Fault detection filter design for discrete-time nonlinear systems— A mixed [formula omitted] optimization

This work investigates the problem of designing a fault detection filter for discrete-time nonlinear systems using ℋ−/ℋ∞-optimization that simultaneously improves robustness against disturbances and enhances sensitivity to faults. Sufficient conditions for the solvability of this problem are provided in the form of a pair of coupled Hamilton–Jacobi inequalities. Both the finite-horizon and the infinite-horizon problems are addressed. The proposed methodology is elaborated through a couple of examples.