Dynamic Programming Principle Research Articles

Optimal investment-consumption-insurance with partial information

We consider an optimal investment, consumption, and life insurance purchase problem for a wage earner. We treat a stochastic factor model that the mean returns of risky assets depend linearly on underlying economic factors formulated as the solutions of linear stochastic differential equations. We discuss the partial information case that the wage earner can not observe the factor process and use only past information of risky assets. Then, our problem is formulated as a stochastic control problem with partial information. Applying the dynamic programming principle, we derive a coupled system of the Hamilton–Jacobi–Bellman (HJB) equation and two backward stochastic differential equations (BSDEs), and obtain the explicit solution. Finally, we strictly prove the verification theorem, and construct the optimal investment-consumption-insurance strategy.

Nonlinear Noncausal Optimal Control of Wave Energy Converters Via Approximate Dynamic Programming

This article proposes a novel nonlinear receding horizon optimal control algorithm for wave energy converter (WECs) with nonlinear dynamics. It is well accepted that the WEC control problem is essentially a noncausal constrained optimal control problem, where the energy output can be improved by incorporating the short-term wave prediction into the control synthesis. Inspired by this fact, we suggest a new nonlinear noncausal optimal control (NNOC) for WECs based on the principle of approximate dynamic programming, which can, first, explicitly use the wave prediction to improve the energy conversion efficiency; second, handle the state and control input constraints; third, reduce the computational burden. Different to the existing linear noncausal optimal control, the derived Hamilton-Jacobi-Bellman equation for NNOC does not have an analytic solution. To tackle this problem, a critic neural network (NN) is adopted to approximate its solution in a receding horizon manor. The weights of NN are determined via a policy iteration algorithm. The resulting NNOC consists of a causal state feedback part and a noncausal feedforward part to explicit incorporate wave prediction information. Numerical simulations are provided to verify the efficacy of the proposed NNOC method.

A policy iteration algorithm for the American put option and free boundary control problems

We develop a solution method for American put options that directly employs the policy iteration principle of dynamic programming. The method iteratively improves exercise policies, obtains monotonically increasing value functions and converges quadratically under reasonable assumptions. We present a numerical implementation that exhibits these features. The same principle is also applied to obtain a monotonically improving policy iteration scheme for general free boundary optimal control problems.

Nonlinear Constrained Optimal Control of Wave Energy Converters With Adaptive Dynamic Programming

In this paper, we address the energy maximization problem of wave energy converters (WEC) subject to nonlinearities and constraints, and present an efficient online control strategy based on the principle of adaptive dynamic programming (ADP) for solving the associated Hamilton–Jacobi–Bellman equation. To solve the derived constrained nonlinear optimal control problem, a critic neural network (NN) is used to approximate the time-dependant optimal cost value and then calculate the practical suboptimal causal control action. The proposed novel WEC control strategy leads to a simplified ADP framework without involving the widely used actor NN. The significantly improved computational efficacy of the proposed control makes it attractive for its practical implementation on a WEC to achieve a reduced unit cost of energy output, which is especially important when the dynamics of a WEC are complicated and need to be described accurately by a high-order model with nonlinearities and constraints. Simulation results are provided to show the efficacy of the proposed control method.

Generalized dynamic programming principle and sparse mean-field control problems

In this paper we study optimal control problems in Wasserstein spaces, which are suitable to describe macroscopic dynamics of multi-particle systems. The dynamics is described by a parametrized continuity equation, in which the Eulerian velocity field is affine w.r.t. some variables. Our aim is to minimize a cost functional which includes a control norm, thus enforcing a control sparsity constraint. More precisely, we consider a nonlocal restriction on the total amount of control that can be used depending on the overall state of the evolving mass. We treat in details two main cases: an instantaneous constraint on the control applied to the evolving mass and a cumulative constraint, which depends also on the amount of control used in previous times. For both constraints, we prove the existence of optimal trajectories for general cost functions and that the value function is viscosity solution of a suitable Hamilton-Jacobi-Bellmann equation. Finally, we discuss an abstract Dynamic Programming Principle, providing further applications in the Appendix.

Optimal consumption and investment strategies with liquidity risk and lifetime uncertainty for Markov regime-switching jump diffusion models

In this paper, we consider the optimal consumption and investment strategies for households throughout their lifetime. Risks such as the illiquidity of assets, abrupt changes of market states, and lifetime uncertainty are considered. Taking the effects of heritage into account, investors are willing to limit their current consumption in exchange for greater wealth at their death, because they can take advantage of the higher expected returns of illiquid assets. Further, we model the liquidity risks in an illiquid market state by introducing frozen periods with uncertain lengths, during which investors cannot continuously rebalance their portfolios between different types of assets. In liquid market, investors can continuously remix their investment portfolios. In addition, a Markov regime-switching process is introduced to describe the changes in the market’s states. Jumps, classified as either moderate or severe, are jointly investigated with liquidity risks. Explicit forms of the optimal consumption and investment strategies are developed using the dynamic programming principle. Markov chain approximation methods are adopted to obtain the value function. Numerical examples demonstrate that the liquidity of assets and market states have significant effects on optimal consumption and investment strategies in various scenarios.

A regular equilibrium solves the extended HJB system

Control problems not admitting the dynamic programming principle are known as time-inconsistent. The game-theoretic approach is to interpret such problems as intrapersonal dynamic games and look for subgame perfect Nash equilibria. A fundamental result of time-inconsistent stochastic control is a verification theorem saying that solving the extended HJB system is a sufficient condition for equilibrium. We show that solving the extended HJB system is a necessary condition for equilibrium, under regularity assumptions. The controlled process is a general Itô diffusion.

Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactions

Establishing the existence of Nash equilibria for partially observed stochastic dynamic games is known to be quite challenging, with the difficulties stemming from the noisy nature of the measurements available to individual players (agents) and the decentralized nature of this information. When the number of players is sufficiently large and the interactions among agents is of the mean-field type, one way to overcome this challenge is to investigate the infinite-population limit of the problem, which leads to a mean-field game. In this paper, we consider discrete-time partially observed mean-field games with infinite-horizon discounted-cost criteria. Using the technique of converting the original partially observed stochastic control problem to a fully observed one on the belief space and the dynamic programming principle, we establish the existence of Nash equilibria for these game models under very mild technical conditions. Then, we show that the mean-field equilibrium policy, when adopted by each agent, forms an approximate Nash equilibrium for games with sufficiently many agents.

Maximizing expected terminal utility of an insurer with high gain tax by investment and reinsurance

This paper investigates optimal investment and proportional reinsurance policies for an insurer who subjects to pay high gain tax. The surplus process of the insurer and the return process of the financial market are both modulated by the external macroeconomic environment. The dynamic of the external macroeconomic environment is specified by a Markov chain with finite states. Once the insurer’s accumulated profits attain a new maximum, they have to pay high gain tax. The objective of the insurer is to maximize the expected terminal utility by investment and reinsurance. The controlled wealth process of the insurer turned out to be a controlled jump diffusion process with reflections and Markov regime switching. By the weak dynamic programming principle (WDPP), we prove that the value function is the unique viscosity solution to the coupled Hamilton-Jacob-Bellman (HJB) equations with first derivative boundary constraints. By the Markov chain approximating method for the HJB equations, we construct a numerical scheme for approximating the viscosity solution to the coupled HJB equations. Two numerical examples are presented to illustrate the impact of both high gain tax and regime switching on the optimal policies.

Pricing Basket Weather Derivatives on Rainfall and Temperature Processes

This paper follows an incomplete market pricing approach to analyze the evaluation of weather derivatives and the viability of a weather derivatives market in terms of hedging. A utility indifference method is developed for the specification of indifference prices for the seller and buyer of a basket of weather derivatives written on rainfall and temperature. The agent’s risk preference is described by an exponential utility function and the prices are derived by dynamic programming principles and corresponding Hamilton Jacobi-Bellman equations from the stochastic optimal control problems. It is found the indifference measure is equal to the physical measure as there is no correlation between the capital market and weather. The fair price of the derivative should be greater than the seller’s indifference price and less than the buyer’s indifference price for market viability and no arbitrage opportunities.

Stochastic Optimal Control Problem with Obstacle Constraints in Sublinear Expectation Framework

In this paper, we consider a stochastic optimal control problem, in which the cost function is defined through a reflected backward stochastic differential equation in sublinear expectation framework. Besides, we study the regularity of the value function and establish the dynamic programming principle. Moreover, we prove that the value function is the unique viscosity solution of the related Hamilton–Jacobi–Bellman–Isaac equation.

A Framework for the Dynamic Programming Principle and Martingale-Generated Control Correspondences

We construct an abstract framework in which the dynamic programming principle (DPP) can be readily proven. It encompasses a broad range of common stochastic control problems in the weak formulation, and deals with problems in the “martingale formulation” with particular ease. We give two illustrations; first, we establish the DPP for general controlled diffusions and show that their value functions are viscosity solutions of the associated Hamilton–Jacobi–Bellman equations under minimal conditions. After that, we show how to treat singular control on the example of the classical monotone-follower problem.

Random walks and random tug of war in the Heisenberg group

We study mean value properties of $$\mathbf{p }$$ -harmonic functions on the first Heisenberg group $${\mathbb {H}}$$ , in connection to the dynamic programming principles of certain stochastic processes. We implement the approach of Peres and Sheffield (Duke Math J 145(1):91–120, 2008) to provide a game-theoretical interpretation of the sub-elliptic $$\mathbf{p }$$ -Laplacian; and of Manfredi et al. (Proc Am Math Soc 138(3):881–889, 2010) to characterize its viscosity solutions via asymptotic mean value expansions.

A Two-Level MPC for Energy Management Including Velocity Control of Hybrid Electric Vehicles

Todays cruise control systems try to keep a constant speed given by the driver without regarding the energy consumption. There is, however, possibilities to save energy by choosing the optimal velocity is neglected. Solving the underlying control problem for long distances with an algorithm that runs on a vehicle control unit is not straightforward, especially when it comes to hybrid electric vehicles (HEV). In order to overcome the computational burden, this paper presents a two-level model predictive control approach. It uses a sequential quadratic program (SQP) that comprises the total travel distance computing a target set for a lower layer that uses discrete state-space dynamic programming and Pontryagin's maximum principle to compute the optimal control for a short horizon. The computational efficiency is shown by a case study for an HEV passenger car with parallel powertrain, which reveals a cost advantage of up to 39% compared to a benchmark solution that exactly follows the velocity of the SQP while both approaches yield the same travel time.

Affine processes under parameter uncertainty

We develop a one-dimensional notion of affine processes under parameter uncertainty, which we call nonlinear affine processes. This is done as follows: given a set Θ of parameters for the process, we construct a corresponding nonlinear expectation on the path space of continuous processes. By a general dynamic programming principle, we link this nonlinear expectation to a variational form of the Kolmogorov equation, where the generator of a single affine process is replaced by the supremum over all corresponding generators of affine processes with parameters in Θ. This nonlinear affine process yields a tractable model for Knightian uncertainty, especially for modelling interest rates under ambiguity.We then develop an appropriate Itô formula, the respective term-structure equations, and study the nonlinear versions of the Vasiček and the Cox–Ingersoll–Ross (CIR) model. Thereafter, we introduce the nonlinear Vasiček–CIR model. This model is particularly suitable for modelling interest rates when one does not want to restrict the state space a priori and hence this approach solves the modelling issue arising with negative interest rates.

Tauberian Theorem for Games with Unbounded Running Cost

We study two game families with total payoffs that are defined either as the Cesaro average (the long run average game family) or Abel average (the discounting game family) of the running costs. We study value functions for all sufficiently small discounts and for all sufficiently large planning horizons (asymptotic approach), investigate a robust strategy that provides a near-optimal total payoff in this case (uniform approach). Assuming the Dynamic Programming Principle, we prove the corresponding Tauberian theorems without requiring the boundedness of the running cost.

The viability kernel of dynamical systems with mixed constraints: A level-set approach

This paper deals with control systems with mixed (state-control) constraints and dynamics in discrete-time. We aim at providing a practical method for computing the viability kernel for such systems. We take a standard approach via optimal control theory, that is, we identify the viability kernel as the effective domain of the value function of a suitable optimal control problem. A dynamic programming principle can be established in order to compute that value function. However, from a practical point of view this may turn out in a slow scheme for such computation, because at each step the mixed constraints need to be verified. In this work we take a different path which has been inspired by the level-set approach studied for optimal control problems with (pure) state constraints. An algorithm is proposed for computing the viability kernel and some numerical examples are shown to demonstrate the proposed method.

Portfolio Optimization for Assets with Stochastic Yields and Stochastic Volatility

In this paper, we consider a stochastic portfolio optimization model for investment on a risky asset with stochastic yields and stochastic volatility. The problem is formulated as a stochastic control problem, and the goal is to choose the optimal investment and consumption controls to maximize the investor’s expected total discounted utility. The Hamilton–Jacobi–Bellman equation is derived by virtue of the dynamic programming principle, which is a second-order nonlinear equation. Using the subsolution–supersolution method, we establish the existence result of the classical solution of the equation. Finally, we verify that the solution is equal to the value function and derive and verify the optimal investment and consumption controls.

Method of terminal control in ascent segment of unmanned aerial vehicle with ballistic phase

Introduction. The solution to the problem on the centroidal motion control synthesis (guidance problem) of an unmanned aerial vehicle (UAV) with long-range capabilities in the boost phase is considered. Control condition requires optimum fuel consumption. The principle of dynamic programming considering the restrictions to the vector modulus of the thrust output is used to solve the problem. The implementation of terminal guidance requires the formation of control as a function of the object state at the end of the ascent phase. The attainment of these boundary conditions determines the further transition to the ballistic flight phase.Materials and Methods. Bellman’s principle of dynamic programming is the most reasonable from the point of view of the implementability of the computationally efficient on-board algorithms and the solution to the problems in the form of synthesis. With natural scarcity of thrust and energy resources on board, this principle enables to obtain solutions free from the switching functions. In this case, the optimal control is a smooth function (without derivative discontinuity) of the current and final parameters of the UAV.Research Results. A new algorithmic method for the synthesis of terminal motion control is developed. Its difference is that the UAV movement control in the ascent phase is formed by the function of the motion actual and terminal parameters. This ensures movement along an energetically optimal trajectory into the given region of space. The problem solution results enable to build closed terminal guidance algorithms for the boost phase of the UAV trajectory with long-range capabilities. Such algorithms have good convergence and injection accuracy due to the prediction of parameters during the flight at a shorter time interval.Discussion and Conclusions. The most preferred is the principle of dynamic programming. It should be used when solving the problem on the centroidal motion control synthesis (guidance problem) of the UAV with long-range capabilities in the boost phase.

Random vibration control for multi-degree-of-freedom mechanical systems with soft actuators

Soft robotics include soft actuators and possess the capacity of large deformation and environmental compatibility. Weak environmental disturbances may deteriorate the operating performance of soft robotics due to the low stiffness of soft actuators. This manuscript investigates multi-degree-of-freedom nonlinear mechanical systems with dielectric elastomer actuators and establishes the bounded/unbounded optimal control strategies to suppress the random vibration around the equilibrium position by adjusting the imposed voltage in real time. First, the constitutive relation of a plan-type dielectric elastomer actuator and then the vibrating equation of the multi-degree-of-freedom nonlinear system around the equilibrium position are derived successively. The bounded/unbounded optimal control problems are then established by adopting the corresponding performance indexes. The bounded/unbounded optimal control strategies are then derived by combining the stochastic averaging technique and stochastic dynamic programming principle. Numerical results on a two-degree-of-freedom nonlinear system illustrate the application and efficacy of the proposed optimal control strategies.