Dynamic Programming Principle Research Articles

Reservoir optimization and machine learning methods

Optimization of storage using neural networks is now commonly achieved by solving a single optimization problem. We first show that this approach allows solving high-dimensional storage problems, but is limited by memory issues. We propose a modification of this algorithm based on the dynamic programming principle and propose neural networks that outperform classical feedforward networks to approximate the Bellman values of the problem. Finally, we study the stochastic linear case and show that Bellman values in storage problems can be accurately approximated using conditional cuts computed by a very recent neural network proposed by the author. This new approximation method combines linear problem solving by a linear programming solver with a neural network approximation of the Bellman values.

Optimal consumption and portfolio selection for retirees with the guarantee of minimum welfare

The optimal consumption and portfolio selection for retirees who participate in a defined benefit pension plan are investigated. The retiree is unable to get full pension if exogenous shock exists. In this case, we assume that government provides the retiree with the guarantee of minimum welfare. Using the dynamic programming principle, we discuss the properties of optimal consumption and investment strategies. Numerical examples are provided to illustrate the effects of guarantee of minimum welfare and other factors on the optimal strategies. We find that the guarantee of minimum welfare has various effects on optimal consumption and investment for the retiree.

Risk-seeking insider trading with partial observation in continuous time

<abstract><p>In this paper, a continuous-time insider trading model is investigated in which an insider is risk-seeking and market makers may receive partial information on the value of a risky asset. With the help of filtering theory and dynamic programming principle, the uniqueness and existence of linear equilibrium is established. It shows that (ⅰ) as time goes by, the residual information decreases, but both the trading intensity and the market liquidity increases, and (ⅱ) with the partial observation accuracy decreasing, both the market liquidity and the residual information will increase while the trading intensity decreases. On the whole, the risk-seeking insider is eager to trade all the trading period, and for market development, it is necessary to increase the insider's risk-preference behavior appropriately.</p></abstract>

Carbon spot prices in equilibrium frameworks associated with climate change

<p style='text-indent:20px;'>At present, it is believed that the best approach to mitigate global warming is the market-based formulation of carbon emission pricing. Thus, in this paper, we work on determining the carbon spot prices in a stochastic equilibrium framework associated with climate change. Two circumstances, differentiated by whether taking carbon trading in the market, are considered. We construct optimization problems and solve them by using dynamic programming principle. The Fourier transform and its properties are fully made use of to return the explicit formulas of carbon prices. In addition, some surprising but interesting properties of the carbon prices are also found. First, the carbon prices happen jumps at the end of the abatement period. Second, the return rates of carbon prices are completely dependent on the climate elements. Finally, we present some numeric results in response to our theoretical results.</p>

Slow persuasion

What are the value and form of optimal persuasion when information can be generated only slowly? We study this question in a dynamic model in which a “sender” provides public information over time subject to a graduality constraint, and a decision maker takes an action in each period. Using a novel “viscosity” dynamic programming principle, we characterize the sender's equilibrium value function and information provision. We show that the graduality constraint inhibits information provision relative to unconstrained persuasion. The gap can be substantial, but closes as the constraint slackens. Contrary to unconstrained persuasion, less‐than‐full information may be provided even if players have aligned preferences but different prior beliefs.

Optimal investment strategy for an insurer with partial information in capital and insurance markets

<p style='text-indent:20px;'>This paper considers a framework in which an insurer determines the optimal investment strategies to maximize the expected utility of terminal wealth. We obtain the optimal investment strategies assuming that both the capital market and the insurance market are partially observable. By employing Bayesian method and filtering theory, we first transform the optimization problem with partial information into the one with complete information. We then achieve the explicit expression of the optimal investment strategy by using dynamic programming principle. In addition, we also derive the optimal investment strategies with complete information in both markets as well as partial information in either market. Finally, we compare the optimal strategies in different models and study value functions numerically to illustrate our results.</p>

Optimal contracts to a principal-agent model with a diffusion coefficient affected by firm size

We study a principal-agent model with its diffusion coefficient affected by firm size. Under the assumptions of linear production technology and exponential preferences, we obtain the explicit solutions of optimal contract of full information in a continuous-time environment. Applying martingale method, we characterize the incentive compatibility condition which is used to deal with the agent's problem. In the case of full information, the amount of optimal effort is a constant and the agent's consumption and principal's dividend are related to firm size. Through dynamic programming principle, the implementable contract in hidden actions is constructed by solving the principal's problem. When the firm size goes to zero, the effort of agent in hidden action case approaches the first-best effort level. In the hidden action case, the impact of firm size on dynamic incentives is shown and the moral hazard results in a reduction on effort.

Moving horizon estimation of vehicle state and parameters

For the active safety control of the vehicle, it is extremely important to estimate the vehicle state in real-time and accurately during the driving process. A joint state and parameter estimation method based on QR decomposition and receding horizon estimation (RHE) is proposed. Firstly, by introducing the receding horizon strategy, the authors optimized the state and parameter estimation with a fixed number of variables, which can better deal with the estimation problem of time-varying parameters. Then, based on the principle of forward dynamic programming, the calculation of arrival cost is transformed into a least square equation, which is solved by QR decomposition. At the same time, an update method of arrival cost based on QR decomposition is given. In this way, the whole receding horizon estimation method is based on the optimization, and the feedback mechanism is introduced to improve the estimation accuracy and speed. The simulation results show that the accuracy of receding horizon estimation is obviously better than that of unscented Kalman filter (UKF), and the arrival cost update method based on QR decomposition is more convenient than the traditional arrival cost update method based on error covariance estimation.

Mean field portfolio games with consumption

We study mean field portfolio games with consumption. For general market parameters, we establish a one-to-one correspondence between Nash equilibria of the game and solutions to some FBSDE, which is proved to be equivalent to some BSDE. Our approach, which is general enough to cover power, exponential and log utilities, relies on martingale optimality principle in Cheridito and Hu (Stochast Dyn 11(02n03):283–299, 2011) and Hu et al. (Ann Appl Probab 15(3):1691–1712, 2005) and dynamic programming principle in Espinosa and Touzi (Math Financ 25(2):221–257, 2015) and Frei and dos Reis (Math Financ Econ 4:161–182, 2011). When the market parameters do not depend on the Brownian paths, we get the unique Nash equilibrium in closed form. As a byproduct, when all market parameters are time-independent, we answer the question proposed in Lacker and Soret (Math Financ Econ 14(2):263–281, 2020): the strong equilibrium obtained in Lacker and Soret (Math Financ Econ 14(2):263–281, 2020) is unique in the essentially bounded space.

Continuous and impulse controls differential game in finite horizon with Nash-equilibrium and application

This paper considers a new class of deterministic finite-time horizon, two-player, zero-sum differential games (DGs) in which the maximizing player is allowed to take continuous and impulse controls whereas the minimizing player is allowed to take impulse control only. We seek to approximate the value function, and to provide a verification theorem for this class of DGs. By means of dynamic programming principle (DPP) in viscosity solution (VS) framework, we first characterize the value function as the unique VS to the related Hamilton–Jacobi-Bellman–Isaacs (HJBI) double-obstacle equation. Next, we prove that an approximate value function exists, that it is the unique solution to an approximate HJBI double-obstacle equation, and converges locally uniformly towards the value function of each player when the time discretization step goes to zero. Moreover, we provide a verification theorem which characterizes a Nash-equilibrium (NE) for the DG control problem considered. Finally, by applying our results, we derive a new continuous-time portfolio optimization model, and we provide related computational algorithms and numerical results.

Stable control algorithms for non-stationary objects based on game methods

The paper considers the issues of constructing stable control algorithms for non-stationary objects based on the concepts of game methods. According to the principle of dynamic programming, control is defined when there are or are no restrictions on control actions. A.N.Tikhonov’s regularization method is used for stable estimation of the perturbation vector using the principle of the least residual estimate, as well as the method for compiling and solving an extended regularized normal system of equations. The obtained expressions make it possible to form control algorithms for linear non-stationary objects, which are stable when the control actions are limited, taking into account the feedback channel signal. At the same time, the use of game theory methods makes it possible to optimize management decisions under the conditions of inefficiency of classical management methods under conditions of a high level of a priori uncertainty of processes in the system under consideration.

Dynamic Programming and a Verification Theorem for the Recursive Stochastic Control Problem of Jump-Diffusion Models With Random Coefficients

In this article, we consider the stochastic optimal control problem for (forward) stochastic differential equations (SDEs) with jump diffusions and random coefficients under a recursive-type objective functional captured by a backward SDE (BSDE). Due to the jump-diffusion process with random coefficients in both the constraint (forward SDE) and the recursive BSDE objective functional, the associated Hamilton–Jacobi–Bellman equation (HJBE) is an integro-type second-order nonlinear stochastic PDE driven by both Brownian motion and (compensated) Poisson process, which we call the integro-type stochastic HJBE (ISHJBE) with jump diffusions. We first prove the dynamic programming principle for the value function using the backward semigroup associated with the recursive objective functional and the precise estimates of BSDEs, by which the continuity of the value function is also shown. Then we establish a verification theorem, which provides a sufficient condition of optimality and characterizes the value function using the (stochastic) solution of the ISHJBE with jump diffusions. Under suitable assumptions, we show the existence and uniqueness of the weak solution to the ISHJBE via the Sobolev space technique. Finally, we apply the verification theorem to the general indefinite linear-quadratic problem and the utility maximization problem; for both problems, explicit optimal solutions are characterized by solving the corresponding ISHJBE with jump diffusions.

Risk-Averse Optimal Control in Continuous Time by Nesting Risk Measures

This paper extends dynamic control problems from a risk-neutral to a risk-averse setting. We establish a limit for consecutive risk-averse decision making by consistently and adequately nesting coherent risk measures. This approach provides a new perspective on multistage optimal control problems in continuous time. For the limiting case, we elaborate a new dynamic programming principle, which is risk averse, and give risk-averse Hamilton–Jacobi–Bellman equations by generalizing the infinitesimal generator. In doing so, we provide a constructive explanation of the driver g in g-expectation, a dynamic risk measure based on backward stochastic differential equations. Funding: This work was supported by Deutsche Forschungsgemeinschaft [Project-ID 416228727 – SFB 1410].

Asymptotic $$C^{1,\gamma }$$-regularity for value functions to uniformly elliptic dynamic programming principles

In this paper we prove an asymptotic \(C^{1,\gamma }\)-estimate for value functions of stochastic processes related to uniformly elliptic dynamic programming principles. As an application, this allows us to pass to the limit with a discrete gradient and then to obtain a \(C^{1,\gamma }\)-result for the corresponding limit PDE.

Wind Markov models and reliability control in wind energy conversion systems

The integration of renewable energy systems affects the entire power system by increasing uncertainty. Power system operators always have to manage uncertainty in one way or another; the fundamental principle in this regard is to guarantee reliability, i.e. balancing load variations with generation resources. This paper addresses the problem of uncertainty management or equivalently reliability control in power systems with wind generation. Our main approach is a reformulation as stochastic optimal control for a hybrid stochastic differential system, namely, by means of modelling the wind speed as a diffusion process in steady state. Accordingly, in theory, the dynamic programming principle provides the only closed-form exact solution to the problem of reliability control. But in practice, its complexity resides in the resolution of the Hamilton–Jacobi–Bellman equation. This paper explores a systematic and feasible alternative: successive approximations of the exact solution that could be referred to as sequential linear-dynamic programming (SLDP in abridged notation). The convergence result of the approximating solution brings together two key points: first, the convergence in the sense of the L2-norm; second, the decay rate of the cost over the course of the iterations until convergence. The effectiveness of the SLDP approach has been demonstrated by simulation experiments.

Deep combinatorial optimisation for optimal stopping time problems: application to swing options pricing.

A new method for stochastic control based on neural networks and using randomisation of discrete random variables is proposed and applied to optimal stopping time problems. The method models directly the policy and does not need the derivation of a dynamic programming principle nor a backward stochastic differential equation. Unlike continuous optimization where automatic differentiation is used directly, we propose a likelihood ratio method for gradient computation. Numerical tests are done on the pricing of American and swing options. The proposed algorithm succeeds in pricing high dimensional American and swing options in a reasonable computation time, which is not possible with classical algorithms.

Uncertain Stochastic Optimal Control with Jump and Its Application in a Portfolio Game

This article describes a class of jump-uncertain stochastic control systems, and derives an Itô–Liu formula with jump. We characterize an optimal control law, that satisfies the Hamilton–Jacobi–Bellman equation with jump. Then, this paper deduces the optimal portfolio game under uncertain stochastic financial markets with jump. The information of players is symmetrical. The financial market is constituted of a risk-free asset and a risky asset whose price process is subjected to the jump-uncertain stochastic Black–Scholes model. The game is formulated by two utility maximization problems, each investor tries to maximize his relative utility, which is the weighted average of terminal wealth difference between his terminal wealth and that of his competitor. Finally, the explicit expressions of equilibrium investment strategies and value functions for the constant absolute risk-averse and constant relative risk-averse utility function are derived by using the dynamic programming principle.

Stochastic Linear Quadratic Optimal Control Problem: A Reinforcement Learning Method

This article adopts a reinforcement learning (RL) method to solve infinite horizon continuous-time stochastic linear quadratic problems, where the drift and diffusion terms in the dynamics may depend on both the state and control. Based on the Bellman’s dynamic programming principle, we presented an online RL algorithm to attain optimal control with partial system information. This algorithm computes the optimal control, rather than estimates the system coefficients, and solves the related Riccati equation. It only requires local trajectory information, which significantly simplifies the calculation process. We shed light on our theoretical findings using two numerical examples.

State and Control Path-Dependent Stochastic Optimal Control With Jumps

We consider the state and control path-dependent stochastic optimal control problem for jump-diffusion models, where the dynamics and the objective functional are dependent on (current and past) paths of state and control processes. We prove the dynamic programming principle of the value functional, for which, unlike the existing literature, the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Skorohod metric</i> is necessary to maintain the separability of càdlàg (state and control) spaces. We introduce the state and control path-dependent integro-type Hamilton–Jacobi–Bellman (PIHJB) equation, which includes the Lévy measure in the corresponding nonlocal path-dependent integral operator. Then, by using the functional Itô calculus of a càdlàg path, we show the verification theorem, which constitutes the sufficient condition for optimality in terms of the solution to the PIHJB equation. We finally apply our verification theorem to the linear-quadratic optimal control problem of jump-diffusion models with delay and the control path-dependent problem, for which the explicit optimal solutions are obtained by solving the corresponding PIHJB equation.

Optimal Investment Strategy for DC Pension Plan with Deposit Loan Spread under the CEV Model

This paper is devoted to determining an optimal investment strategy for a defined-contribution (DC) pension plan with deposit loan spread under the constant elasticity of variance (CEV) model. As far as we know, few studies in the literature have taken loans into account when using the CEV model in financial market contexts. The contribution of this paper is to study the impact of deposit loan spread on DC pension investment strategy. By considering a risk-free asset, a risky asset driven by CEV model, and a loan in the financial market, we first set up the dynamic equation and the asset market model, which are instrumental in achieving the expected utility of ultimate wealth at retirement. Second, the corresponding Hamilton–Jacobi–Bellman (HJB) equation is derived by means of the dynamic programming principle. The explicit expression for the optimal investment strategy is obtained using the Legendre transform method. Finally, different parameters are selected to simulate the explicit solution, and the financial interpretation of the optimal investment strategy is provided. We find that the deposit loan spread has a great impact on the investment strategy of DC pension plans.