Dynamic Programming Principle Research Articles

Optimal Control of Lossy Energy Storage Systems With Nonlinear Efficiency Based on Dynamic Programming and Pontryagin's Minimum Principle

We consider energy storage systems having nonlinear efficiency functions, which are becoming increasingly important as shown in several recent works, and propose an optimal solution based on Pontryagin's minimum principle. A central challenge in such problems is the hard limits on the state variable, which restrict the use of the minimum principle. To address this challenge, we propose to include the capacity constraints in the objective function with a proper weighting constant. We show that this approach allows formulation of the problem based on the classical minimum principle, and eventually leads to an efficient optimal control strategy. The proposed solution is compared to a dynamic programming algorithm. Numeric experiments reveal that for lossless storage devices dynamic programming is beneficial, since it enables fast and accurate solutions when a low number of samples is used. However, for lossy storage devices the situation is the opposite, and the minimum principle provides faster and more accurate solutions, since its computational complexity is almost unaffected by changes in the system parameters.

Quenched Mass Transport of Particles Toward a Target

We consider the stochastic target problem of finding the collection of initial laws of a mean-field stochastic differential equation such that we can control its evolution to ensure that it reaches a prescribed set of terminal probability distributions, at a fixed time horizon. Here, laws are considered conditionally to the path of the Brownian motion that drives the system. This kind of problems is motivated by limiting behavior of interacting particles systems with applications in, for example, agricultural crop management. We establish a version of the geometric dynamic programming principle for the associated reachability sets and prove that the corresponding value function is a viscosity solution of a geometric partial differential equation. This provides a characterization of the initial masses that can be almost surely transported toward a given target, along the paths of a stochastic differential equation. Our results extend those of Soner and Touzi, Journal of the European Mathematical Society (2002) to our setting.

Generalized Risk-Sensitive Optimal Control and Hamilton–Jacobi–Bellman Equation

In this article, we consider the generalized risk-sensitive optimal control problem, where the objective functional is defined by the controlled backward stochastic differential equation (BSDE) with quadratic growth coefficient. We extend the earlier results of the risk-sensitive optimal control problem to the case of the objective functional given by the controlled BSDE. Note that the risk-neutral stochastic optimal control problem corresponds to the BSDE objective functional with linear growth coefficient, which can be viewed as a special case of the article. We obtain the generalized risk-sensitive dynamic programming principle for the value function via the backward semigroup associated with the BSDE. Then we show that the corresponding value function is a viscosity solution to the Hamilton-Jacobi-Bellman equation. Under an additional parameter condition, the viscosity solution is unique, which implies that the solution characterizes the value function. We apply the theoretical results to the risk-sensitive European option pricing problem.

Household Lifetime Strategies under a Self-Contagious Market

In this paper, we consider the optimal strategies in asset allocation, consumption, and life insurance for a household with an exogenous stochastic income under a self-contagious market which is modeled by bivariate self-exciting Hawkes jump processes. By using the Hawkes process, jump intensities of the risky asset depend on the history path of that asset. In addition to the financial risk, the household is also subject to an uncertain lifetime and a fixed retirement date. A lump-sum payment will be paid as a heritage, if the wage earner dies before the retirement date. Under the dynamic programming principle, explicit solutions of the optimal controls are obtained when asset prices follow special jump distributions. For more general cases, we apply the Feynman–Kac formula and develop an iterative numerical scheme to derive the optimal strategies. We also prove the existence and uniqueness of the solution to the fixed point equation and the convergence of an iterative numerical algorithm. Numerical examples are presented to show the effect of jump intensities on the optimal controls.

The Dynkin game with regime switching and applications to pricing game options

This paper is concerned with the Dynkin game (a zero-sum optimal stopping game). The dynamic of the system is modeled by a regime switching diffusion, in which the regime switching mechanism provides the structural changes of the random environment. The goal is to find a saddle point for the payoff functional up to one of the players exiting the game. Taking advantage of the method of penalization and the dynamic programming principle, the value function of the game problem is shown to be the unique viscosity solution to the associated variational inequalities. We also consider a financial example of pricing game option under a regime switching market. Both optimal stopping rules for the buyer and the seller and fair price of the option are numerically demonstrated in this example. All the results are markedly different from the traditional cases without regime switching.

Dynamic Pricing and Optimal Control for a Stochastic Inventory System with Non-Instantaneous Deteriorating Items and Partial Backlogging

In this paper, we consider a problem of the dynamic pricing and inventory control for non-instantaneous deteriorating items with uncertain demand, in which the demand is price-sensitive and governed by a diffusion process. Shortages and remains are permitted, and the backlogging rate is variable and dependent on the waiting time for the next replenishment. In order to maximize the expected total profit, the problem of dynamic pricing and inventory control is described as a stochastic optimal control problem. Based on the dynamic programming principle, the stochastic control model is transformed into a Hamilton-Jacobi-Bellman (HJB) equation. Then, an exact expression for the optimal dynamic pricing strategy is obtained via solving the HJB equation. Moreover, the optimal initial inventory level, the optimal selling pricing, the optimal replenishment cycle and the optimal expected total profit are achieved when the replenishment cycle starts at time 0. Finally, some numerical simulations are presented to demonstrate the analytical results, and the sensitivities analysis on system parameters are carried out to provide some suggestions for managers.

FAIR FACILITY ALLOCATION IN EMERGENCY SERVICE SYSTEM

The request of equal accessibility must be respected to some extent when dealing with problems of designing or rebuilding of emergency service systems. Not only the disutility of the average user but also the disutility of the worst situated user must be taken into consideration. Respecting this principle is called fairness of system design. Unfairness can be mitigated to a certain extent by an appropriate fair allocation of additional facilities among the centres. In this article, two criteria of collective fairness are defined in the connection with the facility allocation problem. To solve the problems, we suggest a series of fast algorithms for solving of the allocation problem. This article extends the family of the original solving techniques based on branch-and-bound principle by newly suggested techniques, which exploit either dynamic programming principle or convexity and monotony of decreasing nonlinearities in objective functions. The resulting algorithms were tested and compared performing numerical experiments with real-sized problem instances. The new proposed algorithms outperform the original approach. The suggested methods are able to solve general min-sum and min-max problems, in which a limited number of facilities should be assigned to individual members from a finite set of providers.

Optimal Reinsurance-Investment Problem under Mean-Variance Criterion with n Risky Assets

Based on the mean-variance criterion, this paper investigates the continuous-time reinsurance and investment problem. The insurer’s surplus process is assumed to follow Cramér–Lundberg model. The insurer is allowed to purchase reinsurance for reducing claim risk. The reinsurance pattern that the insurer adopts is combining proportional and excess of loss reinsurance. In addition, the insurer can invest in financial market to increase his wealth. The financial market consists of one risk-free asset and n correlated risky assets. The objective is to minimize the variance of the terminal wealth under the given expected value of the terminal wealth. By applying the principle of dynamic programming, we establish a Hamilton–Jacobi–Bellman (HJB) equation. Furthermore, we derive the explicit solutions for the optimal reinsurance-investment strategy and the corresponding efficient frontier by solving the HJB equation. Finally, numerical examples are provided to illustrate how the optimal reinsurance-investment strategy changes with model parameters.

On Dynamic Programming Principle for Stochastic Control Under Expectation Constraints

This paper studies the dynamic programming principle using the measurable selection method for stochastic control of continuous processes. The novelty of this work is to incorporate intermediate expectation constraints on the canonical space at each time t. Motivated by some financial applications, we show that several types of dynamic trading constraints can be reformulated into expectation constraints on paths of controlled state processes. Our results can therefore be employed to recover the dynamic programming principle for these optimal investment problems under dynamic constraints, possibly path-dependent, in a non-Markovian framework.

Adaptive Robust Control in Continuous-Time

We propose a continuous-time version of the adaptive robust methodology introduced in Bielecki et al. (2019). An agent solves a stochastic control problem where the underlying uncertainty follows a jump-diffusion process and the agent does not know the drift parameters of the process. The agent considers a set of alternative measures to make the control problem robust to model misspecification and employs a continuous-time estimator to learn the value of the unknown parameters to make the control problem adaptive to the arrival of new information. We use measurable selection theorems to prove the dynamic programming principle of the adaptive robust problem and show that the value function of the agent is characterised by a non-linear partial differential equation. As an example, we derive in closed-form the optimal adaptive robust strategy for an agent who acquires a large number of shares in an order-driven market and illustrates the financial performance of the execution strategy.

Hedging Longevity Risk in Defined Contribution Pension Schemes

Pension schemes all over the world are under increasing pressure to efficiently hedge the longevity risk posed by ageing populations. In this work, we study an optimal investment problem for a defined contribution pension scheme which decides to hedge the longevity risk using a mortality-linked security, typically a longevity bond. The pension scheme invests in the risky assets available in the market, including the longevity bond, by using the contributions from a representative scheme member to ensure a minimum guarantee such that the member is able to purchase a lifetime annuity upon retirement. We transform this constrained optimal investment problem into an unconstrained problem by replicating a self-financing portfolio of future contributions from the member and the minimum guarantee provided by the scheme. We solve the resulting optimisation problem using the dynamic programming principle and through a series of numerical studies reveal that the longevity risk has an important impact on the performance of investment strategies. Our results provide mathematical evidence supporting the use of mortality-linked securities for efficient hedging of the longevity risk.

Convergence of dynamic programming principles for the p-Laplacian

Abstract We provide a unified strategy to show that solutions of dynamic programming principles associated to the p-Laplacian converge to the solution of the corresponding Dirichlet problem. Our approach includes all previously known cases for continuous and discrete dynamic programming principles, provides new results, and gives a convergence proof free of probability arguments.

Simultaneous Impulse and Continuous Control of a Markov Chain in Continuous Time

We consider continuous and impulse control of a Markov chain (MC) with a finite set of states in continuous time. Continuous control determines the intensity of transitions between MC states, while transition times and their directions are random. Nevertheless, sometimes it is necessary to ensure a transition that leads to an instantaneous change in the state of the MC. Since such transitions require different influences and can produce different effects on the state of the MC, such controls can be interpreted as impulse controls. In this work, we use the martingale representation of a controllable MC and give an optimality condition, which, using the principle of dynamic programming, is reduced to a form of quasi-variational inequality. The solution to this inequality can be obtained in the form of a dynamic programming equation, which for an MC with a finite set of states reduces to a system of ordinary differential equations with one switching line. We prove a sufficient optimality condition and give examples of problems with deterministic and random impulse action.

On the dynamic representation of some time-inconsistent risk measures in a Brownian filtration

It is well-known from the work of Kupper and Schachermayer that most law-invariant risk measures are not time-consistent, and thus do not admit dynamic representations as backward stochastic differential equations. In this work we show that in a Brownian filtration the “Optimized Certainty Equivalent” risk measures of Ben-Tal and Teboulle can be computed through PDE techniques, i.e. dynamically. This can be seen as a substitute of sorts whenever they lack time consistency, and covers the cases of conditional value-at-risk and monotone mean-variance. Our method consists of focusing on the convex dual representation, which suggests an expression of the risk measure as the value of a stochastic control problem on an extended the state space. With this we can obtain a dynamic programming principle and use stochastic control techniques, along with the theory of viscosity solutions, which we must adapt to cover the present singular situation.

Feedback control of parametrized PDEs via model order reduction and dynamic programming principle

In this paper, we investigate infinite horizon optimal control problems for parametrized partial differential equations. We are interested in feedback control via dynamic programming equations which is well-known to suffer from the curse of dimensionality. Thus, we apply parametric model order reduction techniques to construct low-dimensional subspaces with suitable information on the control problem, where the dynamic programming equations can be approximated. To guarantee a low number of basis functions, we combine recent basis generation methods and parameter partitioning techniques. Furthermore, we present a novel technique to construct non-uniform grids in the reduced domain, which is based on statistical information. Finally, we discuss numerical examples to illustrate the effectiveness of the proposed methods for PDEs in two space dimensions.

Two-player zero-sum stochastic differential games with regime switching

This paper is concerned with the two-player zero-sum stochastic differential game in a regime switching model with an infinite horizon. The state of the system is characterized by a number of diffusions coupled by a continuous-time finite-state Markov chain. Based on the dynamic programming principle (DPP), the lower and upper value functions are shown to be the unique viscosity solutions of the associated lower and upper Hamilton–Jacobi–Bellman–Isaacs (HJBI) equations, respectively. Moreover, the lower and upper value functions coincide under the Isaacs’ condition, which implies that the game admits a value. All the proofs in this paper are markedly different from those for the case when there is no regime switching.

A non-zero-sum reinsurance-investment game with delay and asymmetric information

<p style='text-indent:20px;'>In this paper, we investigate a non-zero-sum stochastic differential reinsurance-investment game problem between two insurers. Both insurers can purchase proportional reinsurance and invest in a financial market that contains a risk-free asset and a risky asset. We consider the insurers' wealth processes with delay to characterize the bounded memory feature. For considering the effect of asymmetric information, we assume the insurers have access to different levels of information in the financial market. Each insurer's objective is to maximize the expected utility of its performance relative to its competitor. We derive the Hamilton-Jacobi-Bellman (HJB) equations and the general Nash equilibrium strategies associated with the control problem by applying the dynamic programming principle. For constant absolute risk aversion (CARA) insurers, the explicit Nash equilibrium strategies and the value functions are obtained. Finally, we present some numerical studies to draw economic interpretations and find the following interesting results: (1) the insurer with less information completely ignores its own risk aversion factor, but imitates the investment strategy of its competitor who has more information on the financial market, which is a manifestation of the herd effect in economics; (2) the difference between the effects of different delay weights on the strategies is related to the length of the delay time in the framework of the non-zero-sum stochastic differential game, which illustrates that insurers should rationally estimate the correlation between historical performance and future performance based on their own risk tolerance, especially when decision makers consider historical performance over a long period of time.

Stochastic maximum principle, dynamic programming principle, and their relationship for fully coupled forward-backward stochastic controlled systems

Within the framework of viscosity solution, we study the relationship between the maximum principle (MP) from M. Hu, S. Ji and X. Xue [SIAM J. Control Optim.56(2018) 4309–4335] and the dynamic programming principle (DPP) from M. Hu, S. Ji and X. Xue [SIAM J. Control Optim.57(2019) 3911–3938] for a fully coupled forward–backward stochastic controlled system (FBSCS) with a nonconvex control domain. For a fully coupled FBSCS, both the corresponding MP and the corresponding Hamilton–Jacobi–Bellman (HJB) equation combine an algebra equation respectively. With the help of a new decoupling technique, we obtain the desirable estimates for the fully coupled forward–backward variational equations and establish the relationship. Furthermore, for the smooth case, we discover the connection between the derivatives of the solution to the algebra equation and some terms in the first-order and second-order adjoint equations. Finally, we study the local case under the monotonicity conditions as from J. Li and Q. Wei [SIAM J. Control Optim.52(2014) 1622–1662] and Z. Wu [Syst. Sci. Math. Sci.11(1998) 249–259], and obtain the relationship between the MP from Z. Wu [Syst. Sci. Math. Sci.11(1998) 249–259] and the DPP from J. Li and Q. Wei [SIAM J. Control Optim.52(2014) 1622–1662].

Local Lipschitz regularity for functions satisfying a time-dependent dynamic programming principle

We prove in this article that functions satisfying a dynamic programming principle have a local interior Lipschitz type regularity. This DPP is partly motivated by the connection to the normalized parabolic $p$-Laplace operator.

Backward Stochastic Riccati Equation with Jumps Associated with Stochastic Linear Quadratic Optimal Control with Jumps and Random Coefficients

In this paper, we investigate the solvability of matrix valued backward stochastic Riccati equations with jumps (BSREJ), which are associated with a stochastic linear quadratic (SLQ) optimal control problem with random coefficients and driven by both Brownian motion and a Poisson process. By the dynamic programming principle, Doob--Meyer decomposition, and inverse flow technique, the existence and uniqueness of the solution for the BSREJ are established. The difficulties addressed in this issue not only are brought from the high nonlinearity of the generator of the BSREJ like the case driven only by Brownian motion, but also from that (i) the inverse flow of the controlled linear stochastic differential equation driven by Poisson process may not exist without additional technical conditions, and (ii) showing that the inverse matrix term involving a jump process in the generator is well-defined. Utilizing the structure of the optimal control problem, we overcome these difficulties and establish the existence of the solution. In addition, we show the construction of the optimal feedback control with the help of the Riccati equation and the relation between the solution of the Riccati equation and the value function of the SLQ problem, which also implies the uniqueness of BSREJ.