Dynamic Programming Principle Research Articles

Optimal control of stochastic differential equations with random impulses and the Hamilton–Jacobi–Bellman equation

AbstractIn this article, we study the optimal control of stochastic differential equations with random impulses. We optimize the performance index and add the influence of random impulses to the performance index with a random compensation function. Using the idea of stochastic analysis and dynamic programming principle, a new Hamilton–Jacobi–Bellman (HJB) equation is obtained, and the existence and uniqueness of its viscosity solution are proved.

Viscosity solutions approach to finite-horizon continuous-time Markov decision process

This paper investigates the optimal control problems for the finite-horizon continuous-time Markov decision processes with delay-dependent control policies. We develop compactification methods in decision processes and show that the existence of optimal policies. Subsequently, through the dynamic programming principle of the delay-dependent control policies, the differential-difference Hamilton-Jacobi-Bellman (HJB) equation in the setting of discrete space is established. Under certain conditions, we give the comparison principle and further prove that the value function is the unique viscosity solution to this HJB equation. Based on this, we show that among the class of delay-dependent control policies, there is an optimal one which is Markovian.

Stochastic processes under parameter uncertainty

In this paper we study a family of nonlinear (conditional) expectations that can be understood as a stochastic process with uncertain parameters. We develop a general framework which can be seen as a version of the martingale problem method of Stroock and Varadhan with parameter uncertainty. To illustrate our methodology, we explain how it can be used to model nonlinear Lévy processes in the sense of Neufeld and Nutz, and we introduce the new class of stochastic partial differential equations under parameter uncertainty. Moreover, we study properties of the nonlinear expectations. We prove the dynamic programming principle, i.e., the tower property, and we establish conditions for the (strong) USCb–Feller property and a strong Markov selection principle.

Model‐free distributed optimal control for general discrete‐time linear systems using reinforcement learning

AbstractThis article proposes a novel data‐driven framework of distributed optimal consensus for discrete‐time linear multi‐agent systems under general digraphs. A fully distributed control protocol is proposed by using linear quadratic regulator approach, which is proved to be a sufficient and necessary condition for optimal control of multi‐agent systems through dynamic programming and minimum principle. Moreover, the control protocol can be constructed by using local information with the aid of the solution of the algebraic Riccati equation (ARE). Based on the Q‐learning method, a reinforcement learning framework is presented to find the solution of the ARE in a data‐driven way, in which we only need to collect information from an arbitrary follower to learn the feedback gain matrix. Thus, the multi‐agent system can achieve distributed optimal consensus when system dynamics and global information are completely unavailable. For output feedback cases, accurate state information estimation is established such that optimal consensus control is realized. Moreover, the data‐driven optimal consensus method designed in this article is applicable to general digraph that contains a directed spanning tree. Finally, numerical simulations verify the validity of the proposed optimal control protocols and data‐driven framework.

Further results on optimal local and remote control with packet losses and transmission delay

In this article, the finite horizon optimal local and remote control (LRC) problem is further investigated for the case that both the packet losses (PL) and the transmission delays (TD) occur over the unreliable uplink channel from the local controller to the remote controller. In terms of the variational technique and the dynamic programming approach, the LRC problem with PL and TD is solved for the first time. The results derived in this article are innovative in the following aspects: Firstly, the necessary and sufficient solvability conditions are developed using the convex variational technique, illustrating that the solvability of the LRC problem is equivalent to the solvability of the forward and backward stochastic differential equations (FBSDEs). Secondly, by proposing an innovative hierarchical control technique and using the optimality principle of dynamic programming, the explicit optimal linear LRC is derived in terms of the completing square approach. Finally, the derived results of this article are illustrated by a numerical example.

Исследование предиктивных схем управления функционированием организационно-технических систем

Organizational and technical systems are the most common tool for solving national economic problems. However, due to the personnel included in them, they are characterized by stochastic behavior, which raises the issue of developing their probabilistic model. On the other hand, the probabilistic apparatus is a tool for forecasting future phenomena and situations and therefore a mechanism for its use in predictive (forecast) management is needed. For this purpose, the work uses the first probabilistic approximation in the form of a Markov process and its description by the Fokker–Planck–Kolmogorov equation. The purpose of the study is to formulate and solve optimal control problems with probabilistic control quality criteria for a tracking predictive control scheme, as well as to develop a mechanism for metasystem switching of mo¬dels in the case of an adaptive scheme with a predictive model in the control loop. Materials and methods. Based on the analysis of scientific ideas and methodological approaches in predictive management of domestic and foreign authors, as well as mathematical methods and models, two schemes for implementing predictive management were selected to improve the efficiency of the functioning of organizational and technical systems: tracking schemes and adaptive management. Results. The optimal control problem was solved using two methods: Pontryagin's maximum principle and Bellman's principle of dynamic programming. It is shown that in the first case there is a convex upward strategy for achieving the target value of the controlled value with a maximizing approach, and in the second case the result is a convex downward strategy used for the optimal consumption of management resources. In addition, for another predictive control scheme – adaptive with a predictive model in the loop, the effectiveness of using the strategy of metasystem switching of models with the formulation and solution of six metasystem problems has been proven. Conclusion. The proposed approach can be used when designing management systems in organizational and technical systems and choosing behavior strategies in difficult market conditions when solving problems of their development.

Dynamic Programming Principle for Automatic Negotiations

In automatic negotiation, an automated agent is trained to negotiate on behalf of a human negotiator. We consider that the domain of negotiations is known to both the agent and its opponents. In this setting, a greedy concession algorithm (GCA) is employed to find an optimal policy for the agent when the agent's belief about the opponent is known. However, the GCA is computationally expensive to certain class of policies. In this article, we propose a reverse greedy concession algorithm that is computationally less expensive for such class of policies. We also provide an alternate proof to establish that the GCA is a dynamic programming principle.

Optimal Control of Probability on a Target Set for Continuous-Time Markov Chains

In this paper, a stochastic optimal control problem is considered for a continuous-time Markov chain taking values in a denumerable state space over a fixed finite horizon. The optimality criterion is the probability that the process remains in a target set before and at a certain time. The optimal value is a superadditive capacity of target sets. Under some minor assumptions for the controlled Markov process, we establish the dynamic programming principle, based on which we prove that the value function is a classical solution of the Hamilton-Jacobi-Bellman equation on a discrete lattice space. We then prove that there exists an optimal deterministic Markov control under the compactness assumption of control domain. We further prove that the value function is the unique solution of the HJB equation. We also consider the case starting from the outside of target set and give the corresponding results. Finally, we apply our results to two examples.

Transition density function expansion methods for portfolio optimization

AbstractIn this study, we introduce transition density function expansion methods inspired from Yang et al. (J Econom. 2019;209(2):256–288.) to stochastic control issues related to utility maximization, without imposing limitations on the variety of asset price models and utility functions. Utilizing Bellman's dynamic programming principle, we initially recast the conditional expectation via the transition density function pertinent to the diffusion process. Subsequently, we employ the Itô‐Taylor expansion and Delta expansion techniques to the transition density function associated with the multivariate diffusion process, facilitated by a quasi‐Lamperti transformation, aiming to derive explicit recursive expressions for expansion coefficient functions. Our main contributions are that we articulate detailed algorithms, stemming from the backward recursive formulations of the value function and optimal strategies, achieved through discretization methodologies with rigorous proof of expansion convergence in portfolio optimization. Both theoretical and practical demonstrations are presented to validate the convergence of these approximate techniques in addressing stochastic control challenges. To underscore the efficiency and precision of our proposed methods, we apply them to portfolio selection problems within several benchmark models, and highlight the reduced complexity in comparison to the current methodologies.

A Continuous Time Framework for Sequential Goal-Based Wealth Management

We develop a continuous time framework for sequential goals-based wealth management. A stochastic factor process drives asset price dynamics and the client’s goal amount and income. We prove the weak dynamic programming principle for the value function of our control problem, which we show to be the unique viscosity solution of the corresponding Hamilton-Jacobi-Bellman equation. We develop an equivalent and computationally efficient representation of the Hamiltonian, which yields the optimal portfolio within a factor-dependent opportunity set defined by the maximum and minimum variance hypersurfaces. Our analysis shows that it is optimal to fund an expiring goal up to the level where the marginal benefit of additional fundedness is exceeded by the opportunity cost of diverting wealth from future goals. An all-or-nothing investor is more risk averse toward an approaching goal deadline if well funded, but more risk seeking if not on track with upcoming goals, compared with an investor with flexible goals. This paper was accepted by David Simchi-Levi, finance. Funding: This work was supported by the Natural Sciences and Engineering Research Council of Canada [Discovery Grant RGPIN-2020-06290] and Fi-Tek.

Mathematical modelling and optimal control analysis of pandemic dynamics as a hybrid system

The study of epidemics using mathematical modelling is critical in understanding its dynamics and proposing potential control measures. We propose a generalised epidemiological model corresponding to a pandemic wherein its dynamics is represented as a novel hybrid system obtained by coupling a deterministic model with a stochastic model. The hybrid system dynamics is established in individualistic (macroscopic) and intra-individualistic (microscopic) scales. The established hybrid system is then considered the basis for an optimal control problem, with the rate of vaccination and velocity of spatial dynamics taken as the control parameters affecting the system’s trajectory. We define the cost functional constituted by the continuous cost corresponding to the deterministic model and discrete costs corresponding to the transitions in the microscopic scale. The objective of the control problem is to find an optimal control pair of vaccination rate and spatial velocity, which minimises the cost functional. We use the Dynamic Programming Principle (DPP) as the optimisation technique, followed by verification of the value function obtained by DPP as a viscosity solution of the appropriate Hamilton–Jacobi–Bellman equation to analyse the existence of an optimal control pair to the hybrid system. We prove the existence of optimal controls to the multi-scale dynamics for pandemic modelling, along with an abstract method to synthesise it.

Optimal Strategy for Corporate International Investment and Consumption Problem with Stochastic Hyperbolic Discounting

This study investigates the dynamics of a stochastic hyperbolic discounting model in a continuous-time framework to address the complexities associated with the corporate international investment consumption problem (CIICP). By formulating a dynamic programming equation based on the principles of dynamic programming, we seek to untangle the intricacies of CIICP by incorporating log utility. Our research provides valuable insights into the economic ramifications of the proposed model and presents a comprehensive analysis of numerical sensitivities. This investigation not only contributes to a deeper understanding of the CIICP phenomenon but also contributes to more informed decision-making within the field.

Business decision-making of power generators in competitive electricity market

This paper presents a theoretical framework for the business decision-making process of the power generators as price takers when considering the participation of energy storage. The framework assesses rational valuation, optimal sales strategies, and hedging options for power plants with and without a gross sales constraint. The valuation and optimal sales strategy problems are analyzed using a risk-neutral pricing approach, dynamic programming principles, and the trinomial tree model suitable for the regime switching model. The formulation of a price risk hedging scheme flexible and widely used over-the-counter electricity derivative, the electricity contract for difference, as a tool for hedging electricity spot price risk. The minimum variance hedge ratio and its corresponding hedging efficiency formula are derived. In the section of numerical simulations, we first use the EM algorithm to calibrate the electricity spot model based on electricity spot price data of Nord Pool. Numerical simulations are then conducted on the operational decision-making of power generators under three different forms of energy storage. The results of the simulations provide a basis for power generators to evaluate the real-time value of power plants, to select optimal real-time power sales, and to determine the optimal timing of power plant transfer and storage methods.

Optimal strategy for Bayesian two-armed bandit problem with an arched reward function

In the classical two-armed bandit (TAB) problems, the goal of maximizing the total return specifies the optimal decision rule and fosters the development of abundant practicable methods, such as $ \epsilon $-greedy and upper confidence bound for TAB procedures. However, the existence of an arched reward function for the total return breaks the previously updated algorithms, like traditional myopic strategy, and prompts the generation of novel computation and theory for maximizing its expectation. Here we develop a special class of Bayesian two-armed bandit problems by imposing a prior probability on which arm has a greater expectation of the returns and propose a myopic strategy to specify the decision rule. We prove that the proposed myopic strategy is optimal under the structure of the arched reward function by exploring the dynamic programming principle. This article also demonstrates that arched functions are extensions of the monotone functions including the classical linear functions. Meanwhile, we establish a corresponding law of large numbers for our myopic strategy. Simulation studies pose supportive evidence that the newly proposed strategy performs well.

Nonzero-Sum Stochastic Impulse Games with an Application in Competitive Retail Energy Markets

We study a nonzero-sum stochastic differential game with both players adopting impulse controls, on a finite time horizon. The objective of each player is to maximize her total expected discounted profits. The resolution methodology relies on the connection between Nash equilibrium and the corresponding system of quasi-variational inequalities (QVIs in short). We prove, by means of the weak dynamic programming principle for the stochastic differential game, that the equilibrium expected payoff of each player is a constrained viscosity solution to the associated QVIs system in the class of linear growth functions. We also introduce a family of equilibrium expected payoffs converging to our equilibrium expected payoff of each player, and which is characterized as the unique constrained viscosity solutions of an approximation of our QVIs system. This convergence result is useful for numerical purpose. We apply a probabilistic numerical scheme which approximates the solution of the QVIs system to the case of the competition between two electricity retailers. We show how our model reproduces the qualitative behavior of electricity retail competition.

Dynamic programming principle for one kind of stochastic recursive optimal control problem with Markovian switching

Dynamic programming principle for one kind of stochastic recursive optimal control problem with Markovian switching

The defined contribution pension plan after retirement under the criterion of a revised loss considering the economic situation

<abstract> <p>Considering the economic situation, we investigate the optimal asset allocation of defined contribution pension funds with random payouts after retirement under a modified criterion of quadratic loss. The HJB equation is derived adhering to the dynamic programming principle, and the time-consistent optimal investment strategy is designed based on the calculus theory. Finally, under two different risk attitudes, namely surplus preference and risk aversion, the impact of key parameters on the optimal investment strategy and the function of minimum loss at the initial moment is compared and analyzed, the economic significance is demonstrated, and the rationality of the model is verified.</p> </abstract>

A non-zero-sum investment and reinsurance game between two mean–variance insurers with dynamic CVaR constraints

This paper is devoted to investigating a non-zero-sum game between two competing insurers. The insurers can diversify their insurance risks by purchasing proportional reinsurance and investing their collected premiums into a financial market composed of one risk-free asset and one stock. The reinsurance premiums charged by the reinsurer follow the generalized mean–variance premium principle. Moreover, the dynamic CVaR constraints are incorporated in the game problem to control risks. With the dynamic mean–variance objective, we introduce two forward deterministic auxiliary processes to represent the expectations of the insurers’ wealth processes and transform the original time inconsistent game problem into a standard time consistent game problem with two state variables for each insurer. By adopting the dynamic programming principle and the Lagrange duality method, we derive the Nash equilibrium investment–reinsurance strategies for the two insurers. Finally, the effects of several important model parameters on the optimal policies are analyzed by numerical examples.

Optimal investment and benefit payment adjustment strategies for the target benefit plan under partial information

This article investigates the optimal asset allocation and benefit adjustment problem for the target benefit plan (TBP) with predictable returns in an environment of partial information. We assume that the return rate of the stock depends on an observable and an unobservable state variables, and the pension manager estimates the unobservable component from known information through Bayesian learning. Under the criterion of expected exponential utility maximization, we obtain the optimal strategy in closed-form through the dynamic programming principle approach. Furthermore, numerical simulations are conducted by the Monte Carlo method to discuss the impacts of some parameters on the derived optimal strategy and to compare the optimal strategy with the suboptimal strategy when ignoring learning. It turns out that neglecting learning affects the optimal strategy seriously, thereby leading to significant utility losses.

Bellman filtering and smoothing for state–space models

This paper presents a new filter for state–space models based on Bellman’s dynamic-programming principle, allowing for nonlinearity, non-Gaussianity and degeneracy in the observation and/or state-transition equations. The resulting Bellman filter is a direct generalisation of the (iterated and extended) Kalman filter, enabling scalability to higher dimensions while remaining computationally inexpensive. It can also be extended to enable smoothing. Under suitable conditions, the Bellman-filtered states are stable over time and contractive towards a region around the true state at every time step. Static (hyper)parameters are estimated by maximising a filter-implied pseudo log-likelihood decomposition. In univariate simulation studies, the Bellman filter performs on par with state-of-the-art simulation-based techniques at a fraction of the computational cost. In two empirical applications, involving up to 150 spatial dimensions or highly degenerate/nonlinear state dynamics, the Bellman filter outperforms competing methods in both accuracy and speed.