Stochastic Control Problem Research Articles

Dynamic programming principle for optimal control of uncertain random differential equations and its application to optimal portfolio selection

This study aimed to examine an uncertain stochastic optimal control problem premised on an uncertain stochastic process. The proposed approach is used to solve an optimal portfolio selection problem. This paper’s research is relevant because it outlines the procedure for solving optimal control problems in uncertain random environments. We implement Bellman’s principle of optimality method in dynamic programming to derive the principle of optimality. Then the resulting Hamilton-Jacobi-Bellman equation (the equation of optimality in uncertain stochastic optimal control) is used to solve a proposed portfolio selection problem. The results of this study show that the dynamic programming principle for optimal control of uncertain stochastic differential equations can be applied in optimal portfolio selection. Also, the study results indicate that the optimal fraction of investment is independent of wealth. The main conclusion of this study is that, in Itô-Liu financial markets, the dynamic programming principle for optimal control of uncertain stochastic differential equations can be applied in solving the optimal portfolio selection problem.

Stochastic optimal control of pre-exposure prophylaxis for HIV infection for a jump model.

We analyze a stochastic optimal control problem for the PReP vaccine in a model for the spread of HIV. To do so, we use a stochastic model for HIV/AIDS with PReP, where we include jumps in the model. This generalizes previous works in the field. First, we prove that there exists a positive, unique, global solution to the system of stochastic differential equations which makes up the model. Further, we introduce a stochastic control problem for dynamically choosing an optimal percentage of the population to receive PReP. By using the stochastic maximum principle, we derive an explicit expression for the stochastic optimal control. Furthermore, via a generalized Lagrange multiplier method in combination with the stochastic maximum principle, we study two types of budget constraints. We illustrate the results by numerical examples, both in the fixed control case and in the stochastic control case.

Finite Horizon Stochastic H2/H∞ Control for Continuous‐Time Systems Driven by G‐Brownian Motion

ABSTRACTIn this article, we consider a stochastic control problem under volatility ambiguity. In this framework, the state function is described by a stochastic differential equation driven by ‐Brownian motion. Different from the classical approach, the volatility ambiguity is characterized by a family of non‐dominated probability measures. control can be modeled as a “inf‐sup problem” and control can be modeled as a “sup‐sup problem.” With the help of stochastic maximum principle and dynamic programming principle, we derive the stochastic bounded real lemma and two sets of cross coupled generalized differential Riccati equations, which is associated with the solvability of control.

The relationship between maximum principle and dynamic programming principle for stochastic recursive control problem with random coefficients

The relationship between maximum principle and dynamic programming principle for stochastic recursive control problem with random coefficients

Chance-constrained stochastic optimal control of epidemic models: A fourth moment method-based reformulation

This work proposes a methodology for the reformulation of chance-constrained stochastic optimal control problems that ensures reliable uncertainty management of epidemic outbreaks. Specifically, the chance constraints are reformulated in terms of the first four moments of the stochastic state variables through the so-called fourth moment method for reliability. Moreover, a spectral technique is employed to obtain surrogate models of the stochastic state variables, which enables the efficient computation of the required statistics. The practical implementation of the proposed approach is demonstrated via the optimal control of two different stochastic mathematical models of the COVID-19 transmission. The numerical experiments confirm that, unlike those reformulations based on the Chebyshev–Cantelli’s inequality, the proposed method does not exhibit the undesired outcomes that are typically observed when a higher precision is required for the risk level associated to the given chance constraints.

Pathwise Stochastic Control and a Class of Stochastic Partial Differential Equations

AbstractIn this article, we study a stochastic optimal control problem in the pathwise sense, as initially proposed by Lions and Souganidis in [C. R. Acad. Sci. Paris Ser. I Math., 327 (1998), pp. 735-741]. The corresponding Hamilton-Jacobi-Bellman (HJB) equation, which turns out to be a non-adapted stochastic partial differential equation, is analyzed. Making use of the viscosity solution framework, we show that the value function of the optimal control problem is the unique solution of the HJB equation. When the optimal drift is defined, we provide its characterization. Finally, we describe the associated conserved quantities, namely the space-time transformations leaving our pathwise action invariant.

Infinite Horizon Backward Stochastic Difference Equations and Related Stochastic Recursive Control Problems

Infinite Horizon Backward Stochastic Difference Equations and Related Stochastic Recursive Control Problems

On the Gradient Method in One Portfolio Management Problem

This study refines the methodology for solving stochastic optimal control problems with quality criteria that include the sum of the quality functional of the classical formulation and an extremal measure. A two-level optimization solution of these kinds of problems is presented already for the case where the quality functional consists only of the extremal measure. Our study shows the possibility of solving the original time inconsistency problem through solving a two-level optimization problem, where the outer problem is solved by gradient methods since the value function is convex and the inner problem is solved by classical methods. Some experiments were carried out and confirmed the validity of the theory. The results of the study can be applied to the case of portfolio management with quality criteria containing the Conditional Value-at-Risk (CVaR) metric.

Stochastic Control Problems with State Reflections Arising from Relaxed Benchmark Tracking

This paper studies stochastic control problems motivated by optimal consumption with wealth benchmark tracking. The benchmark process is modeled by a combination of a geometric Brownian motion and a running maximum process, indicating its increasing trend in the long run. We consider a relaxed tracking formulation such that the wealth compensated by the injected capital always dominates the benchmark process. The stochastic control problem is to maximize the expected utility of consumption deducted by the cost of the capital injection under the dynamic floor constraint. By introducing two auxiliary state processes with reflections, an equivalent auxiliary control problem is formulated and studied, which leads to the Hamilton-Jacobi-Bellman equation with two Neumann boundary conditions. We establish the existence of a unique classical solution to the dual partial differential equation using some novel probabilistic representations involving the local time of some dual processes together with a tailor-made decomposition-homogenization technique. The proof of the verification theorem on the optimal feedback control can be carried out by some stochastic flow analysis and technical estimations of the optimal control. Funding: L. Bo and Y. Huang are supported by the National Natural Science Foundation of China [Grant 12471451], the Natural Science Basic Research Program of Shaanxi [Grant 2023-JC-JQ-05], the Shaanxi Fundamental Science Research Project for Mathematics and Physics [Grant 23JSZ010], and Fundamental Research Funds for the Central Universities [Grant 20199235177]. X. Yu is supported by the Hong Kong RGC General Research Fund (GRF) [Grant 15304122] and the Research Centre for Quantitative Finance at the Hong Kong Polytechnic University [Grant P0042708].

Pension funds with longevity risk: an optimal portfolio insurance approach

We present a unified framework designed to provide an optimal investment strategy for members of a defined contribution pension plan. Our model guarantees a minimum retirement savings level, expressed as a target annuity, by assuming uncertainty in interest rates, labor income, and mortality during the accumulation phase. To protect the accumulated retirement capital against both investment and longevity risks, the present value of the guaranteed lifetime annuity is regarded as the baseline wealth to hold upon reaching the retirement date, while a purpose-oriented proportion portfolio strategy is employed to invest the residual wealth. By applying standard dynamic programming techniques, we determine a closed-form solution to the stochastic control problem with the objective of maximizing the expected utility of the final surplus, defined as the difference between the accumulated wealth and the target annuity value. The theoretical findings are bolstered by a comprehensive numerical analysis, designed to assess the impact of longevity on investment policies, highlighting the suitability of our proposal in managing defined contribution schemes.

Environmental management and restoration under unified risk and uncertainty using robustified dynamic Orlicz risk

Environmental management and restoration should be designed such that the risk and uncertainty owing to nonlinear stochastic systems can be successfully addressed. We apply the robustified dynamic Orlicz risk to the modeling and analysis of environmental management and restoration to consider both the risk and uncertainty within a unified theory. We focus on the control of a jump-driven hybrid stochastic system that represents macrophyte dynamics. The dynamic programming equation based on the Orlicz risk is first obtained heuristically, from which the associated Hamilton–Jacobi–Bellman (HJB) equation is derived. In the proposed Orlicz risk, the risk aversion of the decision-maker is represented by a power coefficient that resembles a certainty equivalence, whereas the uncertainty aversion is represented by the Kullback–Leibler divergence, in which the risk and uncertainty are handled consistently and separately. The HJB equation includes a new state-dependent discount factor that arises from the uncertainty aversion, which leads to a unique, nonlinear, and nonlocal term. The link between the proposed and classical stochastic control problems is discussed with a focus on control-dependent discount rates. We propose a finite difference method for computing the HJB equation. Finally, the proposed model is applied to an optimal harvesting problem for macrophytes in a brackish lake that contains both growing and drifting populations.

ЗАДАЧА ОПТИМАЛЬНОГО СТОХАСТИЧЕСКОГО УПРАВЛЕНИЯ И ОЦЕНКИ СТОИМОСТИ КОРПОРАТИВНЫХ ОБЛИГАЦИЙ, ЗАВИСЯЩИХ ОТ ВРЕМЕНИ И СЛУЧАЙНЫХ СОСТОЯНИЙ ПАРАМЕТРОВ

In the optimal stochastic control problem, one estimates, using the utility indifference method, the bond price and the premium of a default swap contract (Credit default swap) when the model parameters (market interest rate, drift coef-ficient, and volatility of risky underlying assets) are random functions of time and state. Namely, the interest rate of the risk-free asset depends on time, and the prices of risky assets are described by linear homogeneous stochastic dif-ferential equations (SDEs) with multiplicative noise. To do this, the authors consider a portfolio with a risk-free asset and a risky asset with no default risk. For each such portfolio, the authors determine the amount of risky assets that maximizes the expected utility of its final wealth. This quantity allows one to solve the parabolic partial differential equations arising from the Hamilton-Jacobi-Bellman (HJB) equation by transforming them into ordinary differential equations using the method of variable separation to obtain the instantaneous value function of each portfolio. The authors derive the bond price and the default swap-contract premium (CDS) rate, which are the amounts that provide the same level of the expected utility by investing all of one’s wealth in a portfolio that does not contain these credit instruments, or by investing these amounts in credit instruments and the remainder of one’s wealth in the portfolio.

Deep learning for solving initial path optimization of mean-field systems with memory

We consider the problem of finding the optimal initial investment strategy for a system modelled by a linear McKean–Vlasov (mean-field) stochastic differential equation with delay, driven by Brownian motion and a pure jump Poisson random measure. The goal is to determine the optimal initial values for the system in the period [ − δ , 0 ] , where δ > 0 is a delay constant, before the system starts at t = 0. Due to the delay in the dynamics, the system will, after startup, be influenced by these initial investment values. It is known that linear stochastic delay differential equations are equivalent to stochastic Volterra integral equations. By utilizing this equivalence, we can find implicit expressions for the optimal investment. Moreover, we propose a deep neural network-based algorithm to solve the stochastic control problem with delay. Specifically, we employ a multi-layer feed-forward neural network for control modelling in the interval [ − δ , 0 ] , and use back-propagation to train the feed-forward neural network. The gradient of the loss function is computed using stochastic gradient descent (SGD) with respect to the weights of the network.

Drift Control of High-Dimensional Reflected Brownian Motion: A Computational Method Based on Neural Networks

Motivated by applications in queueing theory, we consider a stochastic control problem whose state space is the d-dimensional positive orthant. The controlled process Z evolves as a reflected Brownian motion whose covariance matrix is exogenously specified, as are its directions of reflection from the orthant’s boundary surfaces. A system manager chooses a drift vector [Formula: see text] at each time t based on the history of Z, and the cost rate at time t depends on both [Formula: see text] and [Formula: see text]. In our initial problem formulation, the objective is to minimize expected discounted cost over an infinite planning horizon, after which we treat the corresponding ergodic control problem. Extending the earlier work by Han et al. [Han J, Jentzen A, Weinan E (2018) Solving high-dimensional partial differential equations using deep learning. Proc. Natl. Acad. Sci. USA 115(34):8505–8510], we develop and illustrate a simulation-based computational method that relies heavily on deep neural network technology. For the test problems studied thus far, our method is accurate to within a fraction of 1% and is computationally feasible in dimensions up to at least [Formula: see text].

Valuation of guaranteed lifelong withdrawal benefit with the long-term care option

In this paper, under the stochastic interest rate framework, we consider the valuation of a Guaranteed Lifelong Withdrawal Benefit (GLWB) annuity product by explicitly incorporating the health state of the policyholder through the long-term care (LTC) option. The product provides policyholders with protection against longevity risk and market downturns, as well as financial support when facing LTC needs. Within the context of dynamic withdrawals, the valuation of the GLWB annuity with the LTC option is characterized as a stochastic optimal control problem. We introduce a novel bang-bang analysis approach without the usual convexity assumption in literature and prove that the optimal withdrawal strategies for the policyholder are constrained to a finite set. Furthermore, we perform a sensitivity analysis on the price determinants of GLWB annuities with and without the LTC option, and provide economic interpretations. Lastly, we investigate the impact of gender on the optimal withdrawal strategy and the fair fee of the annuity with the LTC option.

Meta-Reinforcement Learning for Spacecraft Proximity Operations Guidance and Control in Cislunar Space

Innovative lightweight and model-free guidance algorithms are essential to achieve full autonomy of spacecraft and address future space exploration challenges. In the near future, this type of technology will be essential for cislunar space proximity operations, such as NASA’s Artemis program and its Lunar Gateway project. In this scenario, a meta-reinforcement learning algorithm is employed to address the real-time optimal guidance problem of a spacecraft in the cislunar environment. Non-Keplerian orbits pose complex dynamics, where classic control theory may be less adaptable and more computationally expensive compared to machine learning approaches. Meta-RL stands out for its ability to learn how to learn through experience, training on various tasks to enhance efficiency, and effectiveness in tackling new ones. By modeling a stochastic optimal control problem within the circular restricted three-body problem framework as a Markov decision process, an LSTM-based network agent is trained using proximal policy optimization, considering operational constraints and stochastic effects for safety and robustness evaluation. Additionally, an MLP-based agent and an optimal control pseudo-spectral solution are assessed for comparison. The resulting tool autonomously guides spacecraft in cislunar proximity operations, approximating the optimal control solution with a versatile algorithmic framework. This ensures both robustness and computational efficiency.

The solution to an impulse control problem motivated by optimal harvesting

We consider a stochastic impulse control problem that is motivated by applications such as the optimal exploitation of a natural resource. In particular, we consider a stochastic system whose uncontrolled state dynamics are modelled by a non-explosive positive linear diffusion. The control that can be applied to this system takes the form of one-sided impulsive action. The objective of the control problem is to maximise a discounted performance criterion that rewards the effect of control action but involves a fixed cost at each time of a control intervention. We derive the complete solution to this problem under general assumptions. It turns out that the solution can take four qualitatively different forms, several of which have not been observed in the literature. In two of the four cases, there exist only ε-optimal control strategies. We also show that the boundary classification of 0 may play a critical role in the solution of the problem. Furthermore, we develop a way for establishing the strong solution to a stochastic impulse control problem's optimally controlled SDE.

Dynamic Optimization with Timing Risk

Timing risk refers to a situation in which the timing of an economically important event is unknown (risky) from the perspective of an economic decision maker. While this special class of dynamic stochastic control problems has many applications in economics, the methods used to solve them are not easily accessible within a single, comprehensive survey. We provide a survey of dynamic optimization methods under comprehensive assumptions about the nature of timing risk. We also relax the assumption of full information and summarize optimization with limited information, ambiguity, imperfect hedging, and dynamic inconsistency. Our goal is to provide a concise user guide for specialists and nonspecialists alike.

Dynamic margin optimization

In response to the Global Financial Crisis of 2007–2009, by now, most of the financial transactions must be cleared through central counterparties operating a dynamic margin setting mechanism. High margin calls can reduce counterparty risk in a turbulent market, but at the same time, increase liquidity risk and escalate systemic risk. In this paper, we construct a theoretical model to address this challenge, deriving an optimal margin setting policy framed as a stochastic control problem. Our analysis reveals that an adaptive, countercyclical approach is superior to a purely risk-sensitive strategy, primarily by minimizing the expected loss for the clearing institution.

A general time-inconsistent stochastic optimal control problem with delay

A general time-inconsistent stochastic optimal control problem with delay