Stochastic Control Problem Research Articles

Optimal Investment with a Noisy Signal of Future Stock Prices

We consider an investor who is dynamically informed about the future evolution of one of the independent Brownian motions driving a stock’s price fluctuations. With linear temporary price impact the resulting optimal investment problem with exponential utility turns out to be not only well posed, but it even allows for a closed-form solution. We describe this solution and the resulting problem value for this stochastic control problem with partial observation by solving its convex-analytic dual problem.

Linear quadratic stochastic optimal control with state- and control-dependent noises: A deterministic data approach

This paper investigates an infinite-horizon linear quadratic stochastic optimal control (LQSOC) problem with state- and control-dependent noises, in which two system matrices are unknown. First, a deterministic system is introduced and a relationship among its system trajectory, its control and the matrices to be solved is formulated. Subsequently, based on the deterministic system, an adaptive dynamic programming (ADP) algorithm is established to tackle the LQSOC problem. Then, the convergence analysis is presented by proving the equivalence between the proposed algorithm and an existing policy iteration algorithm. Finally, the effectiveness of the obtained algorithm is verified by a perturbed turbocharged diesel engine optimal control design example.

Convergence with rates for a Riccati-based discretization of SLQ problems with SPDEs

Abstract We consider a new discretization in space (parameter $h>0$) and time (parameter $\tau>0$) of a stochastic optimal control problem, where a quadratic functional is minimized subject to a linear stochastic heat equation with linear noise. Its construction is based on the perturbation of a generalized difference Riccati equation to approximate the related feedback law. We prove a convergence rate of almost ${\mathcal O}(h^{2}+\tau )$ for its solution, and conclude from it a rate of almost ${\mathcal O}(h^{2}+\tau )$ resp. ${\mathcal O}(h^{2}+\tau ^{1/2})$ for computable approximations of the optimal state and control with additive resp. multiplicative noise.

Controlling a generalized Fokker–Planck equation via inputs with nonlocal action

This paper concerns an optimal control problem (P) related to a generalized Fokker–Planck equation. Basic properties of the solutions to the generalized FP equation are derived via a semigroup approach in the space H−1(Rd). Problem (P) is proven to be deeply related to a stochastic optimal control problem (PS) for a McKean–Vlasov equation. The existence of an optimal control is obtained for the deterministic problem (P). The existence of an optimal control is established for an approximating optimal control problem (Ph) related to a backward Euler approximation of the generalized FP equation (with a constant discretization step h). One proves under additional hypotheses that “(Ph) converges to (P)” in a certain sense. First order necessary conditions for (Ph) are derived as well.

Barrier methods based on Jordan–Hilbert algebras for stochastic optimization in spin factors

Infinite-dimensional stochastic second-order cone programming involves minimizing linear functions over intersections of affine linear manifolds with infinite-dimensional second-order cones. However, even though there is a legitimate necessity to explore these methods in general spaces, there is an absence of infinite-dimensional counterparts for these methods. In this paper, we present decomposition logarithmic-barrier interior-point methods based on unital Jordan–Hilbert algebras for this class of optimization problems in the infinite-dimensional setting. The results show that the iteration complexity of the proposed algorithms is independent on the choice of Hilbert spaces from which the underlying spin factors are formed, and so it coincides with the best-known complexity obtained by such methods for the finite-dimensional setting. We apply our results to an important problem in stochastic control, namely the two-stage stochastic multi-criteria design problem. We show that the corresponding infinite-dimensional system in this case is a matrix differential Ricatti equation plus a finite-dimensional system, and hence, it can be solved efficiently to find the search direction.

A Gradient-Aware Search Algorithm for Constrained Markov Decision Processes.

The canonical solution methodology for finite constrained Markov decision processes (CMDPs), where the objective is to maximize the expected infinite-horizon discounted rewards subject to the expected infinite-horizon discounted costs' constraints, is based on convex linear programming (LP). In this brief, we first prove that the optimization objective in the dual linear program of a finite CMDP is a piecewise linear convex (PWLC) function with respect to the Lagrange penalty multipliers. Next, we propose a novel, provably optimal, two-level gradient-aware search (GAS) algorithm which exploits the PWLC structure to find the optimal state-value function and Lagrange penalty multipliers of a finite CMDP. The proposed algorithm is applied in two stochastic control problems with constraints for performance comparison with binary search (BS), Lagrangian primal-dual optimization (PDO), and LP. Compared with the benchmark algorithms, it is shown that the proposed GAS algorithm converges to the optimal solution quickly without any hyperparameter tuning. In addition, the convergence speed of the proposed algorithm is not sensitive to the initialization of the Lagrange multipliers.

Expected power utility maximization with delay for insurers under the 4/2 stochastic volatility model

In this paper, we are interested in the optimal investment and reinsurance strategies of an insurer with delay under the $ 4/2 $ stochastic volatility model. Indeed, the objective of the insurer is to maximize the expected power utility of the terminal wealth and the average wealth on finite time horizon. The other objective is to maximize the growth rate of expected power utility per unit time on infinite time horizon. The wealth of the insurer is described by an approximation of the classical Cramér–Lundberg process. Then, these problems can be formulated as stochastic control problems with delay. A pair of forward-backward stochastic differential equations that are derived via the stochastic maximum principle has an explicit solution obtained by solving a Riccati differential equation. So, the optimal strategy of the finite time horizon problem can be constructed explicitly. And, by investigating asymptotics of the Riccati equation, the infinite time horizon problem can be solved explicitly. Finally, we present some numerical results to illustrate our model, optimal strategies and sensitivities of some parameters.

A Problem of Finite-Horizon Optimal Switching and Stochastic Control for Utility Maximization

A Problem of Finite-Horizon Optimal Switching and Stochastic Control for Utility Maximization

Computationally Tractable Gaussian Process-based Stochastic Predictive Control Using Backoffs

Current stochastic nonlinear model predictive control (SNMPC) hinges on the lack of high-fidelity models that describe the system behavior and the lack of tractable solution methods that handle chance constraints. Motivated by this, a model-and data-driven predictive control approach using Gaussian processes (GP-MDPC) is synthesized in this paper. It exploits GPs to learn the unknown dynamics and apply Taylor expansion for uncertainty propagation through probabilistic modeling. A backoff approximation method is explored to reformulate the chance constraints into tractable expressions. Finally, a finite-horizon stochastic optimal control problem (FH-SOCP) is formulated.

Lagrangian dual method for solving stochastic linear quadratic optimal control problems with terminal state constraints

A stochastic linear quadratic (LQ) optimal control problem with a pointwise linear equality constraint on the terminal state is considered. A strong Lagrangian duality theorem is proved under a uniform convexity condition on the cost functional and a surjectivity condition on the linear constraint mapping. Based on the Lagrangian duality, two approaches are proposed to solve the constrained stochastic LQ problem. First, a theoretical method is given to construct the closed-form solution by the strong duality. Second, an iterative algorithm, called augmented Lagrangian method (ALM), is proposed. The strong convergence of the iterative sequence generated by ALM is proved. In addition, some sufficient conditions for the surjectivity of the linear constraint mapping are obtained.

Sufficient maximum principle for partially observed mean-field stochastic optimal control problems with delays

Sufficient maximum principle for partially observed mean-field stochastic optimal control problems with delays

Constrained mean-variance investment-reinsurance under the Cramér–Lundberg model with random coefficients

In this paper, we study an optimal mean-variance investment-reinsurance problem for an insurer (she) under the Cramér–Lundberg model with random coefficients. At any time, the insurer can purchase reinsurance or acquire new business and invest her surplus in a security market consisting of a risk-free asset and multiple risky assets, subject to a general convex cone investment constraint. We reduce the problem to a constrained stochastic linear-quadratic control problem with jumps whose solution is related to a system of partially coupled stochastic Riccati equations (SREs). Then we devote ourselves to establishing the existence and uniqueness of solutions to the SREs by pure backward stochastic differential equation (BSDE) techniques. We achieve this with the help of approximation procedure, comparison theorems for BSDEs with jumps, log transformation and BMO martingales. The efficient investment-reinsurance strategy and efficient mean-variance frontier are explicitly given through the solutions of the SREs, which are shown to be a linear feedback form of the wealth process and a half-line, respectively.

Well-posedness for a transmission problem connecting first and second-order operators

We establish the existence and uniqueness of viscosity solutions within a domain Ω ⊆ ℝn for a class of equations governed by elliptic and eikonal type equations in disjoint regions. Our primary motivation stems from the Hamilton-Jacobi equation that arises in the context of a stochastic optimal control problem.

The Landscape of Deterministic and Stochastic Optimal Control Problems: One-Shot Optimization Versus Dynamic Programming

The Landscape of Deterministic and Stochastic Optimal Control Problems: One-Shot Optimization Versus Dynamic Programming

Optimal Control of Infinite-Dimensional Differential Systems with Randomness and Path-Dependence and Stochastic Path-Dependent Hamilton–Jacobi Equations

This paper is devoted to the stochastic optimal control problem of infinite-dimensional differential systems allowing for both path-dependence and measurable randomness. As opposed to the deterministic path-dependent cases studied by Bayraktar and Keller [J. Funct. Anal. 275 (2018) 2096-2161], the value function turns out to be a random field on the path space and it is characterized by a stochastic path-dependent Hamilton-Jacobi (SPHJ) equation. A notion of viscosity solution is proposed and the value function is proved to be the unique viscosity solution to the associated SPHJ equation.

First- and Second-Order Maximum Principles for Discrete-Time Stochastic Optimal Control With Recursive Utilities

This paper deals with the discrete-time stochastic optimal control problems with recursive utilities under weakened convexity assumption. A new stochastic maximum principle is established. Moreover, by constructing two new adjoint equations and two new variational equations for the backward stochastic difference equation (BS <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\Delta$</tex-math></inline-formula> E), we obtain the second-order necessary optimality condition of quasi-singular control. Finally, as an illustration, a discrete-time mean-variance portfolio selection mixed with a recursive utility functional optimization problem is solved.

Dynamic programming principle for one kind of stochastic recursive optimal control problem with Markovian switching

Dynamic programming principle for one kind of stochastic recursive optimal control problem with Markovian switching

Stochastic Optimal Linear Control for Generalized Cost Functions With Time-Invariant Stochastic Parameters.

This study presents the design of feedback controllers for generalized cost functions to deal with stochastic optimal control problems. Target linear systems contain time-invariant stochastic parameters that describe system uncertainty. The cost functions involve nonlinear mappings and polynomial forms of the system states and inputs to express various performance metrics. Unfortunately, these properties cause difficulties in solving the problems. Conventional methods, such as the principle of optimality, are not employed to solve such problems owing to the time-invariant parameters. As opposed to the well-known quadratic functions, handling the generalized cost functions is a complicated task. This study overcomes these challenges by deriving an explicit relation between the cost function and the linear feedback gain of a controller. The derived relation enables the feedback gain to be optimized via a gradient method. A theoretical analysis ensures the convergence of the proposed gradient method. A suboptimal feedback controller is obtained to solve the problem, even for the generalized cost. Furthermore, the controller guarantees robust stability of the feedback system even with the stochastic parameters. It is demonstrated that the proposed cost function can express an expectation of a quadratic cost, risk-sensitive cost, polynomial cost, and input-to-state gain. A numerical simulation shows the effectiveness of the proposed method.

On mean-field control problems for backward doubly stochastic systems

This article is concerned with stochastic control problems for backward doubly stochastic differential equations of mean-field type, where the coefficient functions depend on the joint distribution of the state process and the control process. We obtain the stochastic maximum principle which serves as a necessary condition for an optimal control, and we also prove its sufficiency under proper conditions. As a byproduct, we prove the well-posedness for a type of mean-field fully coupled forward-backward doubly stochastic differential equation arising naturally from the control problem, which is of interest in its own right. Some examples are provided to illustrate the applications of our results to control problems in the types of scalar interaction and first order interaction.

A decentralized algorithm for a mean field control problem of piecewise deterministic Markov processes

This paper provides a decentralized approach for the control of a population of N agents to minimize an aggregate cost. Each agent evolves independently according to a Piecewise Deterministic Markov dynamics controlled via unbounded jumps intensities. The N-agent high dimensional stochastic control problem is approximated by the limiting mean field control problem. A Lagrangian approach is proposed. Although the mean field control problem is not convex, it is proved to achieve zero duality gap. A stochastic version of the Uzawa algorithm is shown to converge to the primal solution. At each dual iteration of the algorithm, each agent solves its own small dimensional sub problem by means of the Dynamic Programming Principal, while the dual multiplier is updated according to the aggregate response of the agents. Finally, this algorithm is used in a numerical simulation to coordinate the charging of a large fleet of electric vehicles in order to track a target consumption profile.