Adjoint Processes Research Articles

On the maximum principle for relaxed control problems of nonlinear stochastic systems

We consider optimal control problems for a system governed by a stochastic differential equation driven by a d-dimensional Brownian motion where both the drift and the diffusion coefficient are controlled. It is well known that without additional convexity conditions the strict control problem does not admit an optimal control. To overcome this difficulty, we consider the relaxed model, in which admissible controls are measure-valued processes and the relaxed state process is governed by a stochastic differential equation driven by a continuous orthogonal martingale measure. This relaxed model admits an optimal control that can be approximated by a sequence of strict controls by the so-called chattering lemma. We establish optimality necessary conditions, in terms of two adjoint processes, extending Peng’s maximum principle to relaxed control problems. We show that relaxing the drift and diffusion martingale parts directly as in deterministic control does not lead to a true relaxed model as the obtained controlled dynamics is not continuous in the control variable.

Duality Method for Multidimensional Nonsmooth Constrained Linear Convex Stochastic Control

In this paper, we discuss a general multidimensional linear convex stochastic control problem with nondifferentiable objective function, control constraints, and random coefficients. We formulate an equivalent dual problem, prove the dual stochastic maximum principle and the relation of the optimal control, optimal state, and adjoint processes between primal and dual problems, and illustrate the usefulness of the dual approach with some examples.

Maximum Principle for Stochastic Control of SDEs with Measurable Drifts

In this paper, we consider stochastic optimal control of systems driven by stochastic differential equations with irregular drift coefficient. We establish a necessary and sufficient stochastic maximum principle. To achieve this, we first derive an explicit representation of the first variation process (in the Sobolev sense) of the controlled diffusion. Since the drift coefficient is not smooth, the representation is given in terms of the local time of the state process. Then we construct a sequence of optimal control problems with smooth coefficients by an approximation argument. Finally, we use Ekeland’s variational principle to obtain an approximating adjoint process from which we derive the maximum principle by passing to the limit. The work is notably motivated by the optimal consumption problem of investors paying wealth tax.

Itô’s formula for flows of measures on semimartingales

We establish Itô’s formula along flows of probability measures associated with general semimartingales; this generalizes existing results for flows of measures on Itô processes. Our approach is to first establish Itô’s formula for cylindrical functions and then extend it to the general case via function approximation and localization techniques.This general form of Itô’s formula enables the derivation of dynamic programming equations and verification theorems for McKean–Vlasov controls with jump diffusions and for McKean–Vlasov mixed regular–singular control problems. It also allows for generalizing the classical relationship between the maximum principle and the dynamic programming principle to the McKean–Vlasov singular control setting, where the adjoint process is expressed in terms of the derivative of the value function with respect to the probability measures.

Time-translational symmetry in statistical dynamics dictates Einstein relation, Green-Kubo formula, and their generalizations.

A stochastic dynamics has a natural decomposition into a drift capturing mean rate of change and a martingale increment capturing randomness. They are two statistically uncorrelated, but not necessarily independent, components contributing to the overall fluctuations of the dynamics, representing the uncertainties in the past and in the future. We show that fluctuation-dissipation relations of the two aforementioned components, such as the Einstein relation and the Green-Kubo formula, can be formulated for any stochastic process with a steady state, without additional supposition of the process being Markovian, reversible, or linear. Further, by considering the adjoint process defined by the time reversal at the steady state, we show that reversibility in equilibrium leads to an additional symmetry in the covariance between system's state and drift. Potential directions of further generalizing our results to processes without steady states is briefly discussed.

A Modified Method of Successive Approximations for Stochastic Recursive Optimal Control Problems

Based on the stochastic maximum principle for the partially coupled forward-backward stochastic control system (FBSCS for short), a modified method of successive approximations (MSA for short) is established for stochastic recursive optimal control problems. The second-order adjoint processes are introduced in the augmented Hamiltonian minimization step since the control domain is not necessarily convex. Thanks to the theory of bounded mean oscillation martingales (BMO martingales for short), we give a delicate proof of the error estimate and then prove the convergence of the modified MSA algorithm. In a special case, we obtain a logarithmic convergence rate. When the control domain is convex and compact, a sufficient condition which makes the control returned from the MSA algorithm be a near-optimal control is given for a class of linear FBSCSs.

Sufficient and necessary conditions of near-optimal controls for a diffusion dengue model with Lévy noise

In this paper, a dengue model, which is described by spatial diffusion and Lévy noise, is built, and the sufficient and necessary conditions for near-optimal controls of the stochastic dengue model are given. First, the existence and uniqueness of the global positive solution is proved by constructing a Lyapunov function. Then, to obtain the results, the first-order adjoint equation and several estimates of the variation of the state and adjoint processes are given. Finally, the sufficient and necessary conditions for near-optimal controls are derived by virtue of Pontryagin's stochastic maximum principle, and numerical simulations are introduced to verify the effect of diffusion and Lévy noise.

Backward iterations for solving integral equations with polynomial nonlinearity

The theory of adjoint operators is widely used in solving applied multidimensional problems with Monte Carlo method. Efficient algorithms are constructed using the duality principle for many problems described in linear integral equations of the second kind. On the other hand, important applications of adjoint equations for designing experiments were suggested by G.Marchuk and his colleagues in their respective works. Some results obtained in these fields are also generalized to the case of nonlinear operators. Linearization methods were mostly used for that purpose. Results for Lyapunov-Schmidt nonlinear polynomial equations were obtained in the theory of Monte Carlo methods. However, many interesting questions in this subject area are remained open. New results about dual processes used for solving polynomial equations with Monte Carlo method are presented. In particular, an adjoint Markov process for the branching process and the corresponding unbiased estimate of the functional of the solution to the equation are constructed in a general form. The possibility of constructing an adjoint operator to a nonlinear one is discussed.

A mean-field stochastic linear-quadratic optimal control problem with jumps under partial information

In this article, the stochastic linear-quadratic optimal control problem of mean-field type with jumps under partial information is discussed. The state equation which contains affine terms is a SDE with jumps driven by a multidimensional Brownian motion and a Poisson stochastic martingale measure, and the quadratic cost function contains cross terms. In addition, the state and the control as well as their expectations are contained both in the state equation and the cost functional. This is the so-called optimal control problem of mean-field type. Firstly, the existence and uniqueness of the optimal control is proved. Secondly, the adjoint processes of the state equation is introduced, and by using the duality technique, the optimal control is characterized by the stochastic Hamiltonian system. Thirdly, by applying a decoupling technology, we deduce two integro-differential Riccati equations and get the feedback representation of the optimal control under partial information. Fourthly, the existence and uniqueness of the solutions of two Riccati equations are proved. Finally, we discuss a special case, and establish the corresponding feedback representation of the optimal control by means of filtering technique.

An efficient algorithm for stochastic optimal control problems by means of a least-squares Monte-Carlo method

In this work, we provide discrete optimality conditions of the optimal control problems of stochastic differential equations. Euler and Runge–Kutta methods are used for discretization. A Lagrange multiplier method for a discrete-time stochastic optimal control problem is formulated. The discrete adjoint process is obtained in terms of conditional expectations and for both methods. To estimate these nested conditional expectations at each time step via simulation, we use a very powerful new approach, least-squares Monte-Carlo method, developed by Longstaff–Schwartz. This is the first time to solve a stochastic optimal control problem by calculating the nested conditional expectations numerically with the help of a least-squares Monte-Carlo method. Some examples are studied to test and demonstrate the efficiency of the Lagrange multiplier combined with the least-squares Monte-Carlo method.

Near Optimality of Linear Delayed Doubly Stochastic Control Problem

In this paper, we study a kind of near optimal control problem which is described by linear quadratic doubly stochastic differential equations with time delay. We consider the near optimality for the linear delayed doubly stochastic system with convex control domain. We discuss the case that all the time delay variables are different. We give the maximum principle of near optimal control for this kind of time delay system. The necessary condition for the control to be near optimal control is deduced by Ekeland’s variational principle and some estimates on the state and the adjoint processes corresponding to the system.

Maximum principle for stochastic optimal control problem of forward–backward stochastic difference systems

In this paper, we study the maximum principle for stochastic optimal control problems of forward–backward stochastic difference systems (FBSΔSs). Two types of FBSΔSs are investigated. The first one is described by a partially coupled forward–backward stochastic difference equation (FBSΔE) and the second one is described by a fully coupled FBSΔE. By adopting an appropriate representation of the product rule and an appropriate formulation of the adjoint process, we deduce the adjoint difference equation. Finally, the maximum principle for this optimal control problem with the control domain being convex is established.

Risk‐sensitive maximum principle for stochastic optimal control of mean‐field type Markov regime‐switching jump‐diffusion systems

AbstractWe consider the risk‐sensitive optimal control problem for mean‐field type Markov regime‐switching jump‐diffusion systems driven by Brownian motions and Poisson jumps with (Markovian) switching coefficients. The system is coupled with its mean‐filed term, that is, the expected value of the state process, and the objective functional is of the risk‐sensitive type. Our problem is closely related to the mean‐field type robust optimization problem for a general class of stochastic jump systems due to the inherent feature of the risk‐sensitive objective functional. By establishing the logarithmic transformations of the associated equivalent singular risk‐neutral control problem, we obtain the risk‐sensitive maximum principle type necessary and sufficient conditions for optimality, where the sufficient condition requires an additional convexity assumption. The risk‐sensitive maximum principle in this article is characterized as the variational inequality, together with the first‐ and second‐order (mean‐field type) adjoint processes as well as the auxiliary first‐order adjoint process. Unlike the risk‐neutral and mean‐field free cases, the additional adjoint equation is induced due to the mean‐field coupling term and the risk‐sensitive logarithmic transformation. We apply the risk‐sensitive maximum principle of this article to the risk‐sensitive linear‐quadratic problem, for which an explicit optimal solution is obtained.

Near-optimal control and threshold behavior of an avian influenza model with spatial diffusion on complex networks.

Near-optimization is as sensible and important as optimization for both theory and applications. This paper concerns the near-optimal control of an avian influenza model with saturation on heterogeneous complex networks. Firstly, the basic reproduction number $ \mathcal{R}_{0} $ is defined for the model, which can be used to govern the threshold dynamics of influenza disease. Secondly, the near-optimal control problem was formulated by slaughtering poultry and treating infected humans while keeping the loss and cost to a minimum. Thanks to the maximum condition of the Hamiltonian function and the Ekeland's variational principle, we establish both necessary and sufficient conditions for the near-optimality by several delicate estimates for the state and adjoint processes. Finally, a number of examples presented to illustrate our theoretical results.

Recover Dynamic Utility from Observable Process: Application to the Economic Equilibrium

Decision making under uncertainty is often viewed as an optimization problem under choice criterium, but its calibration raises the inverse problem to recover the criterium from the data. An example is the theory of preference by Samuelson in the 40s, \cite{samuelson1938}. The observable is a so-called increasing characteristic process $\mbX=(\scX_t(x))$ and the objective is to recover a dynamic stochastic utility $\bfU$ in the sense where $U(t,\scX_t(x))$ is a martingale. A linearized version is provided by the first order conditions $U_x(t,\scX_t(x))=Y_t(u_z(x)$, and the additional martingale conditions of the processes $\scX_x(t,x)Y_t(u_z(x))$ and $\scX_t(x)Y_t(u_z(x))$. When $\mbX$ and $\bfY$ are regular solutions of two SDEs with random coefficients, under strongly martingale condition, any revealed utility is solution of a non linear SPDE, and is the stochastic value function of some optimization problem. More interesting is the dynamic equilibrium problem as in He and Leland \cite{HL}, where $Y$ is coupled with $\scX$ so that the monotonicity of $Y_t(z,u_z(z)) $ is lost. Nevertheless, we solve the He \& Leland problem (in random environment), by characterizing all the equilibria: the adjoint process still linear in $y$ (GBM in Markovian case) and the conjugate utilities are a deterministic mixture of stochastic dual power utilities. Besides, the primal utility is the value function of an optimal wealth allocation in the Pareto problem.

Second Order Necessary Conditions for Optimal Control Problems of Stochastic Evolution Equations

In this paper, we obtain some second order necessary conditions for optimal control problems of stochastic evolution equations in infinite dimensions. The control acts on both the drift and diffusion terms while the control region is assumed to be convex. The concepts of relaxed and $V$-transposition solutions (introduced in our previous works) to operator-valued backward stochastic evolution equations are employed to formulate these optimality conditions. The correction part of the second order adjoint process, which does not appear in the Pontryagin-type stochastic maximum principle, plays a fundamental role in our results.

Singular optimal controls for stochastic recursive systems under convex control constraint

In this paper, we study two kinds of singular optimal controls (SOCs for short) problems where the systems governed by forward-backward stochastic differential equations (FBSDEs for short), in which the control has two components: the regular control, and the singular one. Both drift and diffusion terms may involve the regular control variable. The regular control domain is postulated to be convex. Under certain assumptions, in the framework of the Malliavin calculus, we derive the pointwise second-order necessary conditions for stochastic SOC in the classical sense. This condition is described by two adjoint processes, a maximum condition on the Hamiltonian supported by an illustrative example. A new necessary condition for optimal singular control is obtained as well. Besides, as a by-product, a verification theorem for SOCs is derived via viscosity solutions without involving any derivatives of the value functions. It is worth pointing out that this theorem has wider applicability than the restrictive classical verification theorems. Finally, we focus on the connection between the maximum principle and the dynamic programming principle for such SOCs problem without the assumption that the value function is smooth enough.

Stochastic recursive optimal control problem with mixed delay under viscosity solution's framework

SummaryThis article is concerned with the stochastic recursive optimal control problem with mixed delay. The connection between Pontryagin's maximum principle and Bellman's dynamic programming principle is discussed. Without containing any derivatives of the value function, relations among the adjoint processes and the value function are investigated by employing the notions of super‐ and sub‐jets introduced in defining the viscosity solutions. Stochastic verification theorem is also given to verify whether a given admissible control is really optimal.

Maximum principle for infinite horizon optimal control of mean-field backward stochastic systems with delay and noisy memory

In this paper, we consider a problem of optimal control of an infinite horizon mean-field backward stochastic differential equation with delay and noisy memory under partial information. We derive necessary and sufficient maximum principles using Malliavin calculus technique for such a system. A class of mean-field time-advanced stochastic differential equations is introduced as the adjoint process which involves partial derivatives of the Hamiltonian functions and their Malliavin derivatives. To illustrate our theoretical results, we give an example for a linear-quadratic mean-field backward delay stochastic system with noisy memory on infinite horizon to obtain the optimal control. Also, we apply our results to pension fund problems with delay and noisy memory which are arising from the financial market.

The risk-sensitive maximum principle for controlled forward–backward stochastic differential equations

In this paper, we consider the risk-sensitive optimal control problem for forward–backward stochastic differential equations (FBSDEs). We consider two different cases: (Case 1) the drift and diffusion terms of the forward part are independent of the backward part (not fully-coupled FBSDE) and the running cost does not include the backward part; (Case 2) the fully-coupled FBSDE with terminal and initial costs. In both cases, the risk-sensitive objective functional is considered, the corresponding control domain is not necessarily convex, and the diffusion term of the forward part depends on control and state variables. In both cases, by using the nonlinear transformations of the equivalent risk-neutral problems, we obtain two (different) risk-sensitive maximum principles, which are characterized in terms of the variational inequalities. We show that the risk-sensitive maximum principle of (Case 1) consists of the two-coupled adjoint equations with an additional scalar adjoint equation, which are backward stochastic differential equations (BSDEs). The risk-sensitive maximum principle of (Case 2) consists of the two-coupled adjoint equations, which are coupled FBSDEs, with an additional scalar adjoint equation and optimality conditions. In both cases, the additional scalar adjoint equations are BSDEs, which are induced due to the nonlinear transformations of the adjoint processes in the equivalent risk-neutral problems. Through the application of results, we consider the linear–quadratic risk-sensitive optimal control problem for the linear FBSDE, and by using the risk-sensitive maximum principle of (Case 1), we characterize an explicit optimal solution.