Stochastic Dynamical Programming Principle Research Articles

Optimal control of stochastic differential equations with random impulses and the Hamilton–Jacobi–Bellman equation

AbstractIn this article, we study the optimal control of stochastic differential equations with random impulses. We optimize the performance index and add the influence of random impulses to the performance index with a random compensation function. Using the idea of stochastic analysis and dynamic programming principle, a new Hamilton–Jacobi–Bellman (HJB) equation is obtained, and the existence and uniqueness of its viscosity solution are proved.

Optimal Investment Strategies for DC-Pension Fund under Combined Stochastic Volatility Models

We investigate the optimal investment strategies of DC pension under stochastic volatility model using combined Heston-Hull-White (HHW) model with a constant income drawdown. The pension fund manager (PFM) aims to maximize the expected terminal utility of wealth in a complete market setting under constant relative risk aversion (CRRA). The goal of the PFM is to maintain the standard of living of the participants after retirement. We derive the HJB equation associated with the control problem and finally established the close form solution using stochastic dynamic programming principle (SDPP). The results show that the optimal investment and benefit payment strategies converge uniquely with time.

Optimal Asset Allocation for CRRA and CARA Insurers under the Vasicek Interest Rate Model

This paper considers the reinsurance-investment problem with interest rate risks under constant relative risk aversion and constant absolute risk aversion preferences, respectively. Stochastic control theory and dynamic programming principle are applied to investigate the optimal proportional reinsurance-investment strategy for an insurer under the Vasicek stochastic interest rate model. Solving the corresponding Hamilton-Jacobi-Bellman equation via the Legendre transform approach, the optimal premium allocation strategies maximizing the expected utilities of terminal wealth are derived. In addition, several sensitivity analyses and numerical illustrations are given to analyze the impacts of different risk preferences and interest rate fluctuation on the optimal strategies. We find that the asset allocation and reinsurance ratio of the insurer are correlated with risk preference coefficient and interest rate fluctuation, and the insurance company may adjust the reinsurance-investment strategy to deal with interest rate risk.

Stochastic Minimax Vibration Control for Uncertain Nonlinear Quasi-Hamiltonian Systems with Noisy Observations

A stochastic minimax control strategy for uncertain nonlinear quasi-Hamiltonian systems with noisy observations under random excitations is proposed based on the extended Kalman filter and minimax stochastic dynamical programming principle. A structure system with smart sensors and actuators is modeled as a controlled, excited and dissipative Hamiltonian system with noisy observations. The differential equations for the uncertain nonlinear quasi-Hamiltonian system with control and observation under random excitation are given first. The estimated nonlinear stochastic control system with uncertain parameters is obtained from the uncertain quasi-Hamiltonian system with noisy observation. In this case, the optimally estimated state is determined by the observation based on the extended Kalman filter. The dual dynamical programming equation for the estimated uncertain system is then obtained based on the minimax stochastic dynamical programming principle. The worst-case disturbances are determined for bounded uncertain parameters and the optimal control law is determined for the worst case by the programming equation. The proposed minimax control strategy is applied to two uncertain nonlinear stochastic systems with controls and noisy observations. The control effectiveness for the stochastic vibration response reductions of the systems is illustrated with numerical results. The proposed minimax control strategy is applicable to general uncertain nonlinear multi-degree-of-freedom structure systems with noisy observations under random excitations.

Random vibration control for multi-degree-of-freedom mechanical systems with soft actuators

Soft robotics include soft actuators and possess the capacity of large deformation and environmental compatibility. Weak environmental disturbances may deteriorate the operating performance of soft robotics due to the low stiffness of soft actuators. This manuscript investigates multi-degree-of-freedom nonlinear mechanical systems with dielectric elastomer actuators and establishes the bounded/unbounded optimal control strategies to suppress the random vibration around the equilibrium position by adjusting the imposed voltage in real time. First, the constitutive relation of a plan-type dielectric elastomer actuator and then the vibrating equation of the multi-degree-of-freedom nonlinear system around the equilibrium position are derived successively. The bounded/unbounded optimal control problems are then established by adopting the corresponding performance indexes. The bounded/unbounded optimal control strategies are then derived by combining the stochastic averaging technique and stochastic dynamic programming principle. Numerical results on a two-degree-of-freedom nonlinear system illustrate the application and efficacy of the proposed optimal control strategies.

Management strategies for a defined contribution pension fund under the hybrid stochastic volatility model

In this paper, we introduce the Heston–Hull–White (the hybrid) model in the pension fund management. The optimal investment and benefit payments policies for the DC pension fund with an income drawdown option are presented explicitly. In this study, the pension fund manager is allowed to invest the fund wealth in riskless and risky assets. The risk asset price dynamics evolve according to the Hybrid Stochastic Volatility (Heston–Hull–White) model. The goal of the pension fund manager is to maintain the standard of lives for the pension fund members after retirement. We derived the HJB equation and established the closed-form solution for the optimal control policies using stochastic dynamic programming principle. To illustrate the economic implications of the model, we provide a numerical example and simulations. The results reveal that the optimal investment and benefit payments strategies convergence successful with time t. The risk asset price volatility and interest rate mean reversion are inversely proportional to investment policy. The interest rate volatility and risk preference are both directly proportional to investment and benefit payments. The pension fund managers are advised to control the investment and the benefit payments policies to achieve the goals of the pension fund members.

Nonlinear Stochastic Optimal Control of MDOF Partially Observable Linear Systems Excited by Combined Harmonic and Wide-Band Noises

In this paper, nonlinear stochastic optimal control of multi-degree-of-freedom (MDOF) partially observable linear systems subjected to combined harmonic and wide-band random excitations is investigated. Based on the separation principle, the control problem of a partially observable system is converted into a completely observable one. The dynamic programming equation for the completely observable control problem is then set up based on the stochastic averaging method and stochastic dynamic programming principle, from which the nonlinear optimal control law is derived. To illustrate the feasibility and efficiency of the proposed control strategy, the responses of the uncontrolled and optimal controlled systems are respectively obtained by solving the associated Fokker–Planck–Kolmogorov (FPK) equation. Numerical results show the proposed control strategy can dramatically reduce the response of stochastic systems subjected to both harmonic and wide-band random excitations.

Response Evaluation and Optimal Control for Stochastically Excited Vibro-Impact System with Hertzdamp Contact

This manuscript devotes to procedures to evaluate random responses and explore stochastic optimal control for a vibro-impact system, in which the inelastic impact is modeled as Hertzdamp contact model. The dimension of governing equation is halved with respect to the total energy of uncontrolled/controlled system by conducting the stochastic averaging of energy envelope. The stationary probability density and then the response statistics of the uncontrolled system are much easier to solve due to a low-dimensional Fokker-Planck-Kolmogorov equation. The original optimal control problem with respect to states of system is converted to that with respect to system total energy. The latter is quite simpler than the former although it only can be regarded as the suboptimal control of the original system. The optimal control with damping property is determined by adopting the stochastic maximum principle and solving simultaneously a set of forward-backward stochastic differential equations. The accuracy of the proposed procedure for evaluation of the random response is verified by comparing with Monte-Carlo simulations and the efficiency of the optimal control strategy is also illustrated. The consistence between the analytical result from energy dissipative balance technique and that from stochastic averaging of energy envelope is discussed, which means the rigorous restriction on stochastic averaging, i.e., light damping and weak excitation, can be removed. The optimal control strategy from stochastic maximum principle and that from stochastic dynamic programming principle holds close connection based on our related discussion.

Semi-active bounded optimal control of uncertain nonlinear coupling vehicle system with rotatable inclined supports and MR damper under random road excitation

The semi-active optimal vibration control of nonlinear torsion-bar suspension vehicle systems under random road excitations is an important research subject, and the boundedness of MR dampers and the uncertainty of vehicle systems are necessary to consider. In this paper, the differential equations of motion of the coupling torsion-bar suspension vehicle system with MR damper under random road excitation are derived and then transformed into strongly nonlinear stochastic coupling vibration equations. The dynamical programming equation is derived based on the stochastic dynamical programming principle firstly for the nonlinear stochastic system. The semi-active bounded parametric optimal control law is determined by the programming equation and MR damper dynamics. Then for the uncertain nonlinear stochastic system, the minimax dynamical programming equation is derived based on the minimax stochastic dynamical programming principle. The worst-case disturbances and corresponding semi-active bounded parametric optimal control are obtained from the programming equation under the bounded disturbance constraints and MR damper dynamics. The control strategy for the nonlinear stochastic vibration of the uncertain torsion-bar suspension vehicle system is developed. The good effectiveness of the proposed control is illustrated with numerical results. The control performances for the vehicle system with different bounds of MR damper under different vehicle speeds and random road excitations are discussed.

Bounded time-delay control performance of two-degree-of-freedom nonlinear vehicle system under random road excitation using MR damper

The stochastic vibration control of nonlinear vehicle system under random road excitation is an important research subject. In this paper, based on a more practical and complicated nonlinear vehicle system with torsion-bar-support suspension, the vertical coupling motion between suspension and wheel is considered, and the MR damper is used for the vibration control of the vehicle system under random road excitation. The dynamic differential equations of the two-DOF vehicle system with MR damper are established. Then the HJB equation is established according to the stochastic dynamic programming principle. The feedback control law is designed based on the equation and control boundedness of the MR damper. The control effectiveness is evaluated by comparing the stochastic responses of semi-actively and passively controlled systems. The bounded time-delay control characteristics and the improvement of the bounded time-delay control for the nonlinear stochastic vibration of the two-DOF vehicle system are discussed.

Trajectory control and random vibration control of nonlinear system with dielectric elastomer actuator

Dielectric elastomer actuators have gained extensive attention in scientific and industrial communities with the rapid development of soft robot technology. There still remain some questions on the control aspect of nonlinear system with dielectric elastomer actuator. The first is whether the soft actuator can successfully drive the primary structure to track an arbitrary prescribed trajectory. The second is how to suppress the random vibration around the equilibrium position when the primary structure is disturbed by external excitation. This article seeks the answers for these two questions. By directly solving the governing equation of motion, an open-loop control technique is designed to track a prescribed trajectory. The effectiveness of the trajectory tracking technique is investigated and the limitation is illustrated by the influence of inertia of the primary structure. Based on the stochastic averaging of energy envelope and stochastic dynamic programming principle, a clipped control strategy is proposed by slightly adjusting the voltage in real time to suppress the random vibration around the equilibrium position. The good effectiveness and high robustness of the clipped control strategy are verified numerically. This work may provide some guidelines for the control aspect of nonlinear systems with dielectric elastomer actuators.

Optimal bounded control for nonlinear stochastic smart structure systems based on extended Kalman filter

An optimal bounded control strategy for smart structure systems as controlled Hamiltonian systems with random excitations and noised observations is proposed. The basic dynamic equations for a smart structure system with smart sensors and actuators are firstly given. The nonlinear stochastic control system with noised observations is then obtained from the simplified smart structure system, and the system is expressed by generalized Hamiltonian equations with control, random excitation and dissipative forces. The optimal control problem for nonlinear stochastic systems with noised observations includes two parts: optimal state estimation and optimal response control based on estimated states, which are coupled each other. The probability density of optimally estimated systems has generally infinite dimensions based on the separation theorem. The proposed optimal control strategy gives an approximate separate solution. First, the optimally estimated system state is determined by the observations based on the extended Kalman filter, and the estimated nonlinear system with controls and stochastic excitations is obtained which has finite-dimensional probability density. Second, the dynamical programming equation for the estimated system is determined based on the stochastic dynamical programming principle. The control boundedness due to actuator saturation is considered, and the optimal bounded control law is obtained by the programming equation with the bounded control constraint. The optimal control depends on the estimated system state which is determined by noised observations. The proposed optimal bounded control strategy is finally applied to a single-degree-of-freedom nonlinear stochastic system with control and noised observation. The remarkable vibration control effectiveness is illustrated with numerical results. Thus the proposed optimal bounded control strategy is promising for application to nonlinear stochastic smart structure systems with noised observations.

Optimal Consumption and Portfolio Decision with Heston’s SV Model Under HARA Utility Criterion

Abstract This paper studies the optimal consumption-investment strategy with Heston’s stochastic volatility (SV) model under hyperbolic absolute risk aversion (HARA) utility criterion. The financial market is composed of a risk-less asset and a risky asset, whose price process is supposed to be driven by Heston’s SV model. The risky preference of the individual is assumed to satisfy HARA utility, which recovers power utility, exponential utility and logarithm utility as special cases. HARA utility is of general framework in the utility theory and is seldom studied in the existing literatures. Legendre transform-dual technique along with stochastic dynamic programming principle is presented to deal with our problem and the closed-form solution to the optimal consumption-investment strategy is successfully obtained. Finally, some special cases are derived in detail.

Optimal mean–variance asset-liability management with stochastic interest rates and inflation risks

This paper considers an optimal asset-liability management problem with stochastic interest rates and inflation risks under the mean–variance framework. It is assumed that there are $$n+1$$ assets available in the financial market, including a risk-free asset, a default-free zero-coupon bond, an inflation-indexed bond and $$n-2$$ risky assets (stocks). Moreover, the liability of the investor is assumed to follow a geometric Brownian motion process. By using the stochastic dynamic programming principle and Hamilton–Jacobi–Bellman equation approach, we derive the efficient investment strategy and efficient frontier explicitly. Finally, we provide numerical examples to illustrate the effects of model parameters on the efficient investment strategy and efficient frontier.

A stochastic optimal control strategy for viscoelastic systems with actuator saturation

A nonlinear stochastic optimal control strategy for single degree-of-freedom viscoelastic system with actuator saturation is proposed based on the stochastic averaging method and stochastic dynamical programing principle. First, the viscoelastic system is converted into an equivalent nonlinear non-viscoelastic system by replacing the viscoelastic force with amplitude-dependent stiffness and damping. Second, the partially averaged Itô stochastic differential equation for total system energy as one-dimensional controlled diffusion process is derived by using the stochastic averaging method of energy envelope. For the semi-infinite time-interval ergodic control, the dynamical programming equation is established by applying the stochastic dynamical programming principle to the partially averaged Itô equation with a performance index. The optimal saturated control consisting of unbounded optimal control and bang-bang control is determined by solving the dynamical programming equation. Both the analytical and simulation results verify the higher efficiency and chattering attenuation of the proposed control strategy compared with the corresponding bang-bang control.

Optimal Consumption and Portfolio Decision with Convertible Bond in Affine Interest Rate and Heston’s SV Framework

We are concerned with an optimal investment-consumption problem with stochastic affine interest rate and stochastic volatility, in which interest rate dynamics are described by the affine interest rate model including the Cox-Ingersoll-Ross model and the Vasicek model as special cases, while stock price is driven by Heston’s stochastic volatility (SV) model. Assume that the financial market consists of a risk-free asset, a zero-coupon bond (or a convertible bond), and a risky asset. By using stochastic dynamic programming principle and the technique of separation of variables, we get the HJB equation of the corresponding value function and the explicit expressions of the optimal investment-consumption strategies under power utility and logarithmic utility. Finally, we analyze the impact of market parameters on the optimal investment-consumption strategies by giving a numerical example.

Optimal Vibration Control of a Class of Nonlinear Stochastic Systems with Markovian Jump

The semi-infinite time optimal control for a class of stochastically excited Markovian jump nonlinear system is investigated. Using stochastic averaging, each form of the system is reduced to a one-dimensional partially averaged Itô equation of total energy. A finite set of coupled dynamical programming equations is then set up based on the stochastic dynamical programming principle and Markovian jump rules, from which the optimal control force is obtained. The stationary response of the optimally controlled system is predicted by solving the Fokker-Planck-Kolmogorov (FPK) equation associated with the fully averaged Itô equation. Two examples are worked out in detail to illustrate the application and effectiveness of the proposed control strategy.

Stochastic Optimal Control of Stayed Cable Vibrations with Wide-Band Random Wind Excitation Using Axial Support Motion

This paper proposes an optimal control strategy and corresponding method, to use the advantageous characteristics of the axial support motion to control the stayed cable vibrations in plane by wind excitation. Through a thorough investigation of wide-band random wind excitation and spatial relativity of the wind load along the cable, this study derives realistic and accurate motion equations of the control system. These motion equations are simplified into the form of the first four modes using the Galerkin method. The Itô stochastic differential equations of partial average for the system energy are derived using the stochastic averaging method of the quasi-integrable Hamiltonian systems. Furthermore, the dynamic programming equations with performance index are established by the stochastic dynamic programming principle. The random optimal control law is obtained by solving the dynamic programming equations using the approximate method. The numerical results show that, when standard deviations of the axial support motion are the same, the reduction of the standard deviation of the cable acceleration, and the cable displacement under the stochastic optimal control (SOC) law proposed by this paper, is larger than that under bilinear (BL) control. The SOC method proposed in this paper provides better results than the BL method.

Dynamic mean–variance portfolio selection with liability and stochastic interest rate

This paper is concerned with an asset and liability management problem in a continuous-time mean–variance framework, in which interest rate is driven by the Vasicek model and liability process is governed by Brownian motion with drift. Moreover, interest rate and liability dynamics are generally correlated with stock price dynamics. The objective of the investor is to minimize the variance of terminal net wealth for a given terminal expected net wealth. The explicit solutions of the efficient strategy and the efficient frontier are obtained by applying stochastic dynamic programming principle and Lagrange duality theorem. A numerical example is given to illustrate the results obtained.

Probability‐weighted nonlinear stochastic optimal control strategy of quasi‐integrable Hamiltonian systems with uncertain parameters

SummaryThe nonlinear stochastic optimal control problem of quasi‐integrable Hamiltonian systems with uncertain parameters is investigated. The uncertain parameters are described by using a random vector with λ probability density function. First, the partially averaged Itô stochastic differential equations are derived by using the stochastic averaging method for quasi‐integrable Hamiltonian systems. Then, the dynamical programming equation is established based on stochastic dynamical programming principle. By minimizing the dynamical programming equation with respect to control forces, the optimal control forces can be derived, which are functions of the uncertain parameters. The final optimal control forces are then determined by probability‐weighted average of the obtained control forces with the probability density of the uncertain parameters as weighting function. The mean control effectiveness and mean control efficiency are used to evaluate the proposed control strategy. The robustness of the proposed control is measured by using the ratios of the variation coefficients of mean control effectiveness and mean control efficiency to the variation coefficients of uncertain parameters. Finally, two examples are given to illustrate the proposed control strategy and its effectiveness and robustness. Copyright © 2014 John Wiley & Sons, Ltd.