Stochastic Optimal Control Framework Research Articles

Stochastic Framework for Optimal Control of Planetary Reentry Trajectories Under Multilevel Uncertainties

We present a novel stochastic optimal control framework that accounts for various types of uncertainties, with application to reentry trajectory planning. The formulation of the optimal trajectory control problem is presented in the context of an indirect method where a functional objective associated with the terminal vehicle speed is to be minimized. Uncertain input parameters in the optimal trajectory control model, including aerodynamic parameters and initial and terminal conditions, are modeled as aleatory random variables, while the statistical parameters of these aleatory distributions are themselves random variables. The parametric and model uncertainties are simultaneously propagated through an extended polynomial chaos expansion (EPCE) formalism. Several metrics are described to evaluate response statistics and presented as insightful tools for robust decision making. Specifically, the response probability density function (PDF) reflecting influence of both epistemic and aleatory uncertainties is obtained. By sampling over the random variables representing model error, an ensemble of response PDFs is generated and the associated failure probability is estimated as a random variable with its own polynomial chaos expansion. Besides, the sensitivity index functions of response PDF with respect to the statistical parameters are evaluated. Coupling parametric and model uncertainties within the EPCE framework leads to a robust and efficient paradigm for multilevel uncertainty propagation and PDF characterization in general optimal control problems.

Climate Smart Pest Management

This study examines the role of weather and pest infestation forecasts in agricultural pest management, taking into account potential correlation between weather and pest population prediction errors. First, we analytically illustrate the role of the correlation between weather and pest infestation forecast errors in pest management using a stochastic optimal control framework. Next, using stochastic dynamic programming, we empirically simulate optimal pest management trajectory within a growing season, taking into account correlation between weather and pest population prediction errors. We used lentil production in the Palouse area of northern Idaho and eastern Washington as a case study, where pesticide use was restricted due to environmental or health reasons. We showed that the benefit of applying pesticides later in the growing season can outweigh benefits of early application when pesticide use is restricted due to environmental or health regulations. The value of information is close to $9 per acre, approximately 8% of the expected net returns per acre, and close to $12 per acre ($106–$94), or approximately 13% of the expected net returns per acre for the baseline versus the climate change scenarios, respectively.

An approximate stochastic optimal control framework to simulate nonlinear neuro-musculoskeletal models in the presence of noise.

Optimal control simulations have shown that both musculoskeletal dynamics and physiological noise are important determinants of movement. However, due to the limited efficiency of available computational tools, deterministic simulations of movement focus on accurately modelling the musculoskeletal system while neglecting physiological noise, and stochastic simulations account for noise while simplifying the dynamics. We took advantage of recent approaches where stochastic optimal control problems are approximated using deterministic optimal control problems, which can be solved efficiently using direct collocation. We were thus able to extend predictions of stochastic optimal control as a theory of motor coordination to include muscle coordination and movement patterns emerging from non-linear musculoskeletal dynamics. In stochastic optimal control simulations of human standing balance, we demonstrated that the inclusion of muscle dynamics can predict muscle co-contraction as minimal effort strategy that complements sensorimotor feedback control in the presence of sensory noise. In simulations of reaching, we demonstrated that nonlinear multi-segment musculoskeletal dynamics enables complex perturbed and unperturbed reach trajectories under a variety of task conditions to be predicted. In both behaviors, we demonstrated how interactions between task constraint, sensory noise, and the intrinsic properties of muscle influence optimal muscle coordination patterns, including muscle co-contraction, and the resulting movement trajectories. Our approach enables a true minimum effort solution to be identified as task constraints, such as movement accuracy, can be explicitly imposed, rather than being approximated using penalty terms in the cost function. Our approximate stochastic optimal control framework predicts complex features, not captured by previous simulation approaches, providing a generalizable and valuable tool to study how musculoskeletal dynamics and physiological noise may alter neural control of movement in both healthy and pathological movements.

Finite Time Stability and Optimal Finite Time Stabilization for Discrete-Time Stochastic Dynamical Systems

In this paper, we address finite time stability in probability of discrete-time stochastic dynamical systems. Specifically, a stochastic comparison lemma is constructed along with a scalar system involving a generalized deadzone function to establish almost sure convergence and finite time stability in probability. This result is used to provide Lyapunov theorems for finite time stability in probability for Itô-type stationary nonlinear stochastic difference equations involving Lyapunov difference conditions on the minimum of the Lyapunov function itself along with a fractional power of the Lyapunov function. In addition, we establish sufficient conditions for almost sure lower semicontinuity of the stochastic settling-time capturing the average settling time behavior of the discrete-time nonlinear stochastic dynamical system. Furthermore, a stochastic finite-time optimal control framework is developed by exploiting connections between Lyapunov theory for finite time stability in probability and stochastic Bellman theory. In particular, we show that finite time stability in probability of the closed-loop nonlinear system is guaranteed by means of a Lyapunov function that can clearly be seen to be the solution to the steady state form of the stochastic Bellman equation, and hence, guaranteeing both stochastic finite time stability and optimality.

Nonlinear–nonquadratic optimal and inverse optimal control for discrete‐time stochastic dynamical systems

AbstractIn this article, we investigate the role of Lyapunov functions in evaluating nonlinear–nonquadratic cost functionals for Itô‐type nonlinear stochastic difference equations. Specifically, it is shown that the cost functional can be evaluated in closed‐form as long as the cost functional is related in a specific way to an underlying Lyapunov function that guarantees asymptotic stability in probability. This result is then used to analyze discrete‐time linear as well as nonlinear stochastic dynamical systems with polynomial and multilinear cost functionals. Furthermore, a stochastic optimal control framework is developed by exploiting connections between stochastic Lyapunov theory and stochastic Bellman theory. In particular, we show that asymptotic and geometric stability in probability of the closed‐loop nonlinear system is guaranteed by means of a Lyapunov function that can clearly be seen to be the solution to the steady state form of the stochastic Bellman equation, and hence, guaranteeing both stochastic stability and optimality.

Optimal stochastic compensator for time-delayed active structural control systems subjected to random excitations

Active feedback control systems confront with technical challenges when they are applied in civil engineering structures for vibration mitigation; one of which is the potential instability risk inherent in structure-control systems caused by insufficient compensation of time delay, especially in the scenario considering randomness associated with external excitations. To this end, this paper proposes an optimal stochastic compensator (OSC) to improve time-delayed active structural control systems subjected to random excitations. By extending the state vector, the OSC can be denoted as a linear combination between system state at the present time-step and control forces at the previous time-steps. According to the probabilistic criterion in function of reliability metrics of system state and control force, optimal cost-function weights of the OSC for good trade-off between control gain and control cost can be readily derived by introducing the framework of physically-based stochastic optimal (PSO) control. For verification purposes, randomly base-excited structures controlled by time-delayed active tendon systems, comprising single-degree-of-freedom, multi-degree-of-freedom and nonlinear oscillator systems, are investigated. The analysis and results reveal that the control performance is seriously degraded with increasing time delay; however, a rationally-designed OSC can be used to significantly improve control systems. Moreover, a definite relationship in cubic polynomial functions between optimal control cost-function weight of the OSC and time delay is obtained, by which the problem of time-varying delay compensation can be elegantly solved. In addition, the OSC has a good robustness, and can accommodate a larger effective time delay to linear systems than to nonlinear systems.

Optimal dynamic longevity hedge with basis risk

This paper proposes an optimal dynamic strategy for hedging longevity risk in a discrete-time setting. Our proposed hedging strategy relies on standardized mortality-linked securities and minimizes the variance of the hedging error as induced by the population basis risk. While the formulation of our proposed hedging strategy is quite general, we use a stylized pension plan, together with a specified “yearly rollingǥ trading strategy involving q-forwards and a specified stochastic mortality model, to illustrate our proposed strategy. Under these specifications, we show that the resulting hedging problem can be formulated as a stochastic optimal control framework and that a semi-analytic solution can be derived through an extended Bellman equation. Extensive Monte Carlo studies are conducted to highlight the effectiveness of our proposed hedging strategy. We also consider a scheme to approximate the semi-analytic solution in order to reduce the computational time significantly while still retaining its hedge effectiveness. We benchmark our strategy against the “delta” hedging strategy as well as its robustness to q-forwards maturity, reference age, interest rate, and stochastic mortality models. The proposed strategy has many appealing features, including its discrete-time setting which is consistent with market practice and hence conducive to practical implementation, and its generality in that the underlying hedging principle can be applied to other standardized mortality-linked securities and other stochastic models.

The economics of the Sylvo-Cynegetic equilibrium

This work aims at modeling the economics of the sylvo-cynegetic equilibrium in commercial forests, as defined by forestry authorities, with respect to uncertain damages to plantation stands caused by the overabundance of ungulate species. After presenting the stochastic optimal control framework, in which the dynamics of forest resources and the population dynamics of big game species are both considered, we apply the stochastic control to an updated version of the well-known Gordon-Schaefer bioeconomic model. Our results show that the equilibrium can be achieved through the use of a joint control divided between regeneration works and hunting regimes. Furthermore, we find that, in equilibrium, the investment in forestry works ought to be equal to the willingness to undertake the resource stock stabilization in uncertainty. The investment should amount no more than 20% of total costs engaged in a forestry management plan.

Nonlinear Stochastic Optimal Control with Input Saturation Constraints Based on Path Integrals

AbstractThis paper is concerned with a finite‐time nonlinear stochastic optimal control problem with input saturation as a hard constraint on the control input. The stochastic Hamilton‐Jacobi‐Bellman (SHJB) equation associated with this problem is derived, and a concrete solution method is also presented by extending an iteration framework of path integral stochastic optimal control. This is the first result that provides an optimal feedback control rigorously satisfying the saturation constraint as a solution to the SHJB equation. © 2020 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Optimal Dynamic Longevity Hedge with Basis Risk

This paper proposes an optimal dynamic strategy for hedging longevity risk in a discrete-time setting. Our proposed hedging strategy relies on standardized mortality-linked securities and minimizes the variance of the hedging error as induced by the population basis risk. While the formulation of our proposed hedging strategy is quite general, we use a stylized pension plan, together with a specified yearly rolling” trading strategy involving q-forwards and a specified stochastic mortality model, to illustrate our proposed strategy. Under these specifications, we show that the resulting hedging problem can be formulated as a stochastic optimal control framework and that a semi-analytic solution can be derived through the Bellman equation. Extensive Monte Carlo studies are conducted to highlight the effectiveness of our proposed hedging strategy. We also consider a scheme to approximate the semi-analytic solution in order to reduce the computational time significantly while still retaining its hedging effectiveness. We benchmark our strategy against the delta” hedging strategy as well as its robustness to q-forwards’ maturity, reference age, and stochastic mortality models. The proposed strategy has many appealing features, including its discrete-time setting which is consistent with market practice and hence conducive to practical implementation, and its generality in that the underlying hedging principle can be applied to other standardized mortality-linked securities and other stochastic models.

Efficient computation of optimal open-loop controls for stochastic systems

Optimal control is a prominent approach in robotics and movement neuroscience, among other fields of science. Methods for deriving optimal choices of action have been classically devised either in deterministic or stochastic settings. Here, we consider a setting in-between that retains the stochastic aspect of the controlled system but assumes deterministic open-loop control actions. The rationale stems from observations about the neural control of movement which highlighted that relatively stable behaviors can be achieved without feedback circuitry, via open-loop motor commands adequately tuning the mechanical impedance of the neuromusculoskeletal system. Yet, effective methods for deriving optimal open-loop controls for stochastic systems are lacking overall. This work presents a continuous-time approach based on statistical linearization techniques for the efficient computation of optimal open-loop controls for a broad class of stochastic optimal control problems. We first show that non-trivial departure from the optimal solutions of classical deterministic and stochastic approaches may arise for simple synthetic examples, thereby stressing the originality of the framework. We then exemplify its potential relevance to the planning of biological movement by showing that a well-known phenomenon in motor control, referred to as muscle co-contraction, occurs naturally. More generally, this stochastic optimal control framework may be suited to other fields where the design of optimal open-loop actions is relevant.

Pricing bounds and bang-bang analysis of the Polaris variable annuities

This paper studies the no-arbitrage pricing of the ‘Polaris Income Plus Daily’ structured in the ‘Polaris Choice IV’ variable annuities recently issued by the American International Group. Distinct from the withdrawal benefits studied in the literature, Polaris allows the income base to ‘lock in’ the high water mark of the investment account over a certain monitoring period which is related to the timing of the policyholder's first withdrawal. By prudently introducing certain auxiliary state and decision variables, we manage to formulate the pricing model under a Markovian stochastic optimal control framework. By a slight modification of the fee structure, we show the existence of a bang-bang solution to the stochastic control problem: the optimal withdrawal strategy is among a few explicit choices. We consequently design a novel Least Squares Monte Carlo (LSMC) algorithm to approach the optimal solution. Convergence results are established for the algorithm by applying the theory of nonparametric sieve estimation. Compared with existing LSMCs, our algorithm possesses a number of advantages such as memory reduction, preservation of convexity and monotonicity of the continuation value, reducing computational cost of the tuning parameter selection, and evading extrapolation of the value function estimate. Finally, we prove that the obtained pricing result works as an upper bound of the no-arbitrage price of Polaris with the real fee structure. Numerical experiments show that this upper bound is fairly tight.

A Backward Simulation Method for Stochastic Optimal Control Problems

A number of optimal decision problems with uncertainty can be formulated into a stochastic optimal control framework. The Least-Squares Monte Carlo (LSMC) algorithm is a popular numerical method to approach solutions of such stochastic control problems as analytical solutions are not tractable in general. This paper generalizes the LSMC algorithm proposed in Shen and Weng (2017) to solve a wide class of stochastic optimal control models. Our algorithm has three pillars: a construction of auxiliary stochastic control model, an artificial simulation of the post-action value of state process, and a shape-preserving sieve estimation method which equip the algorithm with a number of merits including bypassing forward simulation and control randomization, evading extrapolating the value function, and alleviating computational burden of the tuning parameter selection. The efficacy of the algorithm is corroborated by an application to pricing equity-linked insurance products.

Pricing Bounds and Bang-Bang Analysis of the Polaris Variable Annuities

This paper studies the no-arbitrage pricing of the Income Plus Daily structured the Choice IV variable annuities recently issued by the American International Group. Distinct from the withdrawal benefits studied the literature, Polaris allows the income base to lock in the high water mark of the investment account over a certain monitoring period which is related to the timing of policyholder's first withdrawal. By prudently introducing certain auxiliary state and decision variables, we manage to formulate the pricing model under a Markovian stochastic optimal control framework. By a slight modification of the fee structure, we show the existence of a bang-bang solution to the stochastic control problem: the optimal withdrawal strategy is among a few explicit choices. We consequently design a novel Least Squares Monte Carlo (LSMC) algorithm to approach the optimal solution. Convergence results are established for the algorithm by applying the theory of nonparametric sieve estimation. Compared with existing LSMCs, our algorithm possesses a number of advantages such as memory reduction, preserving convexity and monotonicity of the continuation value, reducing the computational cost of the tuning parameter selection, and evading extrapolating value function estimate. Finally, we prove that the obtained pricing result works as an upper bound of the no-arbitrage price of Polaris with the real fee structure. Numerical experiments show that this upper bound is fairly tight.

Optimal control of heart rate during treadmill exercise

SummaryFeedback control of heart rate (HR) for treadmills is important for exercise intensity specification and prescription. This work aimed to formulate HR control within a stochastic optimal control framework and to experimentally evaluate controller performance. A quadratic cost function is developed and linked to quantitative performance outcome measures, namely, root‐mean‐square tracking error and average control signal power. An optimal polynomial systems design is combined with frequency‐domain analysis of feedback loop properties, with focus on the input sensitivity function, which governs the response to broad‐spectrum HR variability disturbances. These, in turn, are modelled using stochastic process theory. A simple and approximate model of HR dynamics was used for the linear time‐invariant controller design. Twelve healthy male subjects were recruited for comparative experimental evaluation of 3 controllers, giving 36 tests in total. The mean root‐mean‐square tracking error for the optimal controllers was around 2.2 beats per minute. Significant differences were observed in average control signal power for 2 different settings of the control weighting (mean power 22.6 vs 62.5×10−4 m2/s2, high vs low setting, p=2.3×10−5). The stochastic optimal control framework provides a suitable method for attainment of high‐precision, stable, and robust control of HR during treadmill exercise. The control weighting can be used to set the balance between regulation accuracy and control signal intensity, and it has a clear and systematic influence on the shape of the input sensitivity function. Future work should extend the problem formulation to encompass low‐pass compensator and input sensitivity characteristics.

A group-based traffic signal control with adaptive learning ability

Group-based control is an advanced traffic signal strategy capable of dynamically generating phase sequences at intersections. Combined with the phasing scheme, vehicle actuated timing is often adopted to respond to the detected traffic. However, the parameters of a signal controller are often predetermined in practice, and the control performance may suffer from deterioration when dealing with highly fluctuating traffic demand. This study proposes a group-based signal control approach capable of making decisions based on its understanding of traffic conditions at the intersection level. In particular, the control problem is formulated using a framework of stochastic optimal control for multi-agent system in which each signal group is modeled as an intelligent agent. The agents learn how to react to traffic environment and make optimal timing decisions according to the perceived system states. Reinforcement learning, enhanced by multiple-step backups, is employed as the kernel of the intelligent control algorithm, where each agent updates its knowledge on-line based on a sequence of states during the process. In addition, the proposed system is designated to be compatible with the prevailing signal system. A case study was carried out in a simulation environment to compare the proposed control approach with a benchmark controller used in practice, group-based vehicle actuated (GBVA) controller, whose parameters were off-line optimized using a genetic algorithm. Simulation results show that the proposed adaptive group-based control system outperforms the optimized GBVA control system mainly because of its real-time adaptive learning capacity in response to the changes in traffic demand.

Minimization of carbon abatement cost: Modeling, analysis and simulation

In this paper, we consider a problem of minimizing the carbon abatement cost of a country. Two models are built within the stochastic optimal control framework based on two types of abatement policies. The corresponding HJB equations are deduced, and the existence and uniqueness of their classical solutions are established by PDE methods. Using parameters in the models obtained from real data, we carried out numerical simulations via semi-implicit method. Then we discussed the properties of the optimal policies and minimal costs. Our results suggest that a country needs to keep a relatively low economy and population growth rate and keep a stable economy in order to reduce the total carbon abatement cost. In the long run, it's better for a country to seek for more efficient carbon abatement techniques and an environmentally friendly way of economic development.

Multi-usage hydropower single dam management: chance-constrained optimization and stochastic viability

We consider the management of a single hydroelectric dam, subject to uncertain inflows and electricity prices and to a so-called “tourism constraint”: the water storage level must be high enough during the tourist season with high enough probability. We cast the problem in the stochastic optimal control framework: we search at each time t the optimal control as a function of the available information at t. We lay out two approaches. First, we formulate a chance-constrained stochastic optimal control problem: we maximize the expected gain while guaranteeing a minimum storage level with a minimal prescribed probability level. Dualizing the chance constraint by a multiplier, we propose an iterative algorithm alternating additive dynamic programming and update of the multiplier value “a la Uzawa”. Our numerical results reveal that the random gain is very dispersed around its expected value; in particular, low gain values have a relatively high probability to materialize. This is why, to put emphasis on these low values, we outline a second approach. We propose a so-called stochastic viability approach that focuses on jointly guaranteeing a minimum gain and a minimum storage level during the tourist season. We solve the corresponding problem by multiplicative dynamic programming. To conclude, we discuss and compare the two approaches. Keywords

A generalizable adaptive brain-machine interface design for control of anesthesia.

Brain-machine interfaces (BMIs) for closed-loop control of anesthesia have the potential to automatically monitor and control brain states under anesthesia. Since a variety of anesthetic states are needed in different clinical scenarios, designing a generalizable BMI architecture that can control a wide range of anesthetic states is essential. In addition, drug dynamics are non-stationary over time and could change with the depth of anesthesia. Hence for precise control, a BMI needs to track these non-stationarities online. Here we design a BMI architecture that generalizes to control of various anesthetic states and their associated neural signatures, and is adaptive to time-varying drug dynamics. We provide a systematic approach to build general parametric models that quantify the anesthetic state and describe the drug dynamics. Based on these models, we develop an adaptive closed-loop controller within the framework of stochastic optimal feedback control. This controller tracks the non-stationarities in drug dynamics, achieves tight control in a time-varying environment, and removes the need for an offline system identification session. For robustness, the BMI also ensures small drug infusion rate variations at steady state. We test the BMI architecture for control of two common anesthetic states, i.e., burst suppression in medically-induced coma and unconsciousness in general anesthesia. Using numerical experiments, we find that the BMI generalizes to control of both these anesthetic states; in a time-varying environment, even without initial knowledge of model parameters, the BMI accurately controls these two different anesthetic states, reducing bias and error more than 70 times and 9 times, respectively, compared with a non-adaptive system.

Nonlinear Stochastic Control and Information Theoretic Dualities: Connections, Interdependencies and Thermodynamic Interpretations

In this paper, we present connections between recent developments on the linearly-solvable stochastic optimal control framework with early work in control theory based on the fundamental dualities between free energy and relative entropy. We extend these connections to nonlinear stochastic systems with non-affine controls by using the generalized version of the Feynman–Kac lemma. We present alternative formulations of the linearly-solvable stochastic optimal control framework and discuss information theoretic and thermodynamic interpretations. On the algorithmic side, we present iterative stochastic optimal control algorithms and applications to nonlinear stochastic systems. We conclude with an overview of the frameworks presented and discuss limitations, differences and future directions.