Stochastic Optimal Control Method Research Articles

A novel stochastic power flow calculation and optimal control method for microgrid based on multivariate stochastic factors fusion – Sensitivity

The stochasticity of power flow of distributed generations (DGs) and load in the microgrid has great influence on power flow distribution and voltage quality of the distribution network. For improving the voltage quality of the distribution network, the questions need to be further studied, which include the description of the stochasticity of the power flow in the microgrid and the impact of the microgrid into the distribution network on the power flow. Therefore, a novel stochastic power flow calculation and optimal control method for the microgrid based on multivariate stochastic factors fusion-sensitivity (MSFF-sensitivity) is proposed in this paper. Firstly, the multivariate stochastic factors fusion (MSFF) function is developed by using the probability density function to extract the stochasticity and correlation of power flow among different stochastic factors in the microgrid, which are effectively unified. Furthermore, the fusion-sensitivity (F-sensitivity) of the power flow in the microgrid integrated into the distribution network is constructed to accurately characterize the influence degree of various stochastic factors in the microgrid on the power flow of the distribution network. Based on this, the output power of the stochastic factor is adjusted to optimally control the power flow of the distribution network. Finally, the algorithm verification suggests that, compared with the conventional power flow methods, the method proposed in this paper is more suitable for the microgrid. The influence of stochastic power flow on the distribution network can be effectively reduced and the voltage quality of the distribution network can be improved by optimizing control of the power flow in the microgrid integrated into the distribution network.

Stochastic Optimal Control for Robot Manipulation Skill Learning Under Time-Varying Uncertain Environment.

In this article, a novel stochastic optimal control method is developed for robot manipulator interacting with a time-varying uncertain environment. The unknown environment model is described as a nonlinear system with time-varying parameters as well as stochastic information, which is learned via the Gaussian process regression (GPR) method as the external dynamics. Integrating the learned external dynamics as well as the stochastic uncertainties, the complete interaction system dynamics are obtained. Then the iterative linear quadratic Gaussian with learned external dynamics (ILQG-LEDs) method is presented to obtain the optimal manipulation control parameters, namely, the feedforward force, the reference trajectory, as well as the impedance parameters, subject to time-varying environment dynamics. The comparative simulation studies verify the advantages of the presented method, and the experimental studies of the peg-hole-insertion task prove that this method can deal with complex manipulation tasks.

Stochastic control methods for optimization problems in Ornstein-Uhlenbeck spread models

We develop stochastic optimal control methods for spread financial models defined by the Ornstein–Uhlenbeck (OU) processes. To this end, we study the Hamilton - Jacobi - Bellman (HJB) equation using the Feynman - Kac (FK) probability representation. We show an existence and uniqueness theorem for the classical solution of the HJB equation, a quasi-linear partial derivative equation of parabolic type. Then we show a special verification theorem and, as a consequence, construct optimal consumption/investment strategies for power utility functions. Moreover, using fixed point tools we study the numeric approximation for the HJB solution and we establish the convergence rate which, as it turns out in this case, is super geometric, i.e., more rapid than any geometric one. Finally, we illustrate numerically the behavior of the obtained strategies.

Target benefit pension plan with longevity risk and intergenerational equity

AbstractWe study a stochastic model for a target benefit pension plan suffering from rising longevity and falling fertility. Policies for postponing retirement are carried out to hedge the payment difficulties caused by the aging population. The plan members’ contributions are set in advance while the pension payments reflect intergenerational equity by a target payment level and intergenerational risk sharing by an adjustment. The pension fund is invested in both a risk-free asset and a risky asset. Applying the stochastic optimal control methods, we derive analytic solutions for optimal investment and benefit payment strategies which minimize the benefit risk. Besides, an optimal delayed retirement age which can hedge against the aging phenomenon under certain parameters is given. Therefore, it can provide a basis for quantifying the delay of retirement time.

Decentralized Stochastic Disturbance Observer-Based Optimal Frequency Control Method for Interconnected Power Systems With High Renewable Shares

This article proposes a novel, disturbance observer-based decentralized frequency control method for interconnected power systems. The method employs extended Kalman filter (EKF) as an observer to estimate inaccessible dynamic states of the system, including the total disturbance as one of the state variables. An optimal decentralized disturbance observer based controller is suggested for multiarea power systems that compensates the estimated disturbance and further based on minimizing the joint error energies of state estimation error and state tracking error provides its value to the controller for regulating the frequency variation. The proposed method is mathematically designed to be robust against parametric and nonparametric uncertainties. The efficacy and accuracy of the proposed control method is verified considering different types of practical operation scenarios. The results confirm the brilliant and superiority of the proposed method in controlling the frequency in power systems with high renewable shares.

The Two-Stage PI2 Control Strategy

PI <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> is a stochastic optimal control method generally regarded as a reinforcement learning algorithm. Recent work, however, suggests that the reinforcement learning aspect of PI <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> actually appears when optimizing feedforward controls which will lead to optimal closed-loop performance once combined with feedback controls. These feedbacks are necessary to achieve the predicted performance, yet have been largely neglected in the literature and applications due to their complexity. In this letter, we show that the feedbacks actually take a simple-to-implement form for a wide range of system dynamics, paving way for future research and applications of PI <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> . The correctness of the results is demonstrated through numerical simulations.

Reliability-based stochastic optimal control of frame building under near-fault ground motions

Most of stochastic optimal control methods were developed on the basis of Itô stochastic differential equation, which assumes that the external excitation is white Gaussian noise or filtered white Gaussian noise. However, this assumption is far from the real excitations, which hinders the applications of stochastic optimal control theory. In this paper, the reliability-based stochastic optimal control via direct probability integral method (DPIM) for building structure is proposed, which is applicable to performance-based design of general control systems of linear structures under non-stationary and non-white random excitations. Firstly, the DPIM is utilized to accurately and efficiently compute dynamic reliability of the controlled system. Then, the reliability-based objective function is suggested to optimize the parameters of the placed control devices of building structure, and reliability-based maximum story controllability index is established to determine the optimal placement of control devices, thus fulfilling the stochastic optimal control of structure. Finally, a 10-story shear frame building controlled by active tendons under random excitations of near-fault earthquake ground motions verifies the effectiveness of the proposed optimal control method. Moreover, the optimal control schemes of frame structure under near-fault non-pulse and impulsive ground motions are compared, indicating that the velocity pulses of near-fault ground motions impose significant effect on structural responses, and the more control energy is needed for guaranteeing structural safety.

Data-Driven Distributionally Robust Stochastic Control of Energy Storage for Wind Power Ramp Management Using the Wasserstein Metric

The integration of wind energy into the power grid is challenging because of its variability, which causes high ramp events that may threaten the reliability and efficiency of power systems. In this paper, we propose a novel distributionally robust solution to wind power ramp management using energy storage. The proposed storage operation strategy minimizes the expected ramp penalty under the worst-case wind power ramp distribution in the Wasserstein ambiguity set, a statistical ball centered at an empirical distribution obtained from historical data. Thus, the resulting distributionally robust control policy presents a robust ramp management performance even when the future wind power ramp distribution deviates from the empirical distribution, unlike the standard stochastic optimal control method. For a tractable numerical solution, a duality-based dynamic programming algorithm is designed with a piecewise linear approximation of the optimal value function. The performance and utility of the proposed method are demonstrated and analyzed through case studies using the wind power data in the Bonneville Power Administration area for the year 2018.

Learning from demonstration for locally assistive mobility aids

Active assistive systems for mobility aids are largely restricted to environments mapped a-priori, while passive assistance primarily provides collision mitigation and other hand-crafted behaviors in the platform’s immediate space. This paper presents a framework providing active short-term assistance, combining the freedom of location independence with the intelligence of active assistance. Demonstration data consisting of on-board sensor data and driving inputs is gathered from an able-bodied expert maneuvring the mobility aid around a generic interior setting, and used in constructing a probabilistic intention model built with Radial Basis Function Networks. This allows for short-term intention prediction relying only upon immediately available user input and on-board sensor data, to be coupled with real-time path generation based upon the same expert demonstration data via Dynamic Policy Programming, a stochastic optimal control method. Together these two elements provide a combined assistive mobility system, capable of operating in restrictive environments without the need for additional obstacle avoidance protocols. Experimental results in both simulation and on the University of Technology Sydney semi-autonomous wheelchair in settings not seen in training data show promise in assisting users of power mobility aids.

Optimal Incentive Contract for Sales Team with Loss Aversion Preference

When manufacturing enterprises employ sales team (or multiple salesmen) to sell products, there is asymmetric information such as the ability and efforts salesmen. Enterprises can use contracts to incentivize salesmen to work hard to maximize their profits. Assuming that market demand is sensitive to effort, and the salesman can exploit the market by increasing effort, a multi-agent model is established for the case of symmetrical information and asymmetrical information, in which the sales team has a loss aversion preference. In this multi-agent model, the agents’ utility function is non-concave and cannot be solved by traditional methods. We use a backward stochastic differential equation (BSDE) to represent agents’ contract through the martingale representation theorem and use the stochastic optimal control and matrix method to obtain the explicit solution of the optimal contract. Based on the conclusions of the research, an empirical analysis is made on the sales team of an enterprise.

A Stochastic Optimal Control Method Using Statistical Information and Binary Variables with an Application to Online Optimal Mode Management for Plug-in Hybrid Vehicles

A Stochastic Optimal Control Method Using Statistical Information and Binary Variables with an Application to Online Optimal Mode Management for Plug-in Hybrid Vehicles

Data-Driven Stochastic Optimal Iterative Learning Control for Nonlinear Non-Affine Systems With Measurement Data Loss

In this paper, the measurement data loss is considered and two data-driven stochastic optimal iterative learning control (DDSOILC) methods are presented directly for nonlinear network systems. Specifically, an iterative dynamic linearization (IDL) is adopted to construct the linear incremental input output relationship of the repetitive nonlinear network system between two consecutive iterations. In the sequel, a lifted IDL is obtained by defining two super vectors of inputs and outputs over the entire finite time interval. Then, a lifted IDL-based DDSOILC scheme is proposed where the random data loss is described by a Bernoulli distribution of random variable. The results are extended by using a non-lifted IDL where the input-output relationship is described pointwisely. The learning gains of the proposed two methods are iteration-time-variant and can be iteratively estimated using real-time data. The proposed two methods do not depend on any explicit model. Moreover, the proposed non-lifted IDL-based DDSOILC can use more control information than the proposed lifted IDL-based one, and thus it can achieve a better control performance. Both theoretical analysis and simulations verify the efficiency and applicability of the two proposed methods.

Asset liability management for an ordinary insurance system with proportional reinsurance in a CIR stochastic interest rate and Heston stochastic volatility framework

<p style='text-indent:20px;'>This paper investigates the asset liability management problem for an ordinary insurance system incorporating the standard concept of proportional reinsurance coverage in a stochastic interest rate and stochastic volatility framework. The goal of the insurer is to maximize the expectation of the constant relative risk aversion (CRRA) of the terminal value of the wealth, while the goal of the reinsurer is to maximize the expected exponential utility (CARA) of the terminal wealth held by the reinsurer. We assume that the financial market consists of risk-free assets and risky assets, and both the insurer and the reinsurer invest on one risk-free asset and one risky asset. By using the stochastic optimal control method, analytical expressions are derived for the optimal reinsurance control strategy and the optimal investment strategies for both the insurer and the reinsurer in terms of the solutions to the underlying Hamilton-Jacobi-Bellman equations and stochastic differential equations for the wealths. Subsequently, a semi-analytical method has been developed to solve the Hamilton-Jacobi-Bellman equation. Finally, we present numerical examples to illustrate the theoretical results obtained in this paper, followed by sensitivity tests to investigate the impact of reinsurance, risk aversion, and the key parameters on the optimal strategies.

Secondary Frequency Stochastic Optimal Control in Independent Microgrids with Virtual Synchronous Generator-Controlled Energy Storage Systems

With the increasing proportion of renewable energy in microgrids (MGs), its stochastic fluctuation of output power has posed challenges to system safety and operation, especially frequency stability. Virtual synchronous generator (VSG) technology, as one effective method, was used to smoothen frequency fluctuation and improve the system’s dynamic performance, which can simulate the inertia and damping of the traditional synchronous generator. This study outlines the integration of VSG-controlled energy storage systems (ESSs) and traditional synchronous generators so they jointly participate in secondary frequency regulation in an independent MG. Firstly, a new uncertain state-space model for secondary frequency control is established, considering the measurement noises and modelling error. Then, an improved linear quadratic Gaussian (LQG) controller is designed based on stochastic optimal control theory, in which the dynamic performance index weighting matrices are optimized by combining loop transfer recovery (LTR) technology and the distribution estimation algorithm. On the issue of secondary frequency devices’ output power allocation, the dynamic participation factors based on the ESS’s current state of charge (SOC) are proposed to prevent the batteries’ overcharging and overdischarging problems. The energy storage devices’ service life can be prolonged and OPEX (operational expenditure) decreased. Multiple experimental scenarios with real parameters of MGs are employed to evaluate the performance of the proposed algorithm. The results show that, compared with the lead-compensated-proportional-integral-derivative (LC-PID) control and robust μ-control algorithms, the proposed stochastic optimal control method has a faster dynamic response and is more robust, and the fluctuations from renewable energy and power loads can be smoothened more effectively.

An Optimal Investment Strategy and Multiperiod Deposit Insurance Pricing Model for Commercial Banks

We employ the method of stochastic optimal control to derive the optimal investment strategy for maximizing an expected exponential utility of a commercial bank’s capital at some future date T&gt;0. In addition, we derive a multiperiod deposit insurance (DI) pricing model that incorporates the explicit solution of the optimal control problem and an asset value reset rule comparable to the typical practice of insolvency resolution by insuring agencies. By way of numerical simulations, we study the effects of changes in the DI coverage horizon, the risk associated with the asset portfolio of the bank, and the bank’s initial leverage level (deposit-to-asset ratio) on the DI premium while the optimal investment strategy is followed.

State-Regularized Policy Search for Linearized Dynamical Systems

Trajectory-Centric Reinforcement Learning and Trajectory Optimization methods optimize a sequence of feedback-controllers by taking advantage of local approximations of model dynamics and cost functions. Stability of the policy update is a major issue for these methods, rendering them hard to apply for highly nonlinear systems. Recent approaches combine classical Stochastic Optimal Control methods with information-theoretic bounds to control the step-size of the policy update and could even be used to train nonlinear deep control policies. These methods bound the relative entropy between the new and the old policy to ensure a stable policy update. However, despite the bound in policy space, the state distributions of two consecutive policies can still differ significantly, rendering the used local approximate models invalid. To alleviate this issue we propose enforcing a relative entropy constraint not only on the policy update, but also on the update of the state distribution, around which the dynamics and cost are being approximated. We present a derivation of the closed-form policy update and show that our approach outperforms related methods on two nonlinear and highly dynamic simulated systems.

Offset control of traffic signal using cellular automaton traffic model

This paper deals with an offset control of traffic signals using a cellular automaton traffic model. A stochastic optimal control method for distributed traffic signals is modified to achieve coordinated traffic signal control with the proposed offset control method. In the proposed coordinated traffic signal control method, splits of each cycle and common cycle length are calculated using a modified stochastic optimal control method, and then the offset is calculated using an estimation method based on a modified CA traffic model at intervals. Also, simulations are carried out at multiple intersections using a micro traffic simulator. The effectiveness of the proposed coordinated control method is proved by comparing with other traffic signal control methods such as pre-timed signal control, two types of the traditional coordinated control and distributed control.

A stochastic averaging-based optimal control method for nonlinear systems: Application to a building with soil-structure interactions

This paper presents a nonlinear stochastic optimal control algorithm based on an enhanced stochastic averaging method of energy envelope to mitigate the response of nonlinear hysteretic systems. The enhanced stochastic averaging strategy for non-integrable hysteretic systems addresses some of the shortcomings of conventional methods including consideration of a collective contribution of damping terms and the assumption of the independence of stochastic excitations. Hamilton-Jacobi-Bellman equation is used to formulate the optimization problem with a cost function that incorporates high order terms of the energy of the system in order to optimize the dynamic response. Unlike other techniques, this strategy considers the stochastic nonlinear behavior of the system in the optimization process of the control design. The performance of the proposed controller is evaluated for a one-story building on a nonlinear foundation under white noise and historic ground motions. The results are compared with those for the uncontrolled structure and the structure controlled using a linear quadratic regulator algorithm designed using stochastic linearization (SLQR). The proposed method is shown to provide considerable improvements over other control cases. In particular under Kobe ground motion, 14% and 21% reductions are observed for the maximum relative displacement and foundation displacement, respectively compared to the SLQR with similar maximum control force levels.

Application of Model Predictive Control to Control Transient Behavior in Stochastic Manufacturing System Models

Increasing complexity in manufacturing strategies and swift changes in market and consumer requirements have driven recent studies of manufacturing systems, with transient behavior being identified as a key research area. Till date, satisfying consumer demand has focused on steady-state planning of production, mostly using stochastic or deterministic optimal control methods. Due to the difficulty of obtaining optimal control for many practical situations, as well as in evaluating performance under optimal control, these studies have not been conducive to the analysis or control of transient behavior. This paper bridges this gap by applying model predictive control to a manufacturing system modeled as a discrete-time Markov chain. By modifying the initiation of production as probabilities within the Markov chain, a method is proposed to directly control the system to specific expected performance levels and improve its stochastic transient behavior.

Near-Optimal Control for Aircraft Conflict Resolution in the Presence of Uncertainty

In this paper, a stochastic near-optimal control method is proposed for determining aircraft conflict-resolution trajectories in the presence of uncertainty in real time. The prior work developed a stochastic optimal control method for aircraft conflict resolution based on the polynomial chaos expansion and pseudospectral methods. This stochastic optimal control method is extended to generate conflict-resolution trajectories in real time without actually solving the computationally expensive stochastic optimal control problems. The proposed near-optimal conflict-resolution algorithm is based on a recently developed surrogate modeling technique called polynomial chaos kriging, which is used to construct the surrogate models of the optimal conflict-resolution trajectories from a set of precomputed optimal solutions. The near-optimal conflict-resolution trajectories can be accurately generated in real time from the surrogate models with the information of current conditions (for example, current states). Through illustrative aircraft conflict-resolution examples, the performance and effectiveness of the proposed stochastic near-optimal conflict-resolution algorithm are evaluated and demonstrated.