Stochastic Control Algorithm Research Articles

On the Inherent Distributional Robustness of Stochastic and Nominal Model Predictive Control

We define a notion of distributional robustness, via the Wasserstein metric, for closed-loop systems subject to errors in the disturbance distribution used to construct the controller. We then establish sufficient conditions for stochastic model predictive control (SMPC) to satisfy this definition of distributional robustness and establish a similar notion of distributional robustness for economic applications of SMPC. These results address incorrectly or unmodeled disturbances, demonstrate the efficacy of scenario optimization as a means to approximate and solve the SMPC problem, and unify the descriptions of robustness for stochastic and nominal model predictive control. This definition of distributional robustness for closed-loop systems is general and can be applied to other stochastic optimal control algorithms and, potentially, the developing field of distributionally robust control.

A distributed stochastic approximation algorithm for stochastic LQ control with unknown uncertainty

This paper studies a discrete-time stochastic control problem with linear quadratic criteria over an infinite-time horizon. We focus on control systems whose system matrices are associated with random parameters involving unknown statistical properties. We design a distributed stochastic approximation algorithm to tackle the Riccati equation and derive the optimal controller stabilizing the system. The convergence analysis is provided.

Validation of Stochastic Optimal Control Models for Goal-Directed Human Movements on the Example of Human Driving Behavior

Stochastic Optimal Control models represent the state-of-the-art in modeling goal-directed human movements. The linear-quadratic sensorimotor (LQS) model based on signal-dependent noise processes in state and output equation is the current main representative. With our newly introduced Inverse Stochastic Optimal Control algorithm building upon two bi-level optimizations, we can identify its unknown model parameters, namely cost function matrices and scaling parameters of the noise processes, for the first time. In this paper, we use this algorithm to identify the parameters of a deterministic linear-quadratic, a linear-quadratic Gaussian and a LQS model from human measurement data to compare the models’ capability in describing goal-directed human movements. Human steering behavior in a simplified driving task shown to posses similar features as point-ot-point human hand reaching movements serves as our example movement. The results show that the identified LQS model outperforms the others with statistical significance. Particularly, the average human steering behavior is modeled significantly better by the LQS model. This validates the positive impact of signal-dependent noise processes on modeling human average behavior.

A Distributionally Robust Optimization Based Method for Stochastic Model Predictive Control

Two stochastic model predictive control algorithms, which are referred to as distributionally robust model predictive control algorithms, are proposed in this article for a class of discrete linear systems with unbounded noise. Participially, chance constraints are imposed on both of the state and the control, which makes the problem more challenging. Inspired by the ideas from distributionally robust optimization (DRO), two deterministic convex reformulations are proposed for tackling the chance constraints. Rigorous computational complexity analysis is carried out to compare the two proposed algorithms with the existing methods. Recursive feasibility and convergence are proven. Simulation results are provided to show the effectiveness of the proposed algorithms.

Regression Monte Carlo for Impulse Control

I develop a numerical algorithm for stochastic impulse control in the spirit of Regression Monte Carlo for optimal stopping. The approach consists in generating statistical surrogates (aka functional approximators) for the continuation function. The surrogates are recursively trained by empirical regression over simulated state trajectories. In parallel, the same surrogates are used to learn the intervention function characterizing the optimal impulse amounts. I discuss appropriate surrogate types for this task, as well as the choice of training sets. Case studies from forest rotation and irreversible investment illustrate the numerical scheme and highlight its flexibility and extensibility. Implementation in R is provided as a publicly available package posted on GitHub.

Constrained stochastic optimal control with learned importance sampling: A path integral approach

Modern robotic systems are expected to operate robustly in partially unknown environments. This article proposes an algorithm capable of controlling a wide range of high-dimensional robotic systems in such challenging scenarios. Our method is based on the path integral formulation of stochastic optimal control, which we extend with constraint-handling capabilities. Under our control law, the optimal input is inferred from a set of stochastic rollouts of the system dynamics. These rollouts are simulated by a physics engine, placing minimal restrictions on the types of systems and environments that can be modeled. Although sampling-based algorithms are typically not suitable for online control, we demonstrate in this work how importance sampling and constraints can be used to effectively curb the sampling complexity and enable real-time control applications. Furthermore, the path integral framework provides a natural way of incorporating existing control architectures as ancillary controllers for shaping the sampling distribution. Our results reveal that even in cases where the ancillary controller would fail, our stochastic control algorithm provides an additional safety and robustness layer. Moreover, in the absence of an existing ancillary controller, our method can be used to train a parametrized importance sampling policy using data from the stochastic rollouts. The algorithm may thereby bootstrap itself by learning an importance sampling policy offline and then refining it to unseen environments during online control. We validate our results on three robotic systems, including hardware experiments on a quadrupedal robot.

Impedance Optimization for Uncertain Contact Interactions Through Risk Sensitive Optimal Control

This letter addresses the problem of computing optimal impedance schedules for legged locomotion tasks involving complex contact interactions. We formulate the problem of impedance regulation as a trade-off between disturbance rejection and measurement uncertainty. We extend a stochastic optimal control algorithm known as Risk Sensitive Control to take into account measurement uncertainty and propose a formal way to include such uncertainty for unknown contact locations. The approach can efficiently generate optimal state and control trajectories along with local feedback control gains, i.e. impedance schedules. Extensive simulations demonstrate the capabilities of the approach in generating meaningful stiffness and damping modulation patterns before and after contact interaction. For example, contact forces are reduced during early contacts, damping increases to anticipate a high impact event and tracking is automatically traded-off for increased stability. In particular, we show a significant improvement in performance during jumping and trotting tasks with a simulated quadruped robot.

Optimal minimum variance-entropy control of tumour growth processes based on the Fokker-Planck equation.

The authors demonstrated an optimal stochastic control algorithm to obtain desirable cancer treatment based on the Gompertz model. Two external forces as two time-dependent functions are presented to manipulate the growth and death rates in the drift term of the Gompertz model. These input signals represent the effect of external treatment agents to decrease tumour growth rate and increase tumour death rate, respectively. Entropy and variance of cancerous cells are simultaneously controlled based on the Gompertz model. They have introduced a constrained optimisation problem whose cost function is the variance of a cancerous cells population. The defined entropy is based on the probability density function of affected cells was used as a constraint for the cost function. Analysing growth and death rates of cancerous cells, it is found that the logarithmic control signal reduces the growth rate, while the hyperbolic tangent-like control function increases the death rate of tumour growth. The two optimal control signals were calculated by converting the constrained optimisation problem into an unconstrained optimisation problem and by using the real-coded genetic algorithm. Mathematical justifications are implemented to elucidate the existence and uniqueness of the solution for the optimal control problem.

Modeling and Control Using Stochastic Distribution Control Theory for Intersection Traffic Flow

This work investigated stochastic distribution control theory-based traffic signal optimization to achieve a smooth and uniform flow of vehicles through signalized intersections. In this context, the static and linear dynamic stochastic distribution models were developed to express the relationship between the signal timing and the traffic queue length together with its probability density function. Two stochastic distribution control algorithms were designed to control the signal timing at intersections such that the probability density function of the traffic queue of each intersection road segment is made as narrow and as small as possible. Also, a recursive input-output traffic queue estimation model was proposed, which is data-driven and dynamic in nature, to calculate real-time traffic queue length using traffic signal timings and loop-detector data. The control algorithms were evaluated for a one-signal corridor, two-signal corridor, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2 \times 2$ </tex-math></inline-formula> network of signalized intersections. MATLAB simulation examples are provided to demonstrate the use of the proposed algorithms and comparison to the existing widely-used semi-actuated control has been made. Desired results were obtained.

A Personalized Human Drivers’ Risk Sensitive Characteristics Depicting Stochastic Optimal Control Algorithm for Adaptive Cruise Control

This paper presents a personalized stochastic optimal adaptive cruise control (ACC) algorithm for automated vehicles (AVs) incorporating human drivers’ risk-sensitivity under system and measurement uncertainties. The proposed controller is designed as a linear exponential-of-quadratic Gaussian (LEQG) problem, which utilizes the stochastic optimal control mechanism to feedback the deviation from the design car-following target. With the risk-sensitive parameter embedded in LEQG, the proposed method has the capability to characterize risk preference heterogeneity of each AV against uncertainties according to each human drivers’ preference. Further, the established control theory can achieve both expensive control mode and non-expensive control mode via changing the weighting matrix of the cost function in LEQG to reveal different treatments on input. Simulation tests validate the proposed approach can characterize different driving behaviors and its effectiveness in terms of reducing the deviation from equilibrium state. The ability to produce different trajectories and generate smooth control of the proposed algorithm is also verified.

H-Infinity Control of an Adaptive Hybrid Active Power Filter for Power Quality Compensation

This article highlights an optimal robust control technique called H-infinity, which thanks to a particular algorithm offers several solutions in the experimental implementation of harmonic compensators of systems with API-siemens modules. This control and command technique is directly tested on a TLC adaptive hybrid filter topology that provides benefits, such as reduced switching losses when injecting currents in the network, limitation of resonance problems and above all low power consumption at the DC bus level, thus allowing us to obtain results for 105 V to be compared with existing models in the literature which require 600 V for the same performance. This article therefore simultaneously offers two essential contributions to the optimization of harmonic pollution control. A first contribution is essentially based on the H-infinite algorithm and its particularity in its implementation on our TLC hybrid model. The second is on the advantages offered by the TLC-HAPF hybrid topology. The results obtained with this algorithm give us THDs conforming to the IEEE 519-1996 and which are very meaningful compared to the results obtained with other robust and stochastic control algorithms taken under the same conditions.

Inexact Adjoint-based SQP Algorithm for Real-Time Stochastic Nonlinear MPC

This paper presents a real-time algorithm for stochastic nonlinear model predictive control (NMPC). The optimal control problem (OCP) involves a linearization based covariance matrix propagation to formulate the probabilistic chance constraints. Our proposed solution approach uses a tailored Jacobian approximation in combination with an adjoint-based sequential quadratic programming (SQP) method. The resulting algorithm allows the numerical elimination of the covariance matrices from the SQP subproblem, while ensuring Newton-type local convergence properties and preserving the block-sparse problem structure. It allows a considerable reduction of the computational complexity and preserves the positive definiteness of the covariance matrices at each iteration, unlike an exact Jacobian-based implementation. The realtime feasibility and closed-loop control performance of the proposed algorithm are illustrated on a case study of an autonomous driving application subject to external disturbances.

Iterative Learning Stochastic MPC with Adaptive Constraint Tightening for Building HVAC Systems

Most of the existing stochastic model predictive control (SMPC) algorithms for systems subject to random disturbance are designed offline using the distribution information of the uncertainties. In this paper, we propose an iterative learning based MPC for systems subject to time varying stochastic constraints on states. Different from those existing offline design approaches, except for the boundedness, this algorithm does not require to know the distributions or statistics such as the covariances of the uncertainties and the parameters of the controllers are adjusted online using the observations of past state trajectories. By making use of the iterative nature of the process, pointwise in time stochastic constraints are enforced so that it can handle time-varying constraints. Under some proper assumptions, this iterative procedure is shown to be equivalent to a root-searching problem and stochastic approximation theory is applied to show that the empirical average converges to the prescribed expectation in probability. The proposed algorithm is applied to an HVAC control problem to show its effectiveness.

Differential Game for Distributed Power Control in Device-to-Device Communications Underlaying Cellular Networks

In the formulating of power control for wireless networks, the radio channel is commonly formulated using static models of optimization or game theory. In these models, the optimization programming or static game is played from time point to time point. Therefore, this approach neglects the dynamics of the time-varying channels and assumes the statistical independence between the successive time points. In this paper, we utilize differential equations to model the wireless links, then formulate a differential game for the power control problem in device-to-device (D2D) communications underlaying cellular networks. The game players are the D2D pairs, which manage their transmit power by solving the continuous-time optimal control problems. The time-dependant cost function allows us to optimize the long-term expected cost, instead of point-wise instantaneous cost. We formulate the problem in an affine quadratic form that admits analytical solutions. The unique feedback Nash equilibrium of the game is shown to exist. From a stochastic optimal control algorithm, we design a distributed power control mechanism that converges to the game’s equilibrium. The simulation results show that the proposed approach achieves significant performance improvement compared to the point-wise based approaches.

A Unique Approach to 3D Localization in Wireless Sensor Network by Using Adaptive Stochastic Control Algorithm,

A Unique Approach to 3D Localization in Wireless Sensor Network by Using Adaptive Stochastic Control Algorithm,

Stochastic Predictive Control for Partially Observable Markov Decision Processes With Time-Joint Chance Constraints and Application to Autonomous Vehicle Control

This paper describes a stochastic predictive control algorithm for partially observable Markov decision processes (POMDPs) with time-joint chance constraints. We first present the algorithm as a general tool to treat finite space POMDP problems with time-joint chance constraints together with its theoretical properties. We then discuss its application to autonomous vehicle control on highways. In particular, we model decision-making/behavior-planning for an autonomous vehicle accounting for safety in a dynamic and uncertain environment as a constrained POMDP problem and solve it using the proposed algorithm. After behavior is planned, we use nonlinear model predictive control (MPC) to execute the behavior commands generated from the planner. This two-layer control framework is shown to be effective by simulations.

Minimum rational entropy fault tolerant control for non-Gaussian singular stochastic distribution control systems using T-S fuzzy modelling

ABSTRACTIn this paper, a new fault diagnosis (FD) and fault tolerant control (FTC) algorithm for a non-Gaussian nonlinear singular stochastic distribution control (SDC) system is studied. The rational square-root fuzzy logic model is used to approximate the output probability density function of non-Gaussian processes and a Takagi-Sugeno (T-S) fuzzy model is employed to transform the non-Gaussian nonlinear SDC system into a fuzzy SDC system. An adaptive fuzzy fault diagnosis observer is constructed to achieve reconstruction of system state and fault. Based on the estimated fault information, the controller is reconfigured by minimising the performance index with regard to the rational entropy subjected to mean constraint. Minimum rational entropy fault tolerant control is introduced to make the output of the past-fault SDC system still have the minimum uncertainty. Simulation results are provided to demonstrate the validity of the FD and minimum rational entropy FTC algorithm.

Aircraft Trajectory Optimization for Collision Avoidance Using Stochastic Optimal Control

AbstractOptimizing aircraft collision avoidance and performing trajectory optimization are the key problems in an air transportation system. This paper is focused on solving these problems by using a stochastic optimal control approach. The major contribution of this paper is a proposed stochastic optimal control algorithm to dynamically adjust and optimize aircraft trajectory. In addition, this algorithm accounts for random wind dynamics and convective weather areas with changing size. Although the system is modeled by a stochastic differential equation, the optimal feedback control for this equation can be computed as a solution of a partial differential equation, namely, an elliptic Hamilton‐Jacobi‐Bellman equation. In this paper, we solve this equation numerically using a Markov Chain approximation approach, where a comparison of three different iterative methods and two different optimization search methods are presented. Simulations show that the proposed method provides better performance in reducing conflict probability in the system and that it is feasible for real applications.

A survey on online learning and optimization for spark advance control of SI engines

One of the most important factors affecting fuel efficiency and emissions of automotive engines is combustion quality that is usually controlled by managing spark advance (SA) in spark ignition (SI) engines. With increasing sensing capabilities and enhancements in on-board computation capability, online learning and optimization techniques have been the subject of significant research interest. This article surveys the literature of learning and optimization algorithms with applications to combustion quality optimization and control of SI engines. In particular, this paper reviews extremum seeking control algorithms for iterative solution of online optimization problems, stochastic threshold control algorithms for iterative solution of probability control of stochastic knock event, as well as feedforward learning algorithms for iterative solution of operating-point-dependent feedforward adaptation problems. Finally, two experimental case studies including knock probabilistic constrained optimal combustion control and on-board map learning-based combustion control are carried out on an SI gasoline engine.

A Stochastic Control Approach to Maximize Profit on Service Provisioning for Mobile Cloudlet Platforms

The recent emergence of mobile cloud computing has enabled mobile users to offload computing tasks from mobile devices to nearby cloudlets, so as to reduce energy consumption and improve application performance. In this paper, we consider the problem of maximizing the profit of the cloudlets’ managing platform that receives computing requests from mobile users and fulfils these requests by leveraging computing service of participating cloudlets. However, it is very challenging to maximize the operating profit for such a managing platform, due to unpredictable arrival of user requests, dynamic participation of mobile cloudlets, and complexity in computing resource allocations. Based on the Lyapunov optimization technique combined with the technique of weight perturbation, we introduce a new stochastic control algorithm that makes online decisions on computing request admission and dispatching, computing service purchasing, and computing resource allocation. Different from traditional techniques, this algorithm does not require any statistical knowledge of relevant system dynamics, and is efficient for implementation in practice. Theoretical analysis and simulation results have demonstrated both the profit optimality and the system stability achieved by the proposed control algorithm.