Stochastic Feedback Control Research Articles

Robust stochastic stabilization of attitude motion

This study proposes robust stochastic stabilization of rigid body attitude motion within the framework of geometric mechanics, which can globally represent the attitude dynamics model. The system is subject to a stochastic input torque with an unknown variance parameter and an unknown nonlinear diffusion coefficient matrix. Our development starts with introducing a general notion of the stochastic stability in probability within the framework of geometric mechanics. Then, the Morse–Lyapunov (M–L) technique is employed to design a nonlinear continuous stochastic feedback control law. Finally, the asymptotic stability of the system is guaranteed in probability and the control gain parameters are obtained via solving a linear matrix inequality feasibility problem. An estimate of the region of attraction of the system is calculated to provide a better insight for tuning the control gain parameters. Two illustrative examples are performed based on the discretized model of the closed-loop system to demonstrate the effectiveness of the proposed control scheme.

Almost sure stabilization of hybrid systems by feedback control based on discrete-time observations of mode and state

Although the mean square stabilization of hybrid systems by feedback control based on discretetime observations of state and mode has been studied by several authors since 2013, the corresponding almost sure stabilization problem has received little attention. Recently, Mao was the first to study the almost sure stabilization of a given unstable system ẋ(t) = f(x(t)) by a linear discrete-time stochastic feedback control Ax([t/τ]τ)dB/(t) (namely the stochastically controlled system has the form dx(t) = f(x(t))dt + Ax([t/τ]τ)dB/(t), where B(t) is a scalar Brownian, τ > 0, and [t/τ] is the integer part of t/τ. In this paper, we consider a much more general problem. That is, we study the almost sure stabilization of a given unstable hybrid system ẋ(t) = f(x(t), r(t)) by nonlinear discrete-time stochastic feedback control u(x([t/τ]τ)dB(t) (so the stochastically controlled system is a hybrid stochastic system of the form dx(t) = f(x(t), r(t))dt + u(x([t/τ]τ))dB(t), where B(t) is a multi-dimensional Brownian motion and r(t) is a Markov chain.

Stochastic output-feedback model predictive control

A new formulation of Stochastic Model Predictive Output Feedback Control is presented and analyzed as a transposition of Stochastic Optimal Output Feedback Control into a receding horizon setting. This requires lifting the design into a framework involving propagation of the conditional state density, the information state, and solution of the Stochastic Dynamic Programming Equation for an optimal feedback policy, both stages of which are computationally challenging in the general, nonlinear setup. The upside is that the clearance of three bottleneck aspects of Model Predictive Control is connate to the optimality: output feedback is incorporated naturally; dual regulation and probing of the control signal is inherent; closed-loop performance relative to infinite-horizon optimal control is guaranteed. While the methods are numerically formidable, our aim is to develop an approach to Stochastic Model Predictive Control with guarantees and, from there, to seek a less onerous approximation. To this end, we discuss in particular the class of Partially Observable Markov Decision Processes, to which our results extend seamlessly, and demonstrate applicability with an example in healthcare decision making, where duality and associated optimality in the control signal are required for satisfactory closed-loop behavior.

Using probabilistic movement primitives in robotics

Movement Primitives are a well-established paradigm for modular movement representation and generation. They provide a data-driven representation of movements and support generalization to novel situations, temporal modulation, sequencing of primitives and controllers for executing the primitive on physical systems. However, while many MP frameworks exhibit some of these properties, there is a need for a unified framework that implements all of them in a principled way. In this paper, we show that this goal can be achieved by using a probabilistic representation. Our approach models trajectory distributions learned from stochastic movements. Probabilistic operations, such as conditioning can be used to achieve generalization to novel situations or to combine and blend movements in a principled way. We derive a stochastic feedback controller that reproduces the encoded variability of the movement and the coupling of the degrees of freedom of the robot. We evaluate and compare our approach on several simulated and real robot scenarios.

Target Uncertainty Mediates Sensorimotor Error Correction.

Human movements are prone to errors that arise from inaccuracies in both our perceptual processing and execution of motor commands. We can reduce such errors by both improving our estimates of the state of the world and through online error correction of the ongoing action. Two prominent frameworks that explain how humans solve these problems are Bayesian estimation and stochastic optimal feedback control. Here we examine the interaction between estimation and control by asking if uncertainty in estimates affects how subjects correct for errors that may arise during the movement. Unbeknownst to participants, we randomly shifted the visual feedback of their finger position as they reached to indicate the center of mass of an object. Even though participants were given ample time to compensate for this perturbation, they only fully corrected for the induced error on trials with low uncertainty about center of mass, with correction only partial in trials involving more uncertainty. The analysis of subjects’ scores revealed that participants corrected for errors just enough to avoid significant decrease in their overall scores, in agreement with the minimal intervention principle of optimal feedback control. We explain this behavior with a term in the loss function that accounts for the additional effort of adjusting one’s response. By suggesting that subjects’ decision uncertainty, as reflected in their posterior distribution, is a major factor in determining how their sensorimotor system responds to error, our findings support theoretical models in which the decision making and control processes are fully integrated.

Optimal bounded noisy feedback control for damping random vibrations

We consider a stochastic optimal feedback control problem for a single-degree-of-freedom vibrational system, where uncertainty is described by two independent noises. The first of them is induced by the control actions and called internal, whereas the second one acts externally. The drift vector also depends on the control function. The set of pointwise control constraints is assumed to be bounded. The minimization functional is taken as the mean system response energy. The Cauchy problem for the corresponding Hamilton–Jacobi–Bellman (HJB) equation without the control constraints is first investigated. This allows us to find the sought-for feedback control strategy in a specific domain of the space of state and time variables. Then a proper extension to the remaining parts of the space is constructed, and the optimality of the resulting global feedback control strategy is proved. The obtained control law is compared with the dry friction and saturated viscous friction control laws.

Stochastic modelling and feedback control of bistability in a turbulent bluff body wake

A specific feature of three-dimensional bluff body wakes, flow bistability, is a subject of particular recent interest. This feature consists of a random flipping of the wake between two asymmetric configurations and is believed to contribute to the pressure drag of many bluff bodies. In this study we apply the modelling approach recently suggested for axisymmetric bodies by Rigaset al.(J. Fluid Mech., vol. 778, 2015, R2) to the reflectional symmetry-breaking modes of a rectilinear bluff body wake. We demonstrate the validity of the model and its Reynolds number independence through time-resolved base pressure measurements of the natural wake. Further, oscillating flaps are used to investigate the dynamics and time scales of the instability associated with the flipping process, demonstrating that they are largely independent of Reynolds number. The modelling approach is then used to design a feedback controller that uses the flaps to suppress the symmetry-breaking modes. The controller is successful, leading to a suppression of the bistability of the wake, with concomitant reductions in both lateral and streamwise forces. Importantly, the controller is found to be efficient, the actuator requiring only 24 % of the aerodynamic power saving. The controller therefore provides a key demonstration of efficient feedback control used to reduce the drag of a high-Reynolds-number three-dimensional bluff body. Furthermore, the results suggest that suppression of large-scale structures is a fundamentally efficient approach for bluff body drag reduction.

Publisher's Note: Stochastic feedback control of quantum transport to realize a dynamical ensemble of two nonorthogonal pure states [Phys. Rev. B93, 085127 (2016)

Received 25 February 2016DOI:https://doi.org/10.1103/PhysRevB.93.119902©2016 American Physical Society

Stochastic feedback control of quantum transport to realize a dynamical ensemble of two nonorthogonal pure states

A Markovian open quantum system which relaxes to a unique steady state $\rho_{ss}$ of finite rank can be decomposed into a finite physically realizable ensemble (PRE) of pure states. That is, as shown by Karasik and Wiseman [Phys. Rev. Lett. 106, 020406 (2011)], in principle there is a way to monitor the environment so that in the long time limit the conditional state jumps between a finite number of possible pure states. In this paper we show how to apply this idea to the dynamics of a double quantum dot arising from the feedback control of quantum transport, as previously considered by one of us and co-workers [Phys. Rev. B 84, 085302 (2011)]. Specifically, we consider the limit where the system can be described as a qubit, and show that while the control scheme can always realize a two-state PRE, in the incoherent tunneling regime there are infinitely many PREs compatible with the dynamics that cannot be so realized. For the two-state PREs that are realized, we calculate the counting statistics and see a clear distinction between the coherent and incoherent regimes.

Almost Sure Exponential Stability of Stochastic Differential Delay Equations

This paper is concerned with the almost sure exponential stability of the multi-dimensional nonlinear stochastic differential delay equation (SDDE) with variable delays of the form $dx(t) = f(x(t-\delta_1(t)),t)dt + g(x(t-\delta_2(t)),t) dB(t)$, where $\delta_1, \ \delta_2: \mathbb{R}_+\to [0,\tau]$ stand for variable delays. We show that if the corresponding (nondelay) stochastic differential equation (SDE) $dy(t) = f(y(t),t)dt + g(y(t),t) dB(t)$ admits a Lyapunov function (which in particular implies the almost sure exponential stability of the SDE) then there exists a positive number $\tau^*$ such that the SDDE is also almost sure exponentially stable as long as the delay is bounded by $\tau^*$. We provide an implicit lower bound for $\tau^*$ which can be computed numerically. Moreover, our new theory enables us to design stochastic delay feedback controls in order to stabilize unstable differential equations.

Almost Sure Exponential Stabilization by Discrete-Time Stochastic Feedback Control

Given an unstable linear scalar differential equation $\dot{x}(t)=\alpha x(t)$ $(\alpha>0)$ , we will show that the discrete-time stochastic feedback control $\sigma x([t/\tau]\tau)dB(t)$ can stabilize it. That is, we will show that the stochastically controlled system $dx(t)=\alpha x(t)dt+\sigma x([t/\tau]\tau)dB(t)$ is almost surely exponentially stable when $\sigma^{2}>2\alpha$ and $\tau>0$ is sufficiently small, where $B(t)$ is a Brownian motion and $[t/\tau]$ is the integer part of $t/\tau$ . We will also discuss the nonlinear stabilization problem by a discrete-time stochastic feedback control. The reason why we consider the discrete-time stochastic feedback control is because that the state of the given system is in fact observed only at discrete times, say $0,\tau,2\tau,\ldots$ , for example, where $\tau>0$ is the duration between two consecutive observations. Accordingly, the stochastic feedback control should be designed based on these discrete-time observations, namely the stochastic feedback control should be of the form $\sigma x([t/\tau]\tau)dB(t)$ . From the point of control cost, it is cheaper if one only needs to observe the state less frequently. It is therefore useful to give a bound on $\tau$ from below as larger as better.

A generalizable adaptive brain-machine interface design for control of anesthesia.

Brain-machine interfaces (BMIs) for closed-loop control of anesthesia have the potential to automatically monitor and control brain states under anesthesia. Since a variety of anesthetic states are needed in different clinical scenarios, designing a generalizable BMI architecture that can control a wide range of anesthetic states is essential. In addition, drug dynamics are non-stationary over time and could change with the depth of anesthesia. Hence for precise control, a BMI needs to track these non-stationarities online. Here we design a BMI architecture that generalizes to control of various anesthetic states and their associated neural signatures, and is adaptive to time-varying drug dynamics. We provide a systematic approach to build general parametric models that quantify the anesthetic state and describe the drug dynamics. Based on these models, we develop an adaptive closed-loop controller within the framework of stochastic optimal feedback control. This controller tracks the non-stationarities in drug dynamics, achieves tight control in a time-varying environment, and removes the need for an offline system identification session. For robustness, the BMI also ensures small drug infusion rate variations at steady state. We test the BMI architecture for control of two common anesthetic states, i.e., burst suppression in medically-induced coma and unconsciousness in general anesthesia. Using numerical experiments, we find that the BMI generalizes to control of both these anesthetic states; in a time-varying environment, even without initial knowledge of model parameters, the BMI accurately controls these two different anesthetic states, reducing bias and error more than 70 times and 9 times, respectively, compared with a non-adaptive system.

Stochastic optimal open-loop feedback control: Approximate solution of the Hamiltonian system

Considering a dynamic control system with random model parameters and using the stochastic Hamilton approach stochastic open-loop feedback controls can be determined by solving a two-point boundary value problem (BVP) that describes the optimal state and costate trajectory. In general an analytical solution of the BVP cannot be found. This paper presents two approaches for approximate solutions, each consisting of two independent approximation stages. One stage consists of an iteration process with linearized BVPs that will terminate when the optimal trajectories are represented. These linearized BVPs are then solved by either approximation fixed-point equations (first approach) or Taylor-Expansions in the underlying stochastic model parameters (second approach). This approximation results in a deterministic linear BVP, which can be handled by solving a matrix Riccati differential equation.

Cell dynamics and gene expression control in tissue homeostasis and development.

During tissue and organ development and maintenance, the dynamic regulation of cellular proliferation and differentiation allows cells to build highly elaborate structures. The development of the vertebrate retina or the maintenance of adult intestinal crypts, for instance, involves the arrangement of newly created cells with different phenotypes, the proportions of which need to be tightly controlled. While some of the basic principles underlying these processes developing and maintaining these organs are known, much remains to be learnt from how cells encode the necessary information and use it to attain those complex but reproducible arrangements. Here, we review the current knowledge on the principles underlying cell population dynamics during tissue development and homeostasis. In particular, we discuss how stochastic fate assignment, cell division, feedback control and cellular transition states interact during organ and tissue development and maintenance in multicellular organisms. We propose a framework, involving the existence of a transition state in which cells are more susceptible to signals that can affect their gene expression state and influence their cell fate decisions. This framework, which also applies to systems much more amenable to quantitative analysis like differentiating embryonic stem cells, links gene expression programmes with cell population dynamics.

Stochastic stabilization of singular systems with Markovian switchings

This paper studies the stochastic stabilization problem for a class of singular Markovian jump system. The aim is to determine whether or not there is a stochastic feedback controller stabilizing a given singular Markovian jump system (SMJS). A new kind of stochastic controller only in the diffusion part is proposed such that the closed-loop system has a unique solution and is almost surely exponentially admissible. New sufficient condition for the existence of such a controller is given as linear matrix inequalities (LMIs). Based on this, more extensions to transition probability matrix (TPM) with elements partially unknown and system states partially observable are developed. A numerical example is used to demonstrate the effectiveness of the proposed methods.

Tips on Stochastic Optimal Feedback Control and Bayesian Spatiotemporal Models: Applications to Robotics

This tutorial paper presents the expositions of stochastic optimal feedback control theory and Bayesian spatiotemporal models in the context of robotics applications. The presented material is self-contained so that readers can grasp the most important concepts and acquire knowledge needed to jump-start their research. To facilitate this, we provide a series of educational examples from robotics and mobile sensor networks.

On Stochastic Feedback Control for Multi-Antenna Beamforming: Formulation and Low-Complexity Algorithms

Based on the Gauss-Markov channel model, we investigate the stochastic feedback control for transmit beamforming in multiple-input-single-output systems and design practical implementation algorithms leveraging techniques in dynamic programming and reinforcement learning. We first validate the Markov decision process formulation of the underlying feedback control problem with a 4R-variable (4R-V) state, where R is the number of the transmit antennas. Due to the high complexity of finding an optimal feedback policy under the 4R-V state, we consider a reduced 2-V state. As opposed to a previous study that assumes the feedback problem under such a 2-V state remaining an MDP formulation, our analysis indicates that the underlying problem is no longer an MDP. Nonetheless, the approximation as an MDP is shown to be justifiable and efficient. Based on the quantized 2-V state and the MDP approximation, we propose practical implementation algorithms for feedback control with unknown state transition probabilities. In particular, we provide model-based offline and online learning algorithms, as well as a model-free learning algorithm. We investigate and compare these algorithms through extensive simulations and provide their efficiency analysis. According to these results, the application rule of these algorithms is established under both statistically stable and unstable channels.

Kinematic feedback control laws for generating natural arm movements

We propose a stochastic optimal feedback control law for generating natural robot arm motions. Our approach, inspired by the minimum variance principle of Harris and Wolpert (1998 Nature 394 780–4) and the optimal feedback control principles put forth by Todorov and Jordan (2002 Nature Neurosci. 5 1226–35) for explaining human movements, differs in two crucial respects: (i) the endpoint variance is minimized in joint space rather than Cartesian hand space, and (ii) we ignore the dynamics and instead consider only the second-order differential kinematics. The feedback control law generating the motions can be straightforwardly obtained by backward integration of a set of ordinary differential equations; these equations are obtained exactly, without any linear–quadratic approximations. The only parameters to be determined a priori are the variance scale factors, and for both the two-DOF planar arm and the seven-DOF spatial arm, a table of values is constructed based on the given initial and final arm configurations; these values are determined via an optimal fitting procedure, and consistent with existing findings about neuromuscular motor noise levels of human arm muscles. Experiments conducted with a two-link planar arm and a seven-DOF spatial arm verify that the trajectories generated by our feedback control law closely resemble human arm motions, in the sense of producing nearly straight-line hand trajectories, having bell-shaped velocity profiles, and satisfying Fitts Law.

Modeling and Output Feedback Control of Networked Control Systems with Both Time Delays; and Packet Dropouts

This paper is concerned with the problem of modeling and output feedback controller design for a class of discrete-time networked control systems (NCSs) with time delays and packet dropouts. A Markovian jumping method is proposed to deal with random time delays and packet dropouts. Different from the previous studies on the issue, the characteristics of networked communication delays and packet dropouts can be truly reflected by the unified model; namely, both sensor-to-controller (S-C) and controller-to-actuator (C-A) time delays, and packet dropouts are modeled and their history behavior is described by multiple Markov chains. The resulting closed-loop system is described by a new Markovian jump linear system (MJLS) with Markov delays model. Based on Lyapunov stability theory and linear matrix inequality (LMI) method, sufficient conditions of the stochastic stability and output feedback controller design method for NCSs with random time delays and packet dropouts are presented. A numerical example is given to illustrate the effectiveness of the proposed method.

Stochastic stabilization of hybrid differential equations

This paper aims to determine whether or not a stochastic state feedback control can stabilize a given linear or nonlinear hybrid system. New methods are developed and sufficient conditions on the stability for hybrid stochastic differential equations are provided. These results are then used to examine stochastic stabilization by stochastic feedback controls. Two types of structure controls, namely state feedback and output injection, are discussed. Our stabilization criteria are in terms of linear matrix inequalities (LMIs) whence the feedback controls can be designed more easily in practice. A couple of examples and computer simulations are worked out to illustrate our theory.