Stochastic Optimal Feedback Control Research Articles

Uncertainty-resilient constrained rendezvous trajectory optimization via stochastic feedback control and unscented transformation

This paper proposes an innovative approach for constrained rendezvous trajectory optimization (CRTO) using stochastic optimal feedback control and unscented transform (UT) uncertainty quantification. The method overcomes limitations of traditional deterministic CRTO solutions that suffer from uncontrolled terminal state error growth and severely deteriorated path constraint satisfaction when initial state uncertainty, dynamics uncertainty, and navigation uncertainty are considered. The approach involves constructing a stochastic optimal feedback control (SOFC) problem with chance constraints and introducing linear feedback control to regulate both the mean and variance of terminal state errors. UT is employed to approximately quantify the state errors and their propagation, transforming the SOFC problem into an unconstrained deterministic optimization (UDO) problem. Differential dynamic programming (DDP) is then used to solve the UDO problem. The obtained stochastic optimal solution provides a robust rendezvous trajectory and a corresponding explicit closed-loop guidance law, improving terminal accuracy and path constraint satisfaction in the presence of various considered uncertainties. The influence of different term weights in the objective function on terminal accuracy and path constraint satisfaction is also studied. The effectiveness of the proposed approach is verified through Monte Carlo simulation, demonstrating the robustness of the closed-loop control strategy. The results highlight the potential of the method in enhancing terminal accuracy and path constraint satisfaction in uncertain rendezvous scenarios.

A bicycle can be balanced by stochastic optimal feedback control but only with accurate speed estimates.

Balancing a bicycle is typical for the balance control humans perform as a part of a whole range of behaviors (walking, running, skating, skiing, etc.). This paper presents a general model of balance control and applies it to the balancing of a bicycle. Balance control has both a physics (mechanics) and a neurobiological component. The physics component pertains to the laws that govern the movements of the rider and his bicycle, and the neurobiological component pertains to the mechanisms via which the central nervous system (CNS) uses these laws for balance control. This paper presents a computational model of this neurobiological component, based on the theory of stochastic optimal feedback control (OFC). The central concept in this model is a computational system, implemented in the CNS, that controls a mechanical system outside the CNS. This computational system uses an internal model to calculate optimal control actions as specified by the theory of stochastic OFC. For the computational model to be plausible, it must be robust to at least two inevitable inaccuracies: (1) model parameters that the CNS learns slowly from interactions with the CNS-attached body and bicycle (i.e., the internal noise covariance matrices), and (2) model parameters that depend on unreliable sensory input (i.e., movement speed). By means of simulations, I demonstrate that this model can balance a bicycle under realistic conditions and is robust to inaccuracies in the learned sensorimotor noise characteristics. However, the model is not robust to inaccuracies in the movement speed estimates. This has important implications for the plausibility of stochastic OFC as a model for motor control.

A bicycle can be balanced by stochastic optimal feedback control but only with accurate speed estimates

A bicycle can be balanced by stochastic optimal feedback control but only with accurate speed estimates

Approximative Policy Iteration for Exit Time Feedback Control Problems Driven by Stochastic Differential Equations using Tensor Train Format

We consider a stochastic optimal exit time feedback control problem. The Bellman equation is solved approximatively via the Policy Iteration algorithm on a polynomial ansatz space by a sequence of linear equations. As high degree multi-polynomials are needed, the corresponding equations suffer from the curse of dimensionality even in moderate dimensions. We employ tensor-train methods to account for this problem. The approximation process within the Policy Iteration is done via a Least-Squares ansatz and the integration is done via Monte-Carlo methods. Numerical evidences are given for the (multi dimensional) double well potential and a three-hole potential.

Neural Basis of Stochastic Optimal Feedback Control: Current Status and Future Directions

Neural Basis of Stochastic Optimal Feedback Control: Current Status and Future Directions

Stabilization of electrostatic MEMS resonators using a stochastic optimal control

Noise-induced motions are a significant source of uncertainty in the response of electrostatic MEMS where electrical and mechanical sources contribute to noise and can result in sudden and dramatic loss of stability. In this manuscript, we present an analysis for the stabilization of an electrostatically actuated MEMS resonator in the presence of noise processes using a stochastic optimal control scheme. A stochastic lumped-mass model that accounts for the uncertainty in mass, mechanical restoring force, bias voltage, and AC voltage amplitude of the resonator is presented. The Ito^ equations, describing the averaged modulations of the resonator's total energy and phase difference, are obtained via the stochastic averaging of energy envelope. The stochastic optimal control law aiming at minimization the pull-in probability of the resonator is determined via the stochastic dynamic programming equations associated with the Ito^ equations. We show that the resulting stochastic optimal feedback control, with a careful selection of its control gain, can be used to suppress the noise-activated pull-in instability of the disturbed resonator. The impact of the time delay inherent in the feedback control input on the control effectiveness is investigated. In addition, the control scheme also shows superior performance in counteracting the deterministic pull-in instability of the resonator operating in the dynamic pull-in frequency band. Good agreement between the theoretical results and numerical simulation is demonstrated.

Rapid Feedback Responses Parallel the Urgency of Voluntary Reaching Movements

Optimal feedback control is a prominent theory used to interpret human motor behaviour. The theory posits that skilled actions emerge from control policies that link voluntary motor control (feedforward) with flexible feedback corrections (feedback control). It is clear the nervous system can generate flexible motor corrections (reflexes) when performing actions with different goals. We know little, however, about shared features of voluntary actions and feedback control in human movement. Here we reveal a link between the timing demands of voluntary actions and flexible responses to mechanical perturbations. In two experiments, 40 human participants (21 females) made reaching movements with different timing demands. We disturbed the arm with mechanical perturbations at movement onset (Experiment 1) and at locations ranging from movement onset to completion (Experiment 2). We used the resulting muscle responses and limb displacements as a proxy for the control policies that support voluntary reaching movements. We observed an increase in the sensitivity of elbow and shoulder muscle responses and a reduction in limb motion when the task imposed greater urgency to respond to the same perturbations. The results reveal a relationship between voluntary actions and feedback control as the limb was displaced less when moving faster in perturbation trials. Muscle responses scaled with changes in the displacement of the limb in perturbation trials within each timing condition. Across both experiments, human behaviour was captured by simulations based on stochastic optimal feedback control. Taken together, the results highlight flexible control that links sensory processing with features of human reaching movements.

Stochastic output-feedback model predictive control

A new formulation of Stochastic Model Predictive Output Feedback Control is presented and analyzed as a transposition of Stochastic Optimal Output Feedback Control into a receding horizon setting. This requires lifting the design into a framework involving propagation of the conditional state density, the information state, and solution of the Stochastic Dynamic Programming Equation for an optimal feedback policy, both stages of which are computationally challenging in the general, nonlinear setup. The upside is that the clearance of three bottleneck aspects of Model Predictive Control is connate to the optimality: output feedback is incorporated naturally; dual regulation and probing of the control signal is inherent; closed-loop performance relative to infinite-horizon optimal control is guaranteed. While the methods are numerically formidable, our aim is to develop an approach to Stochastic Model Predictive Control with guarantees and, from there, to seek a less onerous approximation. To this end, we discuss in particular the class of Partially Observable Markov Decision Processes, to which our results extend seamlessly, and demonstrate applicability with an example in healthcare decision making, where duality and associated optimality in the control signal are required for satisfactory closed-loop behavior.

Target Uncertainty Mediates Sensorimotor Error Correction.

Human movements are prone to errors that arise from inaccuracies in both our perceptual processing and execution of motor commands. We can reduce such errors by both improving our estimates of the state of the world and through online error correction of the ongoing action. Two prominent frameworks that explain how humans solve these problems are Bayesian estimation and stochastic optimal feedback control. Here we examine the interaction between estimation and control by asking if uncertainty in estimates affects how subjects correct for errors that may arise during the movement. Unbeknownst to participants, we randomly shifted the visual feedback of their finger position as they reached to indicate the center of mass of an object. Even though participants were given ample time to compensate for this perturbation, they only fully corrected for the induced error on trials with low uncertainty about center of mass, with correction only partial in trials involving more uncertainty. The analysis of subjects’ scores revealed that participants corrected for errors just enough to avoid significant decrease in their overall scores, in agreement with the minimal intervention principle of optimal feedback control. We explain this behavior with a term in the loss function that accounts for the additional effort of adjusting one’s response. By suggesting that subjects’ decision uncertainty, as reflected in their posterior distribution, is a major factor in determining how their sensorimotor system responds to error, our findings support theoretical models in which the decision making and control processes are fully integrated.

Optimal bounded noisy feedback control for damping random vibrations

We consider a stochastic optimal feedback control problem for a single-degree-of-freedom vibrational system, where uncertainty is described by two independent noises. The first of them is induced by the control actions and called internal, whereas the second one acts externally. The drift vector also depends on the control function. The set of pointwise control constraints is assumed to be bounded. The minimization functional is taken as the mean system response energy. The Cauchy problem for the corresponding Hamilton–Jacobi–Bellman (HJB) equation without the control constraints is first investigated. This allows us to find the sought-for feedback control strategy in a specific domain of the space of state and time variables. Then a proper extension to the remaining parts of the space is constructed, and the optimality of the resulting global feedback control strategy is proved. The obtained control law is compared with the dry friction and saturated viscous friction control laws.

A generalizable adaptive brain-machine interface design for control of anesthesia.

Brain-machine interfaces (BMIs) for closed-loop control of anesthesia have the potential to automatically monitor and control brain states under anesthesia. Since a variety of anesthetic states are needed in different clinical scenarios, designing a generalizable BMI architecture that can control a wide range of anesthetic states is essential. In addition, drug dynamics are non-stationary over time and could change with the depth of anesthesia. Hence for precise control, a BMI needs to track these non-stationarities online. Here we design a BMI architecture that generalizes to control of various anesthetic states and their associated neural signatures, and is adaptive to time-varying drug dynamics. We provide a systematic approach to build general parametric models that quantify the anesthetic state and describe the drug dynamics. Based on these models, we develop an adaptive closed-loop controller within the framework of stochastic optimal feedback control. This controller tracks the non-stationarities in drug dynamics, achieves tight control in a time-varying environment, and removes the need for an offline system identification session. For robustness, the BMI also ensures small drug infusion rate variations at steady state. We test the BMI architecture for control of two common anesthetic states, i.e., burst suppression in medically-induced coma and unconsciousness in general anesthesia. Using numerical experiments, we find that the BMI generalizes to control of both these anesthetic states; in a time-varying environment, even without initial knowledge of model parameters, the BMI accurately controls these two different anesthetic states, reducing bias and error more than 70 times and 9 times, respectively, compared with a non-adaptive system.

Stochastic optimal open-loop feedback control: Approximate solution of the Hamiltonian system

Considering a dynamic control system with random model parameters and using the stochastic Hamilton approach stochastic open-loop feedback controls can be determined by solving a two-point boundary value problem (BVP) that describes the optimal state and costate trajectory. In general an analytical solution of the BVP cannot be found. This paper presents two approaches for approximate solutions, each consisting of two independent approximation stages. One stage consists of an iteration process with linearized BVPs that will terminate when the optimal trajectories are represented. These linearized BVPs are then solved by either approximation fixed-point equations (first approach) or Taylor-Expansions in the underlying stochastic model parameters (second approach). This approximation results in a deterministic linear BVP, which can be handled by solving a matrix Riccati differential equation.

Tips on Stochastic Optimal Feedback Control and Bayesian Spatiotemporal Models: Applications to Robotics

This tutorial paper presents the expositions of stochastic optimal feedback control theory and Bayesian spatiotemporal models in the context of robotics applications. The presented material is self-contained so that readers can grasp the most important concepts and acquire knowledge needed to jump-start their research. To facilitate this, we provide a series of educational examples from robotics and mobile sensor networks.

Kinematic feedback control laws for generating natural arm movements

We propose a stochastic optimal feedback control law for generating natural robot arm motions. Our approach, inspired by the minimum variance principle of Harris and Wolpert (1998 Nature 394 780–4) and the optimal feedback control principles put forth by Todorov and Jordan (2002 Nature Neurosci. 5 1226–35) for explaining human movements, differs in two crucial respects: (i) the endpoint variance is minimized in joint space rather than Cartesian hand space, and (ii) we ignore the dynamics and instead consider only the second-order differential kinematics. The feedback control law generating the motions can be straightforwardly obtained by backward integration of a set of ordinary differential equations; these equations are obtained exactly, without any linear–quadratic approximations. The only parameters to be determined a priori are the variance scale factors, and for both the two-DOF planar arm and the seven-DOF spatial arm, a table of values is constructed based on the given initial and final arm configurations; these values are determined via an optimal fitting procedure, and consistent with existing findings about neuromuscular motor noise levels of human arm muscles. Experiments conducted with a two-link planar arm and a seven-DOF spatial arm verify that the trajectories generated by our feedback control law closely resemble human arm motions, in the sense of producing nearly straight-line hand trajectories, having bell-shaped velocity profiles, and satisfying Fitts Law.

Modulation of motor control in saccadic behaviors by TMS over the posterior parietal cortex

The right posterior parietal cortex (rPPC) has been found to be critical in shaping visual selection and distractor-induced saccade curvature in the context of predictive as well as nonpredictive visual cues by means of transcranial magnetic stimulation (TMS) interference. However, the dynamic details of how distractor-induced saccade curvatures are affected by rPPC TMS have not yet been investigated. This study aimed to elucidate the key dynamic properties that cause saccades to curve away from distractors with different degrees of curvature in various TMS and target predictability conditions. Stochastic optimal feedback control theory was used to model the dynamics of the TMS saccade data. This allowed estimation of torques, which was used to identify the critical dynamic mechanisms producing saccade curvature. The critical mechanisms of distractor-induced saccade curvatures were found to be the motor commands and torques in the transverse direction. When an unpredictable saccade target occurred with rPPC TMS, there was an initial period of greater distractor-induced torque toward the side opposite the distractor in the transverse direction, immediately followed by a relatively long period of recovery torque that brought the deviated trace back toward the target. The results imply that the mechanisms of distractor-induced saccade curvature may be comprised of two mechanisms: the first causing the initial deviation and the second bringing the deviated trace back toward the target. The pattern of the initial torque in the transverse direction revealed the former mechanism. Conversely, the later mechanism could be well explained as a consequence of the control policy in this model. To summarize, rPPC TMS increased the initial torque away from the distractor as well as the recovery torque toward the target.

Optimal Structural Control under Stochastic Uncertainty: Robust Optimal Open-Loop Feedback Control

AbstractOptimal feedback controls of PD- or PID-type can be approximated very efficiently by optimal open-loop feedback controls based on optimal open-loop controls. Extending the standard construction, stochastic optimal open-loop feedback controls are constructed by taking into account still the random parameter variations in the control system. Hence, corresponding to the standard deterministic case, stochastic optimal open-loop feedback controls are obtained by computing first stochastic optimal open-loop controls on the remaining time intervals tb ≤ t ≤ tf with an arbitrary intermediate starting time point tb, t0 ≤ tb ≤ tf. Evaluating these controls at the corresponding intermediate starting time points tb only, a stochastic optimal open-loop feedback control law is obtained. Stochastic optimal open-loop controls are determined by using a stochastic Hamiltonian approach. Hence, after introducing the stochastic Hamiltonian of the control problem under stochastic uncertainty, the H-minimal controls are determined and the related two-point boundary value problem with random parameters is solved. In order to apply the solution technique in real-time for two-point boundary value problems developed in former papers, also to problems with a non constant system matrices, the occurring fundamental matrix is computed approximatively by appropriate approximations of the system matrix, as e.g. by i) constant system matrices, ii) piecewise constant system matrices, and iii) by discretization of the probability distribution in case of a random system matrix. Error estimates are given.

Stochastic minimax optimal time-delay state feedback control of uncertain quasi-integrable Hamiltonian systems

A stochastic minimax optimal time-delay state feedback control strategy for uncertain quasi-integrable Hamiltonian systems is proposed. First, a stochastic optimal state feedback control problem of uncertain quasi-integrable Hamiltonian system with time delay in feedback control subjected to Gaussian white noise is formulated. Then, the time-delayed state feedback control forces are approximated by the control forces without time delay and the original problem is converted into a stochastic optimal state feedback control problem of an uncertain system without time delay. After that, by following a procedure based on the stochastic averaging method and stochastic differential game, the worst-case disturbances and the optimal controls are obtained from solving a Hamiltonian–Jacobi–Isaacs (HJI) equation. As an example, the stochastic minimax optimal state feedback control of a Duffing oscillator with parametric disturbances and time-delayed feedback control is worked out in detail to illustrate the procedure and effectiveness of the proposed control strategy.

An Optimal Feedback Control Framework for Grasping Objects with Position Uncertainty

As we move, the relative location between our hands and objects changes in uncertain ways due to noisy motor commands and imprecise and ambiguous sensory information. The impressive capabilities humans display for interacting and manipulating objects with position uncertainty suggest that our brain maintains representations of location uncertainty and builds compensation for uncertainty into its motor control strategies. Our previous work demonstrated that specific control strategies are used to compensate for location uncertainty. However, it is an open question whether compensation for position uncertainty in grasping is consistent with the stochastic optimal feedback control, mainly due to the difficulty of modeling natural tasks within this framework. In this study, we develop a stochastic optimal feedback control model to evaluate the optimality of human grasping strategies. We investigate the properties of the model through a series of simulation experiments and show that it explains key aspects of previously observed compensation strategies. It also provides a basis for individual differences in terms of differential control costs-the controller compensates only to the extent that performance benefits in terms of making stable grasps outweigh the additional control costs of compensation. These results suggest that stochastic optimal feedback control can be used to understand uncertainty compensation in complex natural tasks like grasping.

Manifold reaching paradigm: how do we handle target redundancy?

How the central nervous system coordinates the many intrinsic degrees of freedom of the musculoskeletal system is a recurrent question in motor control. Numerous studies addressed it by considering redundant reaching tasks such as point-to-point arm movements, for which many joint trajectories and muscle activations are usually compatible with a single goal. There exists, however, a different, extrinsic kind of redundancy that is target redundancy. Many times, indeed, the final point to reach is neither specified nor unique. In this study, we aim to understand how the central nervous system tackles such an extrinsic redundancy by considering a reaching-to-a-manifold paradigm, more specifically an arm pointing to a long vertical bar. In this case, the endpoint is not defined a priori and, therefore, subjects are free to choose any point on the bar to successfully achieve the task. We investigated the strategies used by subjects to handle this presented choice. Our results indicate both intersubject and intertrial consistency with respect to the freedom provided by the task. However, the subjects' behavior is found to be more variable than during classical point-to-point reaches. Interestingly, the average arm trajectories to the bar and the structure of intertrial endpoint variations could be explained via stochastic optimal control with an energy/smoothness expected cost and signal-dependent motor noise. We conclude that target redundancy is first overcome during movement planning and then exploited during movement execution, in agreement with stochastic optimal feedback control principles, which illustrates how the complementary problems of goal and movement selection may be resolved at once.

Optimal feedback control as a theory of motor coordination.

A central problem in motor control is understanding how the many biomechanical degrees of freedom are coordinated to achieve a common goal. An especially puzzling aspect of coordination is that behavioral goals are achieved reliably and repeatedly with movements rarely reproducible in their detail. Existing theoretical frameworks emphasize either goal achievement or the richness of motor variability, but fail to reconcile the two. Here we propose an alternative theory based on stochastic optimal feedback control. We show that the optimal strategy in the face of uncertainty is to allow variability in redundant (task-irrelevant) dimensions. This strategy does not enforce a desired trajectory, but uses feedback more intelligently, correcting only those deviations that interfere with task goals. From this framework, task-constrained variability, goal-directed corrections, motor synergies, controlled parameters, simplifying rules and discrete coordination modes emerge naturally. We present experimental results from a range of motor tasks to support this theory.