Optimal Feedback Control Research Articles

Optimal navigation for AGVs: A soft actor–critic-based reinforcement learning approach with composite auxiliary rewards

In this paper, we address the problem of real-time navigation and obstacle avoidance for automated guided vehicles (AGVs) in dynamic environments, which is a primary research area in collaborative control systems for AGVs. To overcome the computational inefficiency of recalculating optimal paths every time, we propose an improved Soft Actor–Critic (SAC)-based reinforcement learning methodology. This methodology utilizes a novel composite auxiliary reward structure and sum-tree prioritized experience replay (SAC-SP) to achieve real-time optimal feedback control. First, we formulate the navigation task as a Markov Decision Process that considers both static and dynamic obstacles. To accelerate the active learning of AGVs, we propose a novel strategy that uses composite auxiliary rewards. Next, we train the AGVs using the proposed SAC-SP methodology to handle real-time navigation with the composite auxiliary reward structure. The well-trained policy network can generate effective on-board optimal feedback actions given obstacle positions, targets, and AGV states. Simulation experiments demonstrate that our proposed method can steer AGVs to the destination with high robustness to original conditions and various obstacle restrictions, generating optimal feedback actions in the shortest amount of time.

Simulating Interaction Movements via Model Predictive Control

We present a Model Predictive Control (MPC) framework to simulate movement in interaction with computers, focusing on mid-air pointing as an example. Starting from understanding interaction from an Optimal Feedback Control (OFC) perspective, we assume that users aim at minimizing an internalized cost function, subject to the constraints imposed by the human body and the interactive system. Unlike previous approaches used in HCI, MPC can compute optimal controls for nonlinear systems. This allows to use state-of-the-art biomechanical models and handle nonlinearities that occur in almost any interactive system. Instead of torque actuation, our model employs second-order muscles acting directly at the joints. We compare three different cost functions and evaluate the simulation against user movements in a pointing study. Our results show that the combination of distance, control, and joint acceleration cost matches individual users’ movements best, and predicts movements with an accuracy that is within the between-user variance. To aid HCI researchers and designers in applying our approach for different users, interaction techniques, or tasks, we make our SimMPC framework, including CFAT, a tool to identify maximum voluntary torques in joint-actuated models, publicly available, and give step-by-step instructions.

Optimal Robust Tracking Control of Injection Velocity in an Injection Molding Machine

Injection molding is a critical component of modern industrial operations, and achieving fast and stable control of injection molding machines (IMMs) is essential for producing high-quality plastic products. This paper focuses on solving an optimal tracking control problem of the injection velocity that arises in a typical nonlinear IMM. To this end, an efficient optimal robust controller is proposed and designed. The nonlinear injection velocity servo system is first approximately linearized at iteration points using the first-order Taylor expansion approach. Then, at each time node in the optimization process, the relevant algebraic Riccati equation is introduced, and the solution is used to construct an optimal robust feedback controller. Furthermore, a rigorous Lyapunov theorem analysis is employed to demonstrate the global stability properties of the proposed feedback controller. The results from numerical simulations show that the proposed optimal robust control strategy can successfully and rapidly achieve the best tracking of the intended injection velocity trajectory within a given time.

Consensus as a Nash Equilibrium of a Stochastic Differential Game

In this paper a consensus has been constructed in a social network which is modeled by a stochastic differential game played by agents of that network. Each agent independently minimizes a cost function which represents their motives. A conditionally expected integral cost function has been considered under an agent’s opinion filtration. The dynamic cost functional is minimized subject to a stochastic differential opinion dynamics. As opinion dynamics represents an agent’s differences of opinion from the others as well as from their previous opinions, random influences and stubbornness make it more volatile. An agent uses their rate of change of opinion at certain time point as a control input. This turns out to be a non-cooperative stochastic differential game which have a feedback Nash equilibrium. A Feynman-type path integral approach has been used to determine an optimal feedback opinion and control. This is a new approach in this literature. Later in this paper an explicit solution of a feedback Nash equilibrium opinion is determined.

Optimal prescribed performance tracking control of nonlinear motor driven systems via adaptive dynamic programming

AbstractAlthough optimal regulation problem has been well studied, resolving optimal tracking control via adaptive dynamic programming (ADP) has not been completely resolved, particularly for nonlinear uncertain systems. In this paper, an online adaptive learning method is developed to realize the optimal tracking control design for nonlinear motor driven systems (NMDSs), which adopts the concept of ADP, unknown system dynamic estimator (USDE), and prescribed performance function (PPF). To this end, the USDE in a simple form is first proposed to address the NMDSs with bounded disturbances. Then, based on the estimated unknown dynamics, we define an optimal cost function and derive the optimal tracking control. The derived optimal tracking control is divided into two parts, that is, steady‐state control and optimal feedback control. The steady‐state control can be obtained with the tracking commands directly. The optimal feedback control can be obtained via the concept of ADP based on the PPF; this contributes to improving the convergence of critic neural network (CNN) weights and tracking accuracy of NMDSs. Simulations are provided to display the feasibility of the designed control method.

Singular Optimal Linear Quadratic Regulator of Switched Systems

In this paper, we will investigate the singular linear quadratic regulator (LQR) problem of switched linear system in finite time horizon. The proposed method is transforming into a switched LQR problem by adopting linear transformation. Next, we adopt an embedding transformation method to convert the switched LQR problem to a traditional optimal control problem, so the bang-bang-type solution of the embedded optimal control problem in finite time is the optimal solution to the switched LQR problem. The switching sequence of modes and the switching instants can be calculated by solving a closed-form optimal switching condition. The optimal state feedback control law is determined simultaneously. Then, by solving a sequence of Riccati equation, we find some conditions that ensure switched LQR problem can be convert to the singular LQR problem. Finally, a numerical example is presented to demonstrate the effectiveness of the proposed method.

Time Optimal Feedback Control for 3D Navier–Stokes-Voigt Equations

In this article, we discuss a time optimal feedback control for asymmetrical 3D Navier–Stokes–Voigt equations. Firstly, we consider the existence of the admissible trajectories for the asymmetrical 3D Navier–Stokes–Voigt equations by using the well-known Cesari property and the Fillippove’s theorem. Secondly, we study the existence result of a time optimal control for the feedback control systems. Lastly, asymmetrical Clarke’s subdifferential inclusions and asymmetrical 3D Navier–Stokes–Voigt differential variational inequalities are given to explain our main results.

Optimal tuned outrigger-ATMD systems considering seismic and wind loads

This paper formulates the optimal feedback control model for the outrigger-active tuned mass damper (ATMD) system and investigates the influence of some input variables on the control effect while considering seismic and wind loads. The paper first deduces the joint equation of motion of the outrigger-ATMD system. Different control theories are then used to formulate the optimal control force, the optimal frequency tuning, and the optimal damping tuning. Based on such optimal tuned outrigger-ATMD systems, sensitivity analyses for variables from the ATMD and outrigger system are implemented. With many carefully selected schemes, the control effect and efficiency of the control model of the ATMD are verified in the frequency domain and time domain. It is observed that the fixed-points theory is a better choice for the ATMD tuning and the supplement damping in the outrigger system will suppress the control performance of the ATMD.

Nonlinear Optimal Control for Six-Phase Induction Generator-Based Renewable Energy Systems

The article aims at optimizing six-phase induction generator-based renewable energy systems (6-phase IGs or dual star induction generators) through a novel nonlinear optimal control method. Six-phase induction generators appear to be advantageous compared to three-phase synchronous or asynchronous power generators, in terms of fault tolerance and improved power generation rates. The dynamic model of the six-phase induction generator is first written in a nonlinear and multivariable state-space form. It is proven that this model is differentially flat. The 6-phase IG is approximately linearized around a temporary operating point recomputed at each sampling interval to design the optimal controller. The linearization is based on first-order Taylor series expansion and the Jacobian matrices of the state-space model of the 6-phase IG. A stabilizing optimal (H-infinity) feedback controller is designed for the linearized state-space description of the six-phase IG. The feedback gains of the controller are computed by solving an algebraic Riccati equation at each iteration of the control method. Lyapunov analysis is used to demonstrate global stability for the control loop. The H-infinity Kalman Filter is also used as a robust state estimator, which allows for implementing sensorless control for 6-phase IG-based renewable energy systems. The nonlinear optimal control method achieves fast and accurate tracking of setpoints by the state variables of the 6-phase IG, under moderate variations of the control inputs.

Finite-Time Disturbance Decoupling of Boolean Control Networks

In this article, the finite-time disturbance decoupling problem (FTDDP) of Boolean networks (BNs) and Boolean control networks (BCNs) is studied. A novel necessary and sufficient condition is first provided to determine whether a given BN is finite-time disturbance decoupled. Then, an algorithm is devised to find the shortest time that guarantees that a given BN is finite-time disturbance decoupled. In addition, using a constructed truth matrix, a necessary and sufficient condition for solving the disturbance decoupling problem of BCNs is proposed, and all possible time optimal state feedback controllers are designed. Finally, two examples are introduced to illustrate the feasibility of the obtained results.

Nonlinear optimal control for a 4-DOF SCARA robotic manipulator

AbstractSelective compliance articulated robot arms (SCARA) robotic manipulators find wide use in industry. A nonlinear optimal control approach is proposed for the dynamic model of the 4-degrees of freedom (DOF) SCARA robotic manipulator. The dynamic model of the SCARA robot undergoes approximate linearization around a temporary operating point that is recomputed at each time-step of the control method. The linearization relies on Taylor series expansion and on the associated Jacobian matrices. For the linearized state-space model of the system, a stabilizing optimal (H-infinity) feedback controller is designed. To compute the controller’s feedback gains, an algebraic Riccati equation is repetitively solved at each iteration of the control algorithm. The stability properties of the control method are proven through Lyapunov analysis. The proposed control method is advantageous because: (i) unlike the popular computed torque method for robotic manipulators, it is characterized by optimality and is also applicable when the number of control inputs is not equal to the robot’s number of DOFs and (ii) it achieves fast and accurate tracking of reference setpoints under minimal energy consumption by the robot’s actuators. The nonlinear optimal controller for the 4-DOF SCARA robot is finally compared against a flatness-based controller implemented in successive loops.

Linear feedback control of spreading dynamics in stochastic nonlinear network models: epileptic seizures.

The development of models and approaches for controlling the spreading dynamics of epileptic seizures is an essential step towards new therapies for people with pharmacologically resistant epilepsy. Beyond resective neurosurgery, in which epileptogenic zones (EZs) are the target of surgery, closed-loop control based on intracranial electrical stimulation, applied at the very early stage of seizure evolution, has been the main alternative, e.g. the RNS system from NeuroPace (Mountain View, CA). In this approach the electrical stimulation is delivered to target brain areas after detection of seizure initiation in the EZ. Here, we examined, on model simulations, some of the closed-loop control aspects of the problem. Seizure dynamics and spread are typically modeled with highly nonlinear dynamics on complex brain networks. Despite the nonlinearity and complexity, currently available optimal feedback control approaches are mostly based on linear approximations. Alternative machine learning control approaches might require amounts of data beyond what is commonly available in the intended application. We thus examined how standard linear feedback control approaches perform when applied to nonlinear models of neural dynamics of seizure generation and spread. In particular, we considered patient-specific epileptor network models for seizure onset and spread. The models incorporate network connectivity derived from (diffusion MRI) white-matter tractography, have been shown to capture the qualitative dynamics of epileptic seizures and can be fit to patient data. For control, we considered simple linear quadratic Gaussian (LQG) regulators. The LQG control was based on a discrete-time state-space model derived from the linearization of the patient-specific epileptor network model around a stable fixed point in the regime of preictal dynamics. We show in simulations that LQG regulators acting on the EZ node during the initial seizure period tend to be unstable. The LQG solution for the control law in this case leads to global feedback to the EZ-node actuator. However, if the LQG solution is constrained to depend on only local feedback originating from the EZ node itself, the controller is stable. In this case, we demonstrate that localized LQG can easily terminate the seizure at the early stage and prevent spread. In the context of optimal feedback control based on linear approximations, our results point to the need for investigating in more detail feedback localization and additional relevant control targets beyond epileptogenic zones.

Body Mechanics, Optimality, and Sensory Feedback in the Human Control of Complex Objects.

Humans are adept at a wide variety of motor skills, including the handling of complex objects and using tools. Advances to understand the control of voluntary goal-directed movements have focused on simple behaviors such as reaching, uncoupled to any additional object dynamics. Under these simplified conditions, basic elements of motor control, such as the roles of body mechanics, objective functions, and sensory feedback, have been characterized. However, these elements have mostly been examined in isolation, and the interactions between these elements have received less attention. This study examined a task with internal dynamics, inspired by the daily skill of transporting a cup of coffee, with additional expected or unexpected perturbations to probe the structure of the controller. Using optimal feedback control (OFC) as the basis, it proved necessary to endow the model of the body with mechanical impedance to generate the kinematic features observed in the human experimental data. The addition of mechanical impedance revealed that simulated movements were no longer sensitively dependent on the objective function, a highly debated cornerstone of optimal control. Further, feedforward replay of the control inputs was similarly successful in coping with perturbations as when feedback, or sensory information, was included. These findings suggest that when the control model incorporates a representation of the mechanical properties of the limb, that is, embodies its dynamics, the specific objective function and sensory feedback become less critical, and complex interactions with dynamic objects can be successfully managed.

Event‐triggered‐based integral reinforcement learning output feedback optimal control for partially unknown constrained‐input nonlinear systems

AbstractIn this paper, an adaptive output feedback event‐triggered optimal control algorithm is proposed for partially unknown constrained‐input continuous‐time nonlinear systems. First, a neural network observer is constructed to estimate unmeasurable state. Next, an event‐triggered condition is established, and only when the event‐triggered condition is violated will the event be triggered and the state be sampled. Then, an event‐triggered‐based synchronous integral reinforcement learning (ET‐SIRL) control algorithm with critic‐actor neural networks (NNs) architecture is proposed to solve the event‐triggered Hamilton–Jacobi–Bellman equation under the established event‐triggered condition. The critic and actor NNs are used to approximate cost function and optimal event‐triggered optimal control law, respectively. Meanwhile, the event‐triggered‐based closed‐loop system state and all the neural network weight estimation errors are uniformly ultimately bounded proved by Lyapunov stability theory, and there is no Zeno behavior. Finally, two numerical examples are presented to show the effectiveness of the proposed ET‐SIRL control algorithm.

Nonlinear optimal control for a gas compressor driven by an induction motor

The article proposes a nonlinear optimal control approach for the dynamic model of a gas centrifugal compressor being driven by an induction motor (IM). The dynamic model of the integrated compressor-induction motor system, being initially expressed in a nonlinear and multivariable state-space form, undergoes approximate linearization around a temporary operating point that is recomputed at each time-step of the control method. The linearization relies on first-order Taylor series expansion and on the computation of the associated Jacobian matrices. For the linearized state-space model of the compressor-IM a stabilizing optimal (H-infinity) feedback controller is designed. This controller stands for the solution to the nonlinear optimal control problem of the compressor-IM under model uncertainty and external perturbations. To compute the controller’s feedback gains an algebraic Riccati equation is repetitively solved at each iteration of the control algorithm. The global stability properties of the control method are proven through Lyapunov analysis. Finally, to implement state estimation-based control of the compressor-IM, without the need to measure its entire state vector, the H-infinity Kalman Filter is used as a robust state estimator. The differential flatness properties of the compressor-IM system are proven. The article’s method provides one of the few algorithmically simple and computationally efficient solutions for the nonlinear optimal control problem of the compressor-IM system.

Nonlinear neural control using feedback linearization explains task goals of central nervous system for trunk stability in sitting posture

Objective. Characterizing the task goals of the neural control system for achieving seated stability has been a fundamental challenge in human motor control research. This study aimed to experimentally identify the task goals of the neural control system for seated stability. Approach. Ten able-bodied young individuals participated in our experiments, which allowed us to measure their body motion and muscle activity during perturbed sitting. We used a nonlinear neuromechanical model of the seated human, along with a full-state feedback linearization approach and optimal control theory for identifying the neural control system and characterizing its task goals. Main results. We demonstrated that the neural feedback for trunk stability during seated posture uses angular position, velocity, acceleration, and jerk in a linearized space. The mean squared error between the predicted and measured motor commands was less than 0.6% among all trials and participants, with a median correlation coefficient of more than 0.9. Our identified optimal neural control primarily used trunk angular acceleration and near-minimum muscle activation to achieve seated stability while keeping the deviations of the trunk angular position and acceleration sufficiently small. Significance. Our proposed approach to neural control system identification relied on a performance criterion (e.g. cost function) explaining what the functional goal is and subsequently, finds the control law that leads to the best performance. Therefore, instead of assuming what control schemes the neural control might utilize (e.g. proportional-integral-derivative control), optimal control allows the motor task and the neuromechanical model to dictate a control law that best describes the physiological process. This approach allows for a mechanistic understanding of the neuromuscular mechanisms involved in seated stability and for inferring the task goals used by the neural control system to achieve the targeted motor behavior. Such neural control characterization can contribute to the development of objective balance evaluation tools and of bio-inspired assistive neuromodulation technologies.

Convergence of Regularized Particle Filters for Stochastic Reaction Networks

Filtering for stochastic reaction networks (SRNs) is an important problem in systems/synthetic biology aiming to estimate the state of unobserved chemical species. A good solution to it can provide scientists valuable information about the hidden dynamic state and enable optimal feedback control. Usually, the model parameters need to be inferred simultaneously with state variables, and a conventional particle filter can fail to solve this problem accurately due to sample degeneracy. In this case, the regularized particle filter (RPF) is preferred to the conventional ones, as the RPF can mitigate sample degeneracy by perturbing particles with artificial noise. However, the artificial noise introduces an additional bias to the estimate, and, thus, it is questionable whether the RPF can provide reliable results for SRNs. In this paper, we aim to identify conditions under which the RPF converges to the exact filter in the filtering problem determined by a bimolecular network. First, we establish computationally efficient RPFs for SRNs on different scales using different dynamical models, including the continuous-time Markov process, tau-leaping model, and piecewise deterministic process. Then, by parameter sensitivity analyses, we show that the established RPFs converge to the exact filters if all reactions leading to an increase of the molecular population have linearly growing propensities and some other mild conditions are satisfied simultaneously. This ensures the performance of the RPF for a large class of SRNs, and several numerical examples are presented to illustrate our results.

Multibody dynamic modeling and analysis of cable-driven snake robot considering clearance and friction based on ALE method

Cable-driven snake robot (CDSR) is a kind of slender manipulator with excellent maneuverability, which is widely applied in complex spaces. However, several nonlinear factors, such as clearance and contact between cable and hole, can lead to uncertainty of motion and danger of cable under extreme tension, which can affect the performance of the CDSR. In this paper, a multibody dynamic model of CDSR considering both friction and clearance is proposed. The arbitrary Lagrangian Eulerian (ALE) method is used to model the driving cable. The ALE node is constrained in the hole plane and the contact forces are evaluated using nonlinear impact function to describe the cable-hole interaction. Moreover, contact force regularization is adopted to facilitate convergence. A numerical test demonstrates that the clearance model can describe the contact behavior of cable and hole accurately. Moreover, the effect of clearance sizes on CDSR are studied. It shows that the manipulation precision can be dramatically reduced due to the increase of clearance size. Finally, the driving cable slack phenomenon induced by the coupling effect of clearance and friction is revealed. The presented work is expected to be applied for the compensation of feedback control and tension optimization design of CDSR.

Finite-Time Stabilization and Optimal Feedback Control for Nonlinear Discrete-Time Systems

Finite-time stability involves dynamical systems whose trajectories converge to an equilibrium state in finite time. Sufficient conditions for finite-time stability have recently been developed in the literature for discrete-time dynamical systems. In this article, we build on these results to develop a framework for addressing the problem of optimal nonlinear analysis and feedback control for finite-time stability and finite-time stabilization for nonlinear discrete-time controlled dynamical systems. Finite-time stability of the closed-loop nonlinear system is guaranteed by means of a Lyapunov function that satisfies a difference inequality involving fractional powers and a minimum operator. This Lyapunov function can clearly be seen to be the solution to a difference equation that corresponds to a steady-state form of the Bellman equation, and hence, guaranteeing both finite-time stability and optimality. Finally, a numerical example is presented to demonstrate the efficacy of the proposed finite-time discrete stabilization framework.

Fully Distributed Optimal Consensus for a Class of Second-Order Nonlinear Multiagent Systems With Switching Topologies

This article proposes a novel optimal consensus protocol for a class of leaderless multiagent systems, where each agent is described by a second-order nonlinear system and agents interact with each other on switching networks. Any global information, including the eigenvalues of the Laplacian matrix, is unavailable in the control scheme development. The reference trajectory is designed for each agent, and the corresponding performance function, which reflects the off-track error evolution and control cost, is proposed. The sufficient condition for the synchronization of reference trajectories, which does not rely on the topology dwell time, is established by constructing an appropriate current topology-independent Lyapunov function. Due to the fact that the nonlinear function in the system dynamical equation of each agent is unknown, an equation termed integral reinforcement learning (IRL) equation is provided, and it is strictly proven that the provided IRL equation is equivalent to the given Hamilton–Jacobi–Bellman equation. The model-free optimal feedback control law is then derived based on the IRL technique. In the implementation of the developed control scheme, the neural network approximation tool is adopted, and the scheme is applied to a numerical system to show its effectiveness.