Adaptive Optimal Control Algorithm Research Articles

Adaptive optimal fault-tolerant vibration control and sensitivity analysis of semi-active suspension based on a self-powered magneto-rheological damper

To reuse the energy dissipated by vehicle suspension, a semi-active suspension with a self-powered magneto-rheological damper is proposed. An electromechanical coupling model of self-powered semi-active suspension is established. The energy conversion efficiency is defined and investigated by changing the electrical parameters. By considering unmodeled dynamics and perturbation values, an adaptive optimal fault-tolerant control algorithm is proposed to ensure the vibration-isolation performance. The robust index of the adaptive optimal fault-tolerant control algorithm is constructed using the Lyapunov equation and evaluated by changing the key parameters. The sensitivity of the key parameters to the damping force is investigated using a grey relation analysis approach. Furthermore, multi-objective optimization between the vibration-isolation capability and energy harvesting is conducted. Via analysis, the proposed suspension can harvest more energy near the second resonance range. Compared to passive control and self-powered mode, the adaptive optimal control algorithm mitigates vibration more significantly in the time and frequency domains, respectively, under stochastic excitation. The robust index is most sensitive to inductance and the diameter of the magnetism cylinder. The length of the damping channel and the diameter of the magnetism cylinder influence the sensitivity of key parameters to the damping force most obviously.

Adaptive Optimal Tracking Control of an Underactuated Surface Vessel Using Actor-Critic Reinforcement Learning.

In this article, we present an adaptive reinforcement learning optimal tracking control (RLOTC) algorithm for an underactuated surface vessel subject to modeling uncertainties and time-varying external disturbances. By integrating backstepping technique with the optimized control design, we show that the desired optimal tracking performance of vessel control is guaranteed due to the fact that the virtual and actual control inputs are designed as optimized solutions of every subsystem. To enhance the robustness of vessel control systems, we employ neural network (NN) approximators to approximate uncertain vessel dynamics and present adaptive control technique to estimate the upper boundedness of external disturbances. Under the reinforcement learning framework, we construct actor-critic networks to solve the Hamilton-Jacobi-Bellman equations corresponding to subsystems of surface vessel to achieve the optimized control. The optimized control algorithm can synchronously train the adaptive parameters not only for actor-critic networks but also for NN approximators and adaptive control. By Lyapunov stability theorem, we show that the RLOTC algorithm can ensure the semiglobal uniform ultimate boundedness of the closed-loop systems. Compared with the existing reinforcement learning control results, the presented RLOTC algorithm can compensate for uncertain vessel dynamics and unknown disturbances, and obtain the optimized control performance by considering optimization in every backstepping design. Simulation studies on an underactuated surface vessel are given to illustrate the effectiveness of the RLOTC algorithm.

Regret bounds for online-learning-based linear quadratic control under database attacks

This paper is concerned with understanding and countering the effects of database attacks on a learning-based linear quadratic adaptive controller. This attack targets neither sensors nor actuators, but just poisons the learning algorithm and parameter estimator that is part of the regulation scheme. We focus on the adaptive optimal control algorithm introduced by Abbasi-Yadkori and Szepesvari and provide regret analysis in the presence of attacks as well as modifications that mitigate their effects. A core step of this algorithm is the self-regularized on-line least squares estimation, which determines a tight confidence set around the true parameters of the system with high probability. In the absence of malicious data injection, this set provides an appropriate estimate of parameters for the aim of control design. However, in the presence of attack, this confidence set is not reliable anymore. Hence, we first tackle the question of how to adjust the confidence set so that it can compensate for the effect of the poisonous data. Then, we quantify the deleterious effect of this type of attack on the optimality of control policy by bounding regret of the closed-loop system under attack.

Adaptive NN Optimal Consensus Fault-Tolerant Control for Stochastic Nonlinear Multiagent Systems.

This article investigates the problem of adaptive neural network (NN) optimal consensus tracking control for nonlinear multiagent systems (MASs) with stochastic disturbances and actuator bias faults. In control design, NN is adopted to approximate the unknown nonlinear dynamic, and a state identifier is constructed. The fault estimator is designed to solve the problem raised by time-varying actuator bias fault. By utilizing adaptive dynamic programming (ADP) in identifier-critic-actor construction, an adaptive NN optimal consensus fault-tolerant control algorithm is presented. It is proven that all signals of the controlled system are uniformly ultimately bounded (UUB) in probability, and all states of the follower agents can remain consensus with the leader's state. Finally, simulation results are given to illustrate the effectiveness of the developed optimal consensus control scheme and theorem.

The effect of an adaptive feedback-control system in the structural vibration control

In this work, a novel frequency adaptive optimal control algorithm has been developed that aims to provide the desired response of the structural system with an ideal control effort, appropriate to every excitation frequency range. To achieve this objective, one part of the control gain has been developed to provide frequency response characteristics similar to the structural system. The modified system under the proposed control is verified for stability and the optimal gain constants are obtained by minimizing the H2 norm of the structural output. A thorough investigation of the system shows that the algorithm, at a certain excitation frequency, influences the appropriate dynamic properties of the system, sensitive in that region. This behavior is found to persist in case of a perturbation in the structural system. Next, a theoretical assessment of the proposed control is conducted considering an existing and popular linear–quadratic Gaussian (LQG) control algorithm. The comparative assessment is extended through a numerical illustration considering a simple structural system exposed to earthquake excitations of a varying range of frequencies. A huge reduction in the control force is observed for the proposed control as compared to the LQG control for the same response characteristics of the structure and the efficiency is retained in the case of a perturbation in the initial structural system.

Online adaptive optimal control algorithm based on synchronous integral reinforcement learning with explorations

In this study, we present a novel algorithm, based on synchronous policy iteration, to solve the continuous-time infinite-horizon optimal control problem of input affine system dynamics. The integral reinforcement is measured as an excitation signal to estimate the solution to the Hamilton–Jacobi–Bellman equation. In addition, the proposed method is completely model-free, that is, no a priori knowledge of the system is required. Using the adaptive tuning law, the actor and critic neural networks can simultaneously approximate the optimal value function and policy. The persistence of excitation condition is required to guarantee the convergence of the two networks. Unlike in traditional policy iteration algorithms, the restriction of the initial admissible policy was eliminated using this method. The effectiveness of the proposed algorithm is verified through numerical simulations.

Development of an adaptive artificial neural network model and optimal control algorithm for a data center cyber–physical system

This study aimed to develop an adaptive artificial neural network model (AAM) for the prediction of the rack inlet temperature and cooling system energy, and the optimal control algorithm for cooling system of a containment-type data center. A cyber-physical system (CPS) framework, that incorporated the AAM and control algorithm, was also proposed for the precise control of the data center cooling system. To develop the AAM model and control algorithm, mathematical modeling of a reference physical model was conducted, and training data were acquired from this model. The performance of the proposed AAM and control algorithm was then compared with that of a non-adaptive ANN model (NAAM) in terms of prediction accuracy and control stability. The analysis results indicated that the optimal control algorithm with the AAM exhibited superior prediction accuracy and control stability than the algorithm with the NAAM. In particular, for the AAM-based algorithm under conditions representing a novel data center environment, the root mean square error (RMSE) and coefficient of variation of the RMSE (CV(RMSE)) for the predicted and actual values were 0.22 °C and 1.02%, respectively, for the inlet rack temperature and 0.19 kW and 0.76% for the cooling system energy. The control was also stable, with an MAE of 0.08 °C and a maximum error of 1.17 °C. Based on this analysis, a CPS-based control strategy incorporating an ANN-based optimal control algorithm is expected to be an effective energy efficiency solution for existing data center without changing IT equipment or cooling systems.

An Optimal Primary Frequency Control Based on Adaptive Dynamic Programming for Islanded Modernized Microgrids

In many pilot research and development (R&D) microgrid projects, engine-based generators are employed in their power systems, either generating electrical energy or being mixed with the heat and power technology. One of the critical tasks of such engine-based generation units is the frequency regulation in the islanded mode of modernized microgrid (MMG) operation; MMGs are microgrids equipped with advanced controls to address more emerging scenarios in smart grids. For having a stable and reliable MMG, we need to synthesize an optimal, robust, primary frequency controller for the islanded mode of MMG of the future. This task is challenging because of unknown mechanical parameters, occurrence of uncertain disturbances, uncertainty of loads, operating point variations, and the appearance of engine delays, and hence nonminimum phase dynamics. This article presents an innovative primary frequency control for the engine generators regulating the frequency of an islanded MMG in the context of smart grids. The proposed approach is based on an adaptive optimal output-feedback control algorithm using adaptive dynamic programming (ADP). The convergence of algorithms, along with the stability analysis of the closed-loop system, is also shown in this article. Finally, as experimental validation, hardware-in-the-loop (HIL) test results are provided in order to examine the effectiveness of the proposed methodology practically. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article was motivated by the problem of primary frequency controls in modernized microgrids (MMGs) using engine generators, which are still one of the prime sources of regulating frequency in pilot research and development (R&D) microgrid projects. Although MMGs will be integral parts of the smart grid of the future, their primary controls in the islanded mode are not advanced enough and not considering existing theoretical challenges scientifically. Existing approaches to regulate frequency using industrially accepted methods are highly model-based and not optimal. Besides, they are not considering the nonminimum phase dynamics. These dynamics are mainly associated with the engine delays—an inherent issue of mechanical parts—for islanded microgrids. This article suggests a new adaptive optimal output-feedback control approach based on the adaptive dynamic programming (ADP) to the abovementioned problem under consideration. By using the proposed methodology, MMGs can deal with the issues mentioned earlier, which are challenging. The proposed approach is optimally rejecting uncertain disturbances (considering the load uncertainty and operating point variations) and reducing the impacts of nonminimum phase dynamics caused by the engine delay. Based on our currently available hardware-in-the-loop (HIL) device’s capability of modeling power systems’ components in real time, our HIL-based experiments demonstrate that this approach is feasible.

A novel optimal control design for unknown nonlinear systems based on adaptive dynamic programming and nonlinear model predictive control

AbstractThis paper presents a novel adaptive optimal control algorithm by combining adaptive dynamic programming with nonlinear model predictive control for unknown continuous‐time affine nonlinear systems. The adaptive optimal control design is realized by the model‐critic‐actor architecture. Model neural network, critic neural network and actor neural network are constructed to approximate the system dynamics, the cost function and the optimal control law respectively. The random initialization of neural networks usually influences the control performance, so three neural networks are initialized properly to obtain the suitable initial values so that the control performance is improved. Especially, actor neural network is initialized to approximate the near‐optimal control law which is obtained from nonlinear model predictive control. The convergence of the proposed algorithm is proved by the Lyapunov theory. Finally, simulation results are provided to illustrate the effectiveness of the proposed algorithm.

Model-Free Adaptive Optimal Control for Unknown Nonlinear Multiplayer Nonzero-Sum Game.

In this article, an online adaptive optimal control algorithm based on adaptive dynamic programming is developed to solve the multiplayer nonzero-sum game (MP-NZSG) for discrete-time unknown nonlinear systems. First, a model-free coupled globalized dual-heuristic dynamic programming (GDHP) structure is designed to solve the MP-NZSG problem, in which there is no model network or identifier. Second, in order to relax the requirement of systems dynamics, an online adaptive learning algorithm is developed to solve the Hamilton-Jacobi equation using the system states of two adjacent time steps. Third, a series of critic networks and action networks are used to approximate value functions and optimal policies for all players. All the neural network (NN) weights are updated online based on real-time system states. Fourth, the uniformly ultimate boundedness analysis of the NN approximation errors is proved based on the Lyapunov approach. Finally, simulation results are given to demonstrate the effectiveness of the developed scheme.

Reinforcement learning for adaptive optimal control of continuous-time linear periodic systems

This paper studies the infinite-horizon adaptive optimal control of continuous-time linear periodic (CTLP) systems, using reinforcement learning techniques. By means of policy iteration (PI) for CTLP systems, both on-policy and off-policy adaptive dynamic programming (ADP) algorithms are derived, such that the solution of the optimal control problem can be found without the exact knowledge of the system dynamics. Starting with initial stabilizing controllers, the proposed PI-based ADP algorithms converge to the optimal solutions under mild conditions. Application to the adaptive optimal control of the lossy Mathieu equation demonstrates the efficacy of the proposed learning-based adaptive optimal control algorithm.

Adaptive optimal controller design for a class of LDI-based neural network systems with input time-delays

In this paper, a new online adaptive optimal controller design scheme is studied for a class of nonlinear systems with input time-delays. First, we linearize the original nonlinear systems by means of linear differential inclusion technique. Then the adaptive optimal controller of the linearized systems with input time-delays is obtained by online policy iteration algorithm. It also proves the convergence of the designed adaptive optimal control algorithm. Finally, the effectiveness of the proposed method is verified by two simulation examples.

Adaptive optimal target controlled infusion algorithm to prevent hypotension associated with labor epidural: An adaptive dynamic programming approach

Patients receiving labor epidurals commonly experience arterial hypotension as a complication of neuraxial block. The purpose of this study was to design an adaptive optimal controller for an infusion system to regulate mean arterial pressure. A state–space model relating mean arterial pressure to Norepinephrine (NE) infusion rate was derived for controller design. A data-driven adaptive optimal control algorithm was developed based on adaptive dynamic programming (ADP). The stability and disturbance rejection ability of the closed-loop system were tested via a simulation model calibrated using available clinical data. Simulation results indicated that the settling time was six minutes and the system showed effective disturbance rejection. The results also demonstrate that the adaptive optimal control algorithm would achieve individualized control of mean arterial pressure in pregnant patients with no prior knowledge of patient parameters.

Predictive cruise control of connected and autonomous vehicles via reinforcement learning

Predictive cruise control concerns designing controllers for autonomous vehicles using the broadcasted information from the traffic lights such that the idle time around the intersection can be reduced. This study proposes a novel adaptive optimal control approach based on reinforcement learning to solve the predictive cruise control problem of a platoon of connected and autonomous vehicles. First, the reference velocity is determined for each autonomous vehicle in the platoon. Second, a data-driven adaptive optimal control algorithm is developed to estimate the gains of the desired distributed optimal controllers without the exact knowledge of system dynamics. The obtained controller is able to regulate the headway, velocity, and acceleration of each vehicle in a suboptimal sense. The goal of trip time reduction is achieved without compromising vehicle safety and passenger comfort. Numerical simulations are presented to validate the efficacy of the proposed methodology.

Adaptive Optimal Control for a Class of Nonlinear Systems: The Online Policy Iteration Approach.

This paper studies the online adaptive optimal controller design for a class of nonlinear systems through a novel policy iteration (PI) algorithm. By using the technique of neural network linear differential inclusion (LDI) to linearize the nonlinear terms in each iteration, the optimal law for controller design can be solved through the relevant algebraic Riccati equation (ARE) without using the system internal parameters. Based on PI approach, the adaptive optimal control algorithm is developed with the online linearization and the two-step iteration, i.e., policy evaluation and policy improvement. The convergence of the proposed PI algorithm is also proved. Finally, two numerical examples are given to illustrate the effectiveness and applicability of the proposed method.

A novel distributed optimal adaptive control algorithm for nonlinear multi-agent differential graphical games

In this paper, an online optimal distributed learning algorithm is proposed to solve leader-synchronization problem of nonlinear multi-agent differential graphical games. Each player approximates its optimal control policy using a single-network approximate dynamic programming U+0028 ADP U+0029 where only one critic neural network U+0028 NN U+0029 is employed instead of typical actorcritic structure composed of two NNs. The proposed distributed weight tuning laws for critic NNs guarantee stability in the sense of uniform ultimate boundedness U+0028 UUB U+0029 and convergence of control policies to the Nash equilibrium. In this paper, by introducing novel distributed local operators in weight tuning laws, there is no more requirement for initial stabilizing control policies. Furthermore, the overall closed-loop system stability is guaranteed by Lyapunov stability analysis. Finally, Simulation results show the effectiveness of the proposed algorithm.

Adaptive optimal control of unknown nonlinear systems with different time scales

The adaptive optimal control of unknown nonlinear system with different time scales is considered in this paper. The commonly used singular perturbation theory (SPT) to solve this problem is based on the accurately reduced system model, which is extremely difficult to be obtained in practical application. To overcome this difficulty, an adaptive dynamic programing-based optimal control algorithm with the simplified actor–critic–identifier structure is developed. A different time scales dynamic neural network (DTSDNN) identifier with a novel updating law derived from a properly designed Lyapunov function is proposed to estimate the unknown system dynamics. Furthermore, the critic NN with an improved adaptive law considering the NN weight estimation error information is designed, which can achieve faster convergent speed compared with the commonly used gradient method. Lyapunov approach is used to guarantee exponential convergence to a bounded region in the neighborhood of the optimal control and uniformly ultimately bounded (UUB) stability of the closed-loop system. Two examples are provided to illustrate the effectiveness and applicability of the developed approach.

Adaptive optimal control for integrated active front steering and direct yaw moment based on approximate dynamic programming

In this paper, a novel adaptive optimal control algorithm based on approximate dynamic programming (ADP) approach is proposed for integrated active front steering (AFS) and direct yaw moment control (DYC). The corrective yaw moment and active steering angle are generated online without knowing system dynamics, which is realised by using a neural network (NN) identifier to identify the unknown system dynamics and a critic NN to calculate the optimal control action, respectively. Control commands are executed via active steering angle on front wheels and proper brake torque distribution on the effective wheels. Computer simulations under three different driving manoeuvres, i.e., lane change manoeuvre, step steer manoeuvre and sine with dwell manoeuvre, are carried out to evaluate the proposed control method. Simulation results show that the proposed ADP-based control method demonstrates improved tracking performance in terms of enhancing vehicle handling and stability performance when encountering the varying longitudinal velocity, the uncertain cornering stiffness and the different road/tyre friction coefficients. Model-free and self-adaptive properties of the proposed method provide a new solution to vehicle stability controller design instead of the commonly used model-based methods.

Adaptive optimal control for integrated active front steering and direct yaw moment based on approximate dynamic programming

In this paper, a novel adaptive optimal control algorithm based on approximate dynamic programming (ADP) approach is proposed for integrated active front steering (AFS) and direct yaw moment control (DYC). The corrective yaw moment and active steering angle are generated online without knowing system dynamics, which is realised by using a neural network (NN) identifier to identify the unknown system dynamics and a critic NN to calculate the optimal control action, respectively. Control commands are executed via active steering angle on front wheels and proper brake torque distribution on the effective wheels. Computer simulations under three different driving manoeuvres, i.e., lane change manoeuvre, step steer manoeuvre and sine with dwell manoeuvre, are carried out to evaluate the proposed control method. Simulation results show that the proposed ADP-based control method demonstrates improved tracking performance in terms of enhancing vehicle handling and stability performance when encountering the varying longitudinal velocity, the uncertain cornering stiffness and the different road/tyre friction coefficients. Model-free and self-adaptive properties of the proposed method provide a new solution to vehicle stability controller design instead of the commonly used model-based methods.

A Nash game approach to mixed $${\mathcal {H}}_{2}/{\mathcal {H}}_{\infty }$$ H 2 / H ∞ model predictive control: part 2—stability and robustness

This paper is a note on the stability and robustness of the Nash Game (Vamvoudakis et al. in Adaptive optimal control algorithm for zero-sum nash games with integral reinforcement learning, 2012; Ning et al. in Optim Control Appl Methods. doi:10.1002/oca.2042, 2012; Bouyer et al. in Concurrent games with ordered objectives, 2012) based mixed \({\mathcal {H}}_{2}/{\mathcal {H}}_{\infty }\) Model Predictive Controllers (Aadaleesan and Saha in Mixed \(\mathcal {H}_{2}/\mathcal {H}_{\infty }\) Model Predictive Control for Unstable and Non-Minimum Constrained Processes, 2008; Aadaleesan in Nash Game based Mixed \(\mathcal {H}_{2}/\mathcal {H}_{\infty }\) Model Predictive Control applied with Laguerre-Wavelet Network Model, 2011) for linear state feedback systems addressed in Part 1 (Aadaleesan and Saha in Int J Dyn Control, 2016) of this series. The mixed \({\mathcal {H}}_{2}/{\mathcal {H}}_{\infty }\) MPC proposed in Part 1 (Aadaleesan and Saha in Int J Dyn Control, 2016) and that developed by Orukpe et al. (Model predictive control based on mixed \(\mathcal {H}_2/\mathcal {H}_{\infty }\) control approach, 2007) are compared in this Part 2. The issues of stability and robustness of the multi-criterion optimal control are dealt in this paper using set theoretic concepts.