Adaptive Optimal Control Research Articles

Exponentially stable adaptive optimal control of uncertain LTI systems

SummaryA novel method of an adaptive linear quadratic (LQ) regulation of uncertain continuous linear time‐invariant systems is proposed. Such an approach is based on the direct self‐tuning regulators design framework and the exponentially stable adaptive control technique developed earlier by the authors. Unlike the known solutions, a procedure is proposed to obtain a non‐overparametrized regression equation (RE) with respect to the unknown controller parameters from an initial RE of the LQ‐based reference tracking control system. On the basis of such result, an adaptive law is proposed, which under mild regressor finite excitation condition provides monotonous convergence of the LQ‐controller parameters to an adjustable set of their true values, which bound is defined only by the machine precision. Using the Lyapunov‐based analysis, it is proved that the mentioned law guarantees the exponential stability of the closed‐loop adaptive optimal control system. The simulation examples are provided to validate the theoretical contributions.

Inverse optimally adaptive neural output‐feedback control of stochastic nonlinear systems

SummaryIn this article, for a class of stochastic nonlinear systems with non‐strict feedback, a neural adaptive inverse optimal output feedback control design scheme is presented. First, according to the existing inverse optimal criterion, an auxiliary system is established. On this basis, a novel observer is built to evaluate the unpredictable states. Second, in the control process, neural networks (NNs) are applied to estimate the unknown functions. Based on NNs and the backstepping technology, an adaptive neural inverse optimal output feedback controller is established. It is indicated that the proposed scheme could ensure the semi‐globally uniformly ultimately bounded of the closed‐loop system and also achieve the objective of inverse optimality. Eventually, an example is applied to testify the feasibility of this scheme.

An efficient model‐free adaptive optimal control of continuous‐time nonlinear non‐zero‐sum games based on integral reinforcement learning with exploration

AbstractTo reduce the learning time and space occupation, this study presents a novel model‐free algorithm for obtaining the Nash equilibrium solution of continuous‐time nonlinear non‐zero‐sum games. Based on the integral reinforcement learning method, a new integral HJ equation that can quickly and cooperatively determine the Nash equilibrium strategies of all players is proposed. By leveraging the neural network approximation and gradient descent method, simultaneous continuous‐time adaptive tuning laws are provided for both critic and actor neural network weights. These laws facilitate the estimation of the optimal value function and optimal policy without requiring knowledge or identification of the system's dynamics. The closed‐loop system stability and convergence of weights are guaranteed through the Lyapunov analysis. Additionally, the algorithm is enhanced to reduce the number of auxiliary NNs used in the critic. The simulation results for a two‐player non‐zero‐sum game validate the effectiveness of the proposed algorithm.

Error-based adaptive optimal tracking control of nonlinear discrete-time systems

Error-based adaptive optimal tracking control of nonlinear discrete-time systems

Adaptive optimal backstepping control for strict-feedback nonlinear systems with time-varying partial output constraints

This paper investigates the optimal backstepping control strategy for strict-feedback nonlinear systems subject to input saturation and time-varying partial output constraints. Compared with the existing results on output constraints, the time-varying partial output constraint is a more general constraint that can start and end at any time during the system operation, or remain unconstrained, which can be called as output constraints occurring in a limited time interval (OCOLT). A special barrier function is embedded into the optimal performance index function to satisfy the OCOLT condition under the optimal control framework, and a shift function is introduced to address the function discontinuity caused by the OCOLT problem. Then, an auxiliary system and a disturbance observer are respectively constructed to handle the influence of the input saturation and compounded disturbances. Meanwhile, the simplified reinforcement learning algorithm, employing the identifier-critic-actor architecture, is developed to achieve the optimal control objective within the framework of the backstepping technology. Moreover, the proposed optimal control method ensures the semi-globally uniformly ultimately bounded of all signals within the closed-loop system. Finally, two simulation examples are given to further prove the effectiveness of the presented optimal control approach.

Adaptive optimal control of under-actuated robotic systems using a self-regulating nonlinear weight-adjustment scheme: Formulation and experimental verification.

This paper formulates an innovative model-free self-organizing weight adaptation that strengthens the robustness of a Linear Quadratic Regulator (LQR) for inverted pendulum-like mechatronic systems against perturbations and parametric uncertainties. The proposed control procedure is devised by using an online adaptation law to dynamically adjust the state weighting factors of LQR's quadratic performance index via pre-calibrated state-error-dependent hyperbolic secant functions (HSFs). The updated state-weighting factors re-compute the optimal control problem to modify the state-compensator gains online. The novelty of the proposed article lies in adaptively adjusting the variation rates of the said HSFs via an auxiliary model-free online self-regulation law that uses dissipative and anti-dissipative terms to flexibly re-calibrate the nonlinear function's waveforms as the state errors vary. This augmentation increases the controller's design flexibility and enhances the system's disturbance rejection capacity while economizing control energy expenditure under every operating condition. The proposed self-organizing LQR is analyzed via customized hardware-in-loop (HIL) experiments conducted on the Quanser's single-link rotational inverted pendulum. As compared to the fixed-gain LQR, the proposed SR-EM-STC delivers an improvement of 52.2%, 16.4%, 55.2%, and 42.7% in the pendulum's position regulation behavior, control energy expenditure, transient recovery duration, and peak overshoot, respectively. The experimental outcomes validate the superior robustness of the proposed scheme against exogenous disturbances.

A new optimal adaptive backstepping control approach for nonlinear systems under deception attacks via reinforcement learning

In this paper, a new optimal adaptive backstepping control approach for nonlinear systems under deception attacks via reinforcement learning is presented in this paper. The existence of nonlinear terms in the studied system makes it very difficult to design the optimal controller using traditional methods. To achieve optimal control, RL algorithm based on critic–actor architecture is considered for the nonlinear system. Due to the significant security risks of network transmission, the system is vulnerable to deception attacks, which can make all the system state unavailable. By using the attacked states to design coordinate transformation, the harm brought by unknown deception attacks has been overcome. The presented control strategy can ensure that all signals in the closed-loop system are semi-globally ultimately bounded. Finally, the simulation experiment is shown to prove the effectiveness of the strategy.

Research on adaptive optimal control strategy of virtual synchronous generator inertia and damping parameters

AbstractAs the ratio of newly installed power generation utilizing power electronic converters increases, it results in a decrease in the overall inertia and damping of the power system, as well as the decrease in its ability to resist interference. Aiming to address this issue, the authors propose an adaptive optimal control approach for the parameter in the virtual synchronous generator (VSG). This strategy considers the power angle curve and analyzes the principles of inertia and damping within the oscillation period and its range of values. By coordinating the values of parameters, it achieves the optimization of these parameters. At the same time, in order to take into account the demand for fast regulation response under normal operation control conditions and the demand for synchronization support under disturbance fault conditions, a multi‐mode switching control strategy including virtual synchronization control and conventional PQ control is further proposed. Based on MATLAB/Simulink software, the strategies mentioned above have been proven through simulation. The findings testify the proposed method can significantly enhance the regulation response characteristics of the VSG by optimizing its parameters. Additionally, it enables smooth switching of the control mode independently, based on the specific requirements of different control scenarios. This approach fully leverages the virtual synchronous support and fast regulation characteristics of the power electronic converter, and has practical engineering value.

Reinforcement learning-based optimal adaptive fuzzy control for nonlinear multi-agent systems with prescribed performance

In this paper, the problem of optimal adaptive consensus tracking control for nonlinear multi-agent systems with prescribed performance is investigated. To address the issue of satisfying the initial value conditions in existing results, an improved performance function is employed as the prescribed performance boundary, effectively resolving this problem. Then, by employing the error transformation function, the constrained system is converted into an unconstrained one. Furthermore, fuzzy logic systems are employed to identify unknown system parts. By applying the dynamic surface technique, the problem of "differential explosion", which often occurs in backstepping, is solved. Moreover, a distributed optimal adaptive fuzzy control protocol based on the reinforcement learning actor-critic algorithm is proposed. Under the proposed control scheme, it is proved that all the signals within the closed-loop system are bounded, and the consensus tracking errors have remained within the predefined bounds. Finally, the numerical simulation results demonstrate the effectiveness of the proposed scheme.

A volatile fatty acids adaptive observer-based hierarchical optimal controller design to maximum gas production of two-stage anaerobic digestion process

Volatile fatty acids (VFA), which influence the stability and gas production, are important state variables in anaerobic digestion (AD) process. Three main components of VFA can indicate the metabolic state of AD internal environment and be considered as a better basis for AD monitoring strategies. A VFA adaptive observer (VFA-AO) is proposed to estimate unknown gas production kinetic parameters and unmeasurable states simultaneously, including the concentration of three main components of VFA. Otherwise, using the estimated information, a VFA-AO-based hierarchical optimal controller (VFA-AO-based HOC) is designed to maximize gas production rates of AD. The proposed controller is compared with two existing control algorithms. Simulation results show that, despite the concentration of pollutants in the inlet fluctuates, VFA-AO achieves a reliable states and kinetic parameters estimation, and VFA-AO-based HOC maximizes gas production.

Adaptive hierarchical optimization control for electrohydraulic suspension with resistor-capacitor operator

The controller design for integrated electrohydraulic suspension with leaf spring is a complicated and challenging task, aiming to achieve optimal dynamic performance for heavy vehicles. Considering the hysteresis property of leaf spring, parameter uncertainties and nonlinearity, a novel hierarchical optimization control strategy including upper and bottom controllers is proposed to effectively suppress the vertical vibration of vehicle body. To describe the hysteresis property of leaf spring, a novel resistor-capacitor operator hysteresis model is identified by experiment and optimized by genetic algorithm. Based on hysteresis model, the filter adaptive backstepping control with sprung mass uncertainty is designed as upper controller to generate the desired active force for electrohydraulic actuator. Then, the projection adaptive fuzzy sliding mode control is designed as bottom controller for nonlinear actuator with time-varying fluid parameter to accurately track the required target force. Additionally, a stratified sampling algorithm is investigated to optimize the parameters of hierarchical control strategy for addressing multi-objective optimization issue of electrohydraulic suspension with leaf spring. Finally, the results show that the proposed method has better robustness and effectiveness, and it can ensure ride comfort, safety and optimal performance.

Adaptive fixed-time inverse optimal consensus of multi-agent systems with limited-time interval state constraints

This article presents a study on fixed-time adaptive fuzzy inverse optimal consensus for multi-agent systems (MASs) subject to limited-time interval state constraints. To handle the constraints that occur within a limited-time interval during the system's operating cycle, we introduce a shifting function to shift the boundary and a state-dependent transformation function to transform the constrained MASs into non-constrained ones. We then propose an adaptive fixed-time inverse optimal consensus controller that achieves the convergence of consensus error within a fixed-time and inverse optimality without directly learning the solution of the Hamilton-Jacobi-Bellman (HJB) equations. To establish the criteria for fixed-time stabilization, we design an auxiliary controller instead of the auxiliary system and introduce two Lyapunov functions to prove the inverse optimality and fixed-time stabilization. The developed consensus scheme ensures that the synchronization error asymptotically converges to a user-predefined region within a fixed settling time. We validate the effectiveness of the developed scheme through a simulation example.

Optimal adaptive fuzzy fault-tolerant control applied on a quadrotor attitude stabilization based on particle swarm optimization

This article discusses the problem of stabilizing the attitude control of a quadrotor system, which is subject to uncertainty, external disturbances, and sensor and actuator faults. To address these challenges, the control stage employs universal approximators, such as fuzzy systems, which estimate the system’s uncertainties and eliminate nonaffine nonlinear actuator faults. In addition, the particle swarm optimization technique is used to adjust the adaptive parameters and fuzzy initial values. The robust control term is carefully designed to handle approximation errors, time-varying sensors, and external disturbances. To solve the issue of the unavoidable algebraic loop during the actuator approximation phase, a Butterworth low-pass filter is integrated. This approach automatically deals with external disturbances, and no further approximation is necessary. Furthermore, the controller can be reconfigured online to enable fast-fault compensation without requiring a fault detection or isolation unit. To prove the global stability and boundedness of all signals in the closed-loop system, Lyapunov theory is used.

Decentralized Adaptive Neural Inverse Optimal Control of Nonlinear Interconnected Systems.

Existing methods on decentralized optimal control of continuous-time nonlinear interconnected systems require a complicated and time-consuming iteration on finding the solution of Hamilton-Jacobi-Bellman (HJB) equations. In order to overcome this limitation, in this article, a decentralized adaptive neural inverse approach is proposed, which ensures the optimized performance but avoids solving HJB equations. Specifically, a new criterion of inverse optimal practical stabilization is proposed, based on which a new direct adaptive neural strategy and a modified tuning functions method are proposed to design a decentralized inverse optimal controller. It is proven that all the closed-loop signals are bounded and the goal of inverse optimality with respect to the cost functional is achieved. Illustrative examples validate the performance of the methods presented.

Optimal model-free adaptive control based on reinforcement Q-Learning for solar thermal collector fields

Optimal model-free adaptive control based on reinforcement Q-Learning for solar thermal collector fields

An Online Model-Following Projection Mechanism Using Reinforcement Learning

In this paper, we propose a model-free adaptive learning solution for a model-following control problem. This approach employs policy iteration, to find an optimal adaptive control solution. It utilizes a moving finite-horizon of model-following error measurements. In addition, the control strategy is designed by using a projection mechanism that employs Lagrange dynamics. It allows for real-time tuning of derived actor-critic structures to find the optimal model-following strategy and sustain optimized adaptation performance. Finally, the efficacy of the proposed framework is emphasized through a comparison with sliding mode and high-order model-free adaptive control approaches.

A small-gain approach to inverse optimal adaptive control of nonlinear systems with unmodeled dynamics

This work formulates and solves the problem of inverse optimal adaptive control for nonlinear systems with parametric and dynamic uncertainties. A concept and sufficient conditions for solving this problem are first introduced and derived based on adaptive control Lyapunov function method. One difficulty obstructing current works to solve this problem is that inverse optimality and stability of the closed-loop system are not necessarily achieved simultaneously in the presence of both parametric uncertainties and unmodeled dynamics. To this end, a new auxiliary system and a cost functional that puts integral penalty on the state, control, parameter estimate and unmodeled dynamics are proposed to develop a new inverse optimal adaptive feedback approach. Then a small-gain approach is proposed to establish the links between inverse optimality and stability of the closed-loop system. It is proved that if inverse optimality is achieved and the small-gain condition holds, the closed-loop system is uniformly bounded and the system state converges to an arbitrarily small neighborhood of the origin. Finally, an example is given to illustrate the obtained results.

Adaptive Optimal Control of Linear Discrete-Time Networked Control Systems with Two-Channel Stochastic Dropouts

Adaptive Optimal Control of Linear Discrete-Time Networked Control Systems with Two-Channel Stochastic Dropouts

Adaptive optimal secure wind power generation control for variable speed wind turbine systems via reinforcement learning

As the utilization of wind energy continues to grow, it is crucial to prioritize the identification of vulnerabilities, raise awareness, and develop strategies for cybersecurity defense. False data injection (FDI) attacks, if targeted at the communication between the rotor speed sensor and the wind turbine (WT) controller, can potentially disrupt the normal operation of the system. These attacks have the capability to overload the drive-train and significantly reduce the power generation efficiency of the wind turbine. So, this note presents an adaptive optimal secure control strategy entailing reinforcement learning (RL) neural network (NN) using the filter error to compensate the detrimental effects of FDI attack as well as actuator fault for WT systems. The Hamilton–Jacobi–Bellman (HJB) equation is constructed and solved to obtain the optimal control policy. Since the HJB is inextricably intertwined with intrinsic nonlinearity and complexity, solving this equation is quiet challenging. To tackle this issue and also approximate the solution of the HJB, an actor–critic-based reinforcement learning (RL) strategy is used, in which actor and critic NNs are applied to execute control action and assess control performance, respectively. To detect FDI attack, an anomaly detection is developed using a nonlinear observer/estimator. Stability analysis is performed using Lyapunov theory which guarantees semi-global uniformly ultimately bounded (SGUUB) of the error signal. Finally, simulation results verify the effectiveness of the proposed control approach.

Evolution, Design, and Future Trajectories on Bipedal Wheel-legged Robot: A Comprehensive Review

This comprehensive review delves into the realm of bipedal wheel-legged robots, focusing on their design, control, and applications in assistive technology and disaster mitigation. Drawing insights from various fields such as robotics, computer science, and biomechanics, it offers a holistic understanding of these robots' stability, adaptability, and efficiency. The analysis encompasses optimization techniques, sensor integration, machine learning, and adaptive control methods, evaluating their impact on robot capabilities. Emphasizing the need for adaptable, terrain-aware control algorithms, the review explores the untapped potential of machine learning and soft robotics in enhancing performance across diverse operational scenarios. It highlights the advantages of hybrid models combining legged and wheeled mobility while stressing the importance of refining control frameworks, trajectory planning, and human-robot interactions. The concept of integrating soft and compliant mechanisms for improved adaptability and resilience is introduced. Identifying gaps in current research, the review suggests future directions for investigation in the fields of robotics and control engineering, addressing the evolution and terrain adaptability of bipedal wheel-legged robots, control, stability, and locomotion, as well as integrated sensory and perception systems, microcontrollers, cutting-edge technology, and future design and control directions.