Adaptive Dynamic Programming Algorithm Research Articles

Optimal Voltage Recovery Learning Control for Microgrids with N-Distributed Generations via Hybrid Iteration Algorithm

Considering that the nonlinearity and uncertainty of the microgrid model complicate the derivation and design of the optimal controller, an adaptive dynamic programming (ADP) algorithm is designed to solve the model-free non-zero-sum game. By combining the advantages of policy iteration and value iteration, an optimal learning control scheme based on hybrid iteration is constructed to provide stringent real power sharing for the nonlinear and coupled microgrid systems with N-distributed generations. First, using non-zero-sum differential game strategy, a novel distributed secondary voltage recovery consensus optimal control protocol is built using a hybrid iteration method to realize the voltage recovery of microgrids. Then, the data of the system state and input are gathered along a dynamic system trajectory and a data-driven optimal controller learns the game solution without microgrid physics information, enhancing convenience and efficiency in practical applications. Furthermore, the convergence analysis is given in detail, and it is proved that the control protocol can converge to the optimal solution so as to ensure the stability of the voltage recovery of the microgrid system. Convergence analysis proves the convergence of the the protocol to the optimal solution, ensuring voltage recovery stability. Simulation results validate the feasibility and effectiveness of the proposed scheme.

Zero‐sum game‐based H∞ optimal finite‐time prescribed performance control for nonlinear multiagent systems with actuator faults

AbstractThis article proposes an optimal leader‐follower consensus control protocol with finite‐time prescribed performance for nonlinear multiagent systems suffering from the actuator fault based on the zero‐sum game framework. First, to guarantee the specified performance of the system within a finite time frame, a novel error transformation function (ETF) is constructed. The ETF proposed in this paper can adjust the mapping range of error transformation to achieve better error convergence performance, thereby enhancing the flexibility of the controller design. Then, to compensate for the negative impact of actuator fault, an error‐related item is separated from the optimal performance index function to design an online fault estimator, which can obtain a better error estimation accuracy. Furthermore, to lessen the need for difficult‐to‐verify persistent excitation signal in adaptive dynamic programming algorithm, both historical and real‐time data are applied to adjust the updating law of neural network weight. Moreover, it demonstrates that the signals within the closed‐loop system are bounded and the expected finite‐time prescribed performance can be ensured in the presence of faults. Finally, a numerical example of multiple single‐link robots manipulator system is executed to validate the efficacy of the developed scheme.

Data‐driven adaptive optimal control for discrete‐time periodic systems

AbstractIn this paper, a problem of data‐driven optimal control is studied for discrete‐time periodic systems with unknown system matrices and input matrices. For this problem, a value iteration‐based adaptive dynamic programming algorithm is proposed to obtain the suboptimal controller. The core of the algorithm proposed in this paper is to obtain an approximation of the unique positive definite solution of the algebraic Riccati equation and the optimal feedback gain matrix by using the collected real‐time data of the system states and control inputs. Without an initial stabilizing feedback gain, the proposed algorithm could be activated by an arbitrary bounded control input. Finally, the effectiveness of the proposed approach is demonstrated by two examples.

Adaptive event‐triggered control and observer design for discrete‐time nonlinear Markov jump systems with DoS attacks using policy iteration‐based adaptive dynamic programming

AbstractThis article addresses adaptive event‐triggered discrete‐time nonlinear Markov jump systems (MJs) with DoS attacks, where the introduced DoS attacks are considered as more general stochastic models with fixed trigger frequency and duration. To solve the optimal control problem, we use an adaptive dynamic programming (ADP) algorithm based on policy iteration (PI). The approximate process is as follows: the performance index function (PIF) is first updated by the iteration policy in advance, and the control policy is obtained from the PIF. Subsequently, an approximate estimation of the optimal PIF and the optimal control policy is made using the actor‐critic structure obtained through neural network techniques. In order to reduce the occupied communication resources required for control policy iteration, we introduce an adaptive event triggering mechanism with an adaptive triggering threshold, which reduces the conservatism of resource occupation by the PIF compared to the fixed‐threshold ETM. In addition, an observer identifying the unknown dynamics part of the system is designed. Finally, using the Lyapunov function, it is shown that the designed control policy ensures the stability and convergence of the MJS, and the designed observer is effective. Simulation examples are given to verify the feasibility of the controller and the observer.

Neural network-based adaptive critic control for saturated nonlinear systems with full state constraints via a novel event-triggered mechanism

This paper investigates the issue of event-triggered optimal control for saturated nonlinear systems with full state constraints. A smooth function is defined to map the constrained states, then the considered system can be transformed into a state unconstrained system with unknown approximate errors. In addition, a novel event-triggered mechanism is proposed via an adaptive dynamic programming algorithm, which can obtain the optimal control law without constructing the so-called event-triggered HJB equation. Moreover, in order to conquer the difficulty caused by the persistence of excitation conditions, the experience replay technique is utilized to design the critic update law. Meanwhile, it is demonstrated that all signals are uniformly ultimately bounded. Finally, two simulation examples are given to demonstrate the effectiveness of the control strategy.

Optimal Spin Polarization Control for the Spin-Exchange Relaxation-Free System Using Adaptive Dynamic Programming.

This work is the first to solve the 3-D spin polarization control (3DSPC) problem of atomic ensembles, which controls the spin polarization to achieve arbitrary states with the cooperation of multiphysics fields. First, a novel adaptive dynamic programming (ADP) structure is proposed based on the developed multicritic multiaction neural network (MCMANN) structure with nonquadratic performance functions, as a way to solve the multiplayer nonzero-sum game (MP-NZSG) problem in 3DSPC under the constraints of asymmetric saturation inputs. Then, we utilize the MCMANNs to implement the multicritic multiaction ADP (MCMA-ADP) algorithm, whose convergence is proven by the compression mapping principle. Finally, the MCMA-ADP is deployed in the spin-exchange relaxation-free (SERF) system to provide a set of control laws in 3DSPC that fully exploits the multiphysics fields to achieve arbitrary spin polarization states. Numerical simulations support the theoretical results.

Evolution-guided value iteration for optimal tracking control

In this article, an evolution-guided value iteration (EGVI) algorithm is established to address optimal tracking problems for nonlinear nonaffine systems. Conventional adaptive dynamic programming algorithms rely on gradient information to improve the policy, which adheres to the first order necessity condition. Nonetheless, these methods encounter limitations when gradient information is intricate or system dynamics lack differentiability. In response to this challenge, evolutionary computation is leveraged by EGVI to search for the optimal policy without requiring an action network. The competition within the policy population serves as the driving force for policy improvement. Therefore, EGVI can effectively handle complex and non-differentiable systems. Additionally, this innovative method has the potential to enhance exploration efficiency and bolster the robustness of algorithms due to its population-based characteristics. Furthermore, the convergence of the algorithm and the stability of the policy are investigated based on the EGVI framework. Finally, the effectiveness of the established method is comprehensively demonstrated through two simulation experiments.

Homotopic ADP based [formula omitted] integral sliding mode secure control for Markov jumping cyber-physical systems

This paper addresses the sliding mode secure control problem for continuous-time cyber-physical systems (CPSs) subject to unmatched disturbances and actuator deception attacks. To capture different operating conditions and account for random component failures in CPSs, a Markov jumping model is introduced to characterize the system's component matrices. Next, a novel model-free integral sliding mode function and a corresponding sliding mode controller are presented to mitigate the impact of external disturbances and deception attacks on Markov jumping cyber-physical systems. An algorithm based on adaptive dynamic programming (ADP) and homotopic policy iteration (PI) is applied to develop the sliding mode controller, called the online homotopic ADP algorithm, which avoids the requirement for an initial stabilizing matrix compared to existing PI algorithms. Lastly, the mode-independent integral sliding mode control scheme is studied to a two-mode Markov jumping vertical take-off and landing helicopter system, providing guaranteed reachability of sliding variables and stability of sliding mode dynamics. Importantly, this is achieved without any reliance on knowledge of system dynamics.

Adaptive Virotherapy Strategy for Organism With Constrained Input Using Medicine Dosage Regulation Mechanism.

In this article, the constrained adaptive control strategy based on virotherapy is investigated for organism using the medicine dosage regulation mechanism (MDRM). First, the tumor-virus-immune interaction dynamics is established to model the relations among the tumor cells (TCs), virus particles, and the immune response. The adaptive dynamic programming (ADP) method is extended to approximately obtain the optimal strategy for the interaction system to reduce the populations of TCs. Due to the consideration of asymmetric control constraints, the nonquadratic functions are proposed to formulate the value function such that the corresponding Hamilton-Jacobi-Bellman equation (HJBE) is derived which can be deemed as the cornerstone of ADP algorithms. Then, the ADP method of a single-critic network architecture which integrates MDRM is proposed to obtain the approximate solutions of HJBE and eventually derive the optimal strategy. The design of MDRM makes it possible for the dosage of the agentia containing oncolytic virus particles to be regulated timely and necessarily. Furthermore, the uniform ultimate boundedness of the system states and critic weight estimation errors is validated by Lyapunov stability analysis. Finally, simulation results are given to show the effectiveness of the derived therapeutic strategy.

Optimal synchronization for multi-agent systems: A performance-dependent switching topology

This paper proposes an optimal output synchronization control method for heterogeneous multi-agent systems (HMASs) under a performance-dependent switching topology and DoS attacks. First, local and global switched performance index functions (SPIF) and SPIF-dependent topology switching law are proposed, respectively, thus, the control performance and topology quality can be quantitatively expressed. Second, an adaptive dynamic programming (ADP) algorithm with mode switching is proposed, aimed at dealing with the difficult Hamilton–Jacobi–Bellman equation, as well as the analytical complexity caused by the switching dynamics. The convergence of the switched ADP algorithm is proven to ensure its correct implementation. Then, for different topologies, multi-mode Actor–Critic neural networks (NNs) are built for each agent to calculate optimized control policies and SPIF, respectively. Furthermore, an NN-based state compensation mechanism is designed to expand the applicability of the designed switched ADP algorithm when the leader’s output transmission is unreliable. Finally, the results of numerical examples confirm that the proposed method is feasible.

Output information-based intermittent optimal control for continuous-time nonlinear systems with unmatched uncertainties via adaptive dynamic programming

Intermittent control stands as a valuable strategy for resource conservation and cost reduction across diverse systems. Nonetheless, prevailing research is intractable to address the challenges posed by robust optimal intermittent control of nonlinear input-affine systems with unmatched uncertainties. This paper aims to fill this gap. Initially, we introduce an enhanced finite-time intermittent control approach to ensure stability within nonlinear dynamic systems harboring bounded errors. A neural networks (NNs) state observer is constructed to estimate system information. Subsequently, an optimal intermittent controller that operates within a finite time span, guaranteeing system stability by employing the Hamilton–Jacobi–Bellman (HJB) methodology. Furthermore, we devise an output information-based event-triggered intermittent (ETI) approach rooted in the robust adaptive dynamic programming (ADP) algorithm, furnishing an optimal intermittent control law. In this process, a critic NNs is introduced to estimate the cost function and optimal intermittent controller. Simulation results show that our proposed method is superior to existing intermittent control strategies.

Linear quadratic stochastic optimal control with state- and control-dependent noises: A deterministic data approach

This paper investigates an infinite-horizon linear quadratic stochastic optimal control (LQSOC) problem with state- and control-dependent noises, in which two system matrices are unknown. First, a deterministic system is introduced and a relationship among its system trajectory, its control and the matrices to be solved is formulated. Subsequently, based on the deterministic system, an adaptive dynamic programming (ADP) algorithm is established to tackle the LQSOC problem. Then, the convergence analysis is presented by proving the equivalence between the proposed algorithm and an existing policy iteration algorithm. Finally, the effectiveness of the obtained algorithm is verified by a perturbed turbocharged diesel engine optimal control design example.

Hybrid Iteration ADP Algorithm to Solve Cooperative, Optimal Output Regulation Problem for Continuous-Time, Linear, Multiagent Systems: Theory and Application in Islanded Modern Microgrids With IBRs

This paper proposes a novel adaptive dynamic programming (ADP) algorithm, named hybrid iteration (HI), to solve the cooperative, optimal output regulation problem (CO <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> RP) for continuous-time, linear, multi-agent systems. Unlike traditional ADP algorithms, i.e., policy iteration (PI) and value iteration (VI), HI does not need an initial stabilizing control policy required by PI. At the same time, it maintains a faster convergence rate compared with VI. First, a model-based HI algorithm is proposed to solve the CO <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> RP. Based on the proposed HI algorithm, a data-driven, adaptive, optimal controller is developed to solve the cooperative, adaptive, optimal output regulation problem without using any information about the physics of the system. Instead, the states/input information collected along the trajectories of the dynamic system is employed. The proposed data-driven HI is applied to the adaptive, optimal secondary voltage control (also known as voltage restoration control) of an islanded modern microgrid based on inverter-based resources. Compared with the VI and PI algorithms, comparative simulation results demonstrate that the proposed HI approach is significantly able to save the convergence time of the central processing unit (also known as CPU) deployed, reduce the number of learning iterations, and remove the requirement of the initial stabilizing control policy. Comparative experiments reveal the practicality and superiority of the proposed methodology.

Distributed Optimal Attitude Synchronization Control of Multiple QUAVs via Adaptive Dynamic Programming.

This article proposes a distributed optimal attitude synchronization control strategy for multiple quadrotor unmanned aerial vehicles (QUAVs) through the adaptive dynamic programming (ADP) algorithm. The attitude systems of QUAVs are modeled as affine nominal systems subject to parameter uncertainties and external disturbances. Considering attitude constraints in complex flying environments, a one-to-one mapping technique is utilized to transform the constrained systems into equivalent unconstrained systems. An improved nonquadratic cost function is constructed for each QUAV, which reflects the requirements of robustness and the constraints of control input simultaneously. To overcome the issue that the persistence of excitation (PE) condition is difficult to meet, a novel tuning rule of critic neural network (NN) weights is developed via the concurrent learning (CL) technique. In terms of the Lyapunov stability theorem, the stability of the closed-loop system and the convergence of critic NN weights are proved. Finally, simulation results on multiple QUAVs show the effectiveness of the proposed control strategy.

Neuro-Optimal Event-Triggered Impulsive Control for Stochastic Systems via ADP.

This article presents a novel neural-network-based optimal event-triggered impulsive control method. First, a novel general-event-based impulsive transition matrix (GITM) is constructed to represent the probability distribution evolving characteristics regarding all system states across the impulsive actions, rather than the prefixed timing sequence. On the foundation of this GITM, the event-triggered impulsive adaptive dynamic programming (ETIADP) algorithm and its high-efficiency version (HEIADP) are developed to deal with the optimization problems for stochastic systems with event-triggered impulsive controls. It is shown that the obtained controller design scheme can reduce the computational and communication burden caused by updating the controller periodically. By analyzing the admissibility, monotonicity, and optimality properties of ETIADP and HEIADP, we further establish the approximation error bound of the neural networks to address the connection between the ideal and neural-network-based realizations of the present methods. It is proven that the iterative value functions of both the ETIADP and HEIADP algorithms fall in a small neighborhood of the optimum as the iteration index increases to infinity. By adopting a novel task synchronization mechanism, the proposed HEIADP algorithm fully utilizes the computing resources of multiprocessor systems (MPSs), while significantly reducing the memory requirement compared to traditional ADP approaches. Finally, we carry out a numerical study to show that the proposed methods can fulfill the desired goals.

Event-Triggered Robust Adaptive Dynamic Programming for Multiplayer Stackelberg-Nash Games of Uncertain Nonlinear Systems.

In this article, an event-triggered robust adaptive dynamic programming (ETRADP) algorithm is developed to solve a class of multiplayer Stackelberg-Nash games (MSNGs) for uncertain nonlinear continuous-time systems. Considering the different roles of players in the MSNG, the hierarchical decision-making process is described as the designed value functions for the leader and all followers, which assist to transform the robust control problem of the uncertain nonlinear system into an optimal regulation problem of the nominal system. Then, an online policy iteration algorithm is formulated to solve the derived coupled Hamilton-Jacobi equation. Meanwhile, an event-triggered mechanism is designed to alleviate computational and communication burdens. Moreover, critic neural networks (NNs) are constructed to obtain the event-triggered approximate optimal control polices for all players, which constitute the Stackelberg-Nash equilibrium of the MSNG. By using Lyapunov's direct method, the stability of the closed-loop uncertain nonlinear system is guaranteed under the ETRADP-based control scheme in the sense of uniform ultimate boundedness. Finally, a numerical simulation is provided to demonstrate the effectiveness of the present ETRADP-based control scheme.

Aperiodically Intermittent Event-Triggered Optimal Average Consensus for Nonlinear Multi-Agent Systems.

This article is concerned with average consensus of multi-agent systems via intermittent event-triggered strategy. First, a novel intermittent event-triggered condition is designed and the corresponding piecewise differential inequality for the condition is established. Using the established inequality, several criteria on average consensus are obtained. Second, the optimality has been investigated based on average consensus. The optimal intermittent event-triggered strategy in the sense of Nash equilibrium and corresponding local Hamilton-Jacobi-Bellman equation are derived. Third, the adaptive dynamic programming algorithm for the optimal strategy and its neural network implementation with actor-critic architecture are also given. Finally, two numerical examples are presented to show the feasibility and effectiveness of our strategies.

Online Adaptive Dynamic Programming for Optimal Self-Learning Control of VTOL Aircraft Systems With Disturbances

In this paper, a novel online adaptive dynamic programming control algorithm is developed to solve optimal control problems for Vertical Take Off and Landing (VTOL) aircraft systems. Considering the input disturbances of the VTOL systems and adding perturbation interference term to the value function, a robust policy iteration-based ADP method will be introduced to obtain the optimal controller with an approximate optimal control approach. The convergence property is developed to ensure the value function converges to a finite neighborhood of the optimal one. The uniform ultimate boundedness (UUB) of the iterative errors will be strictly proved by Lyapunov stability theory. Finally, we will give mathematical simulation results with the developed method. Compared with sliding mode control (SMC) and linear quadratic regulator (LQR) methods, the simulation results illustrate better overall performance of the novel method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —As an effective tool to solve the optimal control problem of complex nonlinear systems, adaptive dynamic programming (ADP) can effectively overcome the computational problems caused by “curse of dimensionality”. VTOL aircraft system is a typical strongly coupled, underactuate, non-minimum phase system, and the sudden random disturbances will make the control task more challenging. Aiming at the particularity of VTOL system, a new online policy iteration-based ADP algorithm is developed to obtain optimal control with an approximate optimal control strategy in this paper. Decoupling and disturbances can be effectively avoided. convergence will be analyzed to make sure the value function converge to a finite neighborhood of the optimal value function. The UUB property of the system will be proved by the Lyapunov approach.

A Hybrid Controller for Musculoskeletal Robots Targeting Lifting Tasks in Industrial Metaverse.

In manufacturing, musculoskeletal robots have gained more attention with the potential advantages of flexibility, robustness, and adaptability over conventional serial-link rigid robots. Focusing on the fundamental lifting tasks, a hybrid controller is proposed to overcome control challenges of such robots for widely applications in industry. The metaverse technology offers an available simulated-reality-based platform to verify the proposed method. The hybrid controller contains two main parts. A muscle-synergy-based radial basis function (RBF) network is proposed as the feedforward controller, which is able to characterize the phasic and the tonic muscle synergies simultaneously. The adaptive dynamic programming (ADP) is applied as the feedback controller to address the optimal control problem. The actor-critic structure is applied in the ADP-based controller, where the critic network is trained to approximate the optimal performance index and the actor network is trained to compute the optimal muscle excitations. Furthermore, the convergence and stability of the ADP algorithm are also analyzed. Finally, experiments have been designed to verify the effectiveness of this hybrid controller on an upper limb musculoskeletal system, and the comparisons with other controllers are also illustrated. The results show that the proposed controller can obtain a satisfactory performance for lifting tasks.

Neural Adaptive H∞ Sliding-Mode Control for Uncertain Nonlinear Systems with Disturbances Using Adaptive Dynamic Programming.

This paper focuses on a neural adaptive H∞ sliding-mode control scheme for a class of uncertain nonlinear systems subject to external disturbances by the aid of adaptive dynamic programming (ADP). First, by combining the neural network (NN) approximation method with a nonlinear disturbance observer, an enhanced observer framework is developed for estimating the system uncertainties and observing the external disturbances simultaneously. Then, based on the reliable estimations provided by the enhanced observer, an adaptive sliding-mode controller is meticulously designed, which can effectively counteract the effects of the system uncertainties and the separated matched disturbances, even in the absence of prior knowledge regarding their upper bounds. While the remaining unmatched disturbances are attenuated by means of H∞ control performance on the sliding surface. Moreover, a single critic network-based ADP algorithm is employed to learn the cost function related to the Hamilton-Jacobi-Isaacs equation, and thus, the H∞ optimal control is obtained. An updated law for the critic NN is proposed not only to make the Nash equilibrium achieved, but also to stabilize the sliding-mode dynamics without the need for an initial stabilizing control. In addition, we analyze the uniform ultimate boundedness stability of the resultant closed-loop system via Lyapunov's method. Finally, the effectiveness of the proposed scheme is verified through simulations of a single-link robot arm and a power system.