Iterative Adaptive Dynamic Programming Research Articles

An Iterative Adaptive Dynamic Programming Approach for Macroscopic Fundamental Diagram-Based Perimeter Control and Route Guidance

Macroscopic fundamental diagrams (MFDs) have been widely adopted to model the traffic flow of large-scale urban networks. Coupling perimeter control and regional route guidance (PCRG) is a promising strategy to decrease congestion heterogeneity and reduce delays in large-scale MFD-based urban networks. For MFD-based PCRG, one needs to distinguish between the dynamics of (a) the plant that represents reality and is used as the simulation tool and (b) the model that contains easier-to-measure states than the plant and is used for devising controllers, that is, the model-plant mismatch should be considered. Traditional model-based methods (e.g., model predictive control (MPC)) require an accurate representation of the plant dynamics as the prediction model. However, because of the inherent network uncertainties, such as uncertain dynamics of heterogeneity and demand disturbance, MFD parameters could be time-varying and uncertain. Conversely, existing data-driven methods (e.g., reinforcement learning) do not consider the model-plant mismatch and the limited access to plant-generated data, for example, subregional OD-specific accumulations. Therefore, we develop an iterative adaptive dynamic programming (IADP)-based method to address the limited data source induced by the model-plant mismatch. An actor-critic neural network structure is developed to circumvent the requirement of complete information on plant dynamics. Performance comparisons with other PCRG schemes under various scenarios are carried out. The numerical results indicate that the IADP controller trained with a limited data source can achieve comparable performance with the “benchmark” MPC approach using perfect measurements from the plant. The results also validate the IADP’s robustness against various uncertainties (e.g., demand noise, MFD error, and trip distance heterogeneity) when minimizing the total time spent in the urban network. These results demonstrate the great potential of the proposed scheme in improving the efficiency of multiregion MFD systems. Funding: The work was jointly funded by the National Natural Science Foundation of China under [Grant 72071214] (R. Zhong), the Dit4Tram project from the European Union’s Horizon 2020 Research and Innovation Programme under [Grant 953783] (N. Geroliminis), and the Research Student Attachment Programme of The Hong Kong Polytechnic University (C. Chen). Supplemental Material: The online appendix is available at https://doi.org/10.1287/trsc.2023.0091 .

Novel Discounted Adaptive Critic Control Designs With Accelerated Learning Formulation.

Inspired by the successive relaxation method, a novel discounted iterative adaptive dynamic programming framework is developed, in which the iterative value function sequence possesses an adjustable convergence rate. The different convergence properties of the value function sequence and the stability of the closed-loop systems under the new discounted value iteration (VI) are investigated. Based on the properties of the given VI scheme, an accelerated learning algorithm with convergence guarantee is presented. Moreover, the implementations of the new VI scheme and its accelerated learning design are elaborated, which involve value function approximation and policy improvement. A nonlinear fourth-order ball-and-beam balancing plant is used to verify the performance of the developed approaches. Compared with the traditional VI, the present discounted iterative adaptive critic designs greatly accelerate the convergence rate of the value function and reduce the computational cost simultaneously.

Convergence Analysis of Value Iteration Adaptive Dynamic Programming for Continuous-Time Nonlinear Systems.

This article is concerned with the convergence property and error bounds analysis of value iteration (VI) adaptive dynamic programming for continuous-time (CT) nonlinear systems. The size relationship between the total value function and the single integral step cost is described by assuming a contraction assumption. Then, the convergence property of VI is proved while the initial condition is an arbitrary positive semidefinite function. Moreover, the accumulated effects of approximation errors generated in each iteration are taken into consideration while using approximators to implement the algorithm. Based on the contraction assumption, the error bounds condition is proposed, which ensures the approximated iterative results converge to a neighborhood of the optimum, and the relation between the optimal solution and approximated iterative results is also derived. To make the contraction assumption more concrete, an estimation way is proposed to derive a conservative value of the assumption. Finally, three simulation cases are given to validate the theoretical results.

An integrated structure of operational performance-oriented monitoring and degradation recovery based on iterative adaptive dynamic programming

Driven by the increasing industrial demands for process safety and product quality, this paper is devoted to operational performance monitoring and performance degradation recovery for closed-loop control systems under abnormal working conditions. To this end, a new operational performance indicator is designed and its benchmark with an explicit sensitivity criteria for selecting historical data, to detect the system degradation timely and accurately. Then, by embedding the operational performance indicator as a trigger condition, a data-driven performance degradation recovery method with the aid of iterative adaptive dynamic programming (ADP) algorithm is investigated. This implementation shows that iterative ADP can be applied efficiently by utilizing both value iteration and measurement data to recover the anomaly induced system performance degradation. Finally, a case study on the continuous stirred tank reactor system is provided to validate the effectiveness of these proposed methods.

Policy Iteration for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Aimed at infinite horizon optimal control problems of discrete time-varying nonlinear systems, in this paper, a new iterative adaptive dynamic programming algorithm, which is the discrete-time time-varying policy iteration (DTTV) algorithm, is developed. The iterative control law is designed to update the iterative value function which approximates the index function of optimal performance. The admissibility of the iterative control law is analyzed. The results show that the iterative value function is non-increasingly convergent to the Bellman-equation optimal solution. To implement the algorithm, neural networks are employed and a new implementation structure is established, which avoids solving the generalized Bellman equation in each iteration. Finally, the optimal control laws for torsional pendulum and inverted pendulum systems are obtained by using the DTTV policy iteration algorithm, where the mass and pendulum bar length are permitted to be time-varying parameters. The effectiveness of the developed method is illustrated by numerical results and comparisons.

Dynamic event-triggered control for discrete-time nonlinear Markov jump systems using policy iteration-based adaptive dynamic programming

This paper investigates a dynamic event-triggered optimal control problem of discrete-time (DT) nonlinear Markov jump systems (MJSs) via exploring policy iteration (PI) adaptive dynamic programming (ADP) algorithms. The performance index function (PIF) defined in each subsystem is updated by utilizing an online PI algorithm, and the corresponding control policy is derived via solving the optimal PIF. Then, we adopt neural network (NN) techniques, including an actor network and a critic network, to estimate the iterative PIF and control policy. Moreover, the designed dynamic event-triggered mechanism (DETM) is employed to avoid wasting additional resources when the estimated iterative control policy is updated. Finally, based on the Lyapunov difference method, it is proved that the system stability and the convergence of all signals can be guaranteed under the developed control scheme. A simulation example for DT nonlinear MJSs with two system modes is presented to demonstrate the feasibility of the control design scheme.

Hamiltonian-Driven Adaptive Dynamic Programming With Approximation Errors

In this article, we consider an iterative adaptive dynamic programming (ADP) algorithm within the Hamiltonian-driven framework to solve the Hamilton-Jacobi-Bellman (HJB) equation for the infinite-horizon optimal control problem in continuous time for nonlinear systems. First, a novel function, "min-Hamiltonian," is defined to capture the fundamental properties of the classical Hamiltonian. It is shown that both the HJB equation and the policy iteration (PI) algorithm can be formulated in terms of the min-Hamiltonian within the Hamiltonian-driven framework. Moreover, we develop an iterative ADP algorithm that takes into consideration the approximation errors during the policy evaluation step. We then derive a sufficient condition on the iterative value gradient to guarantee closed-loop stability of the equilibrium point as well as convergence to the optimal value. A model-free extension based on an off-policy reinforcement learning (RL) technique is also provided. Finally, numerical results illustrate the efficacy of the proposed framework.

Spiking Adaptive Dynamic Programming Based on Poisson Process for Discrete-Time Nonlinear Systems.

In this article, a new iterative spiking adaptive dynamic programming (SADP) method based on the Poisson process is developed to solve optimal impulsive control problems. For a fixed time interval, combining the Poisson process and the maximum likelihood estimation (MLE), the three-tuple of state, spiking interval, and probability of Poisson distribution can be computed, and then, the iterative value functions and iterative control laws can be obtained. A property analysis method is developed to show that the value functions converge to optimal performance index function as the iterative index increases from zero to infinity. Finally, two simulation examples are given to verify the effectiveness of the developed algorithm.

Fault-Tolerant Optimal Spacecraft Attitude Maneuver: An Incremental Model Approach

This paper addresses the fault-tolerant optimal attitude control problem for a rigid spacecraft with external disturbance, model uncertainties, and actuator faults. Firstly, the incremental nonlinear control technology is used to simplify the attitude control system into an incremental nominal model with a synthetic uncertainty/fault term that is estimated by a nonsingular terminal sliding mode disturbance observer. Secondly, considering a quadratic performance index, the original fault-tolerant optimal control problem is transformed into a guaranteed performance optimal problem of the nominal model. Benefiting from the simplicity of the incremental model, such a modified optimal control problem is solved online by the modified policy iteration adaptive dynamic programming without any initial stable policy assumption. Finally, the attitude angles of the closed-loop system are proved theoretically to be uniformly ultimately bounded at the desired state by using the Lyapunov methods. Simulation results are given to verify the effectiveness of the proposed fault-tolerant optimal attitude control strategy.

Robust oxygen excess ratio control of PEMFC systems using adaptive dynamic programming

To improve the power generation efficiency and cells lifetime of proton exchange membrane fuel cells, the oxygen excess ratio (OER) control problem is investigated in this paper. With the use of the double-loop cascade control structure, our work reinforces the iteration adaptive dynamic programming (ADP) algorithm and develops a robust OER control strategy drawing upon rich tools and techniques in sliding mode control theory. Our contribution is threefold. First, the proposed OER control policy is insensitive to various dynamics and disturbances by using refined super-twisting sliding mode controller in the external control loop. Second, the ADP-based controller is employed in the internal control loop to approximate the real optimal tracking control policy. The algorithm is computationally efficient and easy to implement in hardware platforms. Third, simulation results show that the proposed OER control strategy has a quick, smooth response, which can be further refined by means of numerous advanced neural network algorithms.

Event-triggered optimal control for discrete-time multi-player non-zero-sum games using parallel control

This paper presents a novel event-triggered optimal control (ETOC) method for discrete-time (DT) multi-player non-zero-sum games (NZSGs). First, a novel event-triggered algorithm is developed for DT multi-player NZSGs based on the time-triggered optimal value functions. Therefore, the developed event-triggered algorithm only needs to solve the time-triggered Hamilton–Jacobi-Bellman (HJB) equations. Then, the asymptotic stability of the closed-loop system is proved. Additionally, we show that an upper bound for the sum of the actual performance indices of all the players can be determined in advance. A key step in the implementation of the developed event-triggered algorithm is to obtain the next state of the actual system, which is difficult to implement on the actual system. Thus, a parallel control method is utilized to predict the next state by constructing the parallel system for the actual system and obtain the optimal value functions. The method that combines the developed event-triggered algorithm and parallel control is called the event-triggered optimal parallel control (ETOPC) method. The neural network (NN) technique and the iterative adaptive dynamic programming (IADP) technique are employed in parallel control. Moreover, the control stability is shown further in the consideration of the NN weight approximation errors. Finally, two simulations justify the theoretical conjectures.

Neural‐network‐based control for discrete‐time nonlinear systems with denial‐of‐service attack: The adaptive event‐triggered case

AbstractThis article investigates a neural network (NN)‐based control problem for unknown discrete‐time nonlinear systems with a denial‐of‐service (DoS) attack and an adaptive event‐triggered mechanism (ETM). The considered DoS attacks are described by the occurrence frequency and durations and hence more general in comparison with existing stochastic models. To the addressed problem, a novel adaptive rule adjusting the triggering threshold of ETM is constructed to govern the communication schedule, and an NN‐based observer is designed to identify the system dynamics where a piecewise update rule of NN weights is adopted to handle the challenge of the complex time series coming from both ETM and DoS attacks. In light of this kind of protocol‐ and attack‐induced switched systems, a sufficient condition dependent on the occurrence frequency and durations of DoS attacks is elaborately established via the analysis of input‐to‐state stability. Furthermore, an iteration adaptive dynamic programming approach is proposed to handle the addressed control issue, and the boundedness is discussed to both the estimation errors of the Luenberger‐type observer and the identified errors of NN weights of observer networks as well as actor‐critic networks with the help of the Lyapunov theory. Finally, a simulation example is utilized to illustrate the usefulness of the proposed controller design scheme.

Optimal DoS attack scheduling for multi-sensor remote state estimation over interference channels

In this paper, we investigate the optimal denial-of-service attack scheduling problems in a multi-sensor case over interference channels. Multiple attackers aim to degrade the performance of remote state estimation under attackers’ energy constraints. The attack decision of one attacker may be affected by the others while all attackers find their own optimal strategies to degrade estimation performance. Consequently, the Markov decision process and Markov cooperative game in two different information scenarios are formulated to study the optimal attack strategies for multiple attackers. Because of the complex computations of the high-dimensional Markov decision process (Markov cooperative game) as well as the limited information for attackers, we propose a value iteration adaptive dynamic programming method to approximate the optimal solution. Moreover, the structural properties of the optimal solution are analyzed. In the Markov cooperative game, the optimal joint attack strategy which admits a Nash equilibrium is studied. Several numerical simulations are provided to illustrate the feasibility and effectiveness of the main results.

Policy Iteration and Event-Triggered Robust Adaptive Dynamic Programming for Large-Scale Systems

Abstract In this paper, an event-triggered robust optimal control approach is proposed for large-scale systems with both parametric and dynamic uncertainties through robust adaptive dynamic programming, policy iteration, and small-gain techniques. By using the input and output data, the unmeasurable states are reconstructed instead of constructing a Luenberger observer. Starting from an admissible control policy, an event-based feedback control policy is learned to save the communication resources and reduce the number of control updates. The closed-loop stability and the convergence of the proposed algorithm are analyzed by using Lyapunov and small-gain techniques. A practical example of multimachine power systems with governor controllers is given to demonstrate the effectiveness of the proposed method.

A partial policy iteration ADP algorithm for nonlinear neuro-optimal control with discounted total reward

This paper constructs a partial policy iteration adaptive dynamic programming (ADP) algorithm to solve the optimal control problem of nonlinear systems with discounted total reward. Compared with traditional policy iteration ADP algorithm, the approach updates the iterative control law only in a local region of the global system state space. With the benefit of this feature, the overall computational burden at each iteration for processing units can be significantly reduced. Hence, this feature enables our algorithm to be successfully executed on low-performance devices such as smartphones, smartwatches and the Internet of Things (IoT) objects. We provide the convergency analysis to show that the generated sequence of value functions is monotonically nonincreasing and can finally reach a local optimum. In addition, the corresponding local policy space is developed theoretically for the first time. Besides, when the sequence of the local system state spaces is chosen properly, we prove that the developed algorithm is capable of finding the global optimal performance index function for the nonlinear systems. Finally, we present a numerical simulation to demonstrate the effectiveness of the proposed algorithm.

Approximately Optimal Control of Discrete-Time Nonlinear Switched Systems Using Globalized Dual Heuristic Programming

Based on the idea of data-driven control, a novel iterative adaptive dynamic programming (ADP) algorithm based on the globalized dual heuristic programming (GDHP) technique is used to solve the optimal control problem of discrete-time nonlinear switched systems. In order to solve the Hamilton–Jacobi–Bellman (HJB) equation of switched systems, the iterative ADP method is proposed and the strict convergence analysis is also provided. Three neural networks are constructed to implement the iterative ADP algorithm, where a novel model network is designed to identify the system dynamics, a critic network is used to approximate the cost function and its partial derivatives, and an action network is provided to obtain the approximate optimal control law. Two simulation examples are described to illustrate the effectiveness of the proposed method by comparing with the heuristic dynamic programming (HDP) and dual heuristic programming (DHP) methods.

Optimal SINR-Based DoS Attack Scheduling for Remote State Estimation via Adaptive Dynamic Programming Approach

This article considers the energy-limited denial-of-service attack scheduling problem on remote state estimation under signal-to-interference-plus-noise ratio-based channels. The goal of the attacker is to design the optimal attack strategy to degrade the control performance of cyber-physical systems and to reduce his energy consumption. First, to weigh the importance between the current and future rewards, an optimization problem with a discount factor is formulated, which is used to reflect the attacker’s goal. Next, a Markov decision problem (MDP) is formulated to solve the optimization problem. Due to the difficulty of solving the high-dimensional MDP with unknown transition and reward functions, a value iteration adaptive dynamic programming method is proposed to achieve an approximate optimal solution. Also, convergence analysis of the proposed algorithm is carried out. Finally, simulation results are presented to show the efficiency and feasibility of the obtained results.

Improved value iteration for neural-network-based stochastic optimal control design

In this paper, a novel value iteration adaptive dynamic programming (ADP) algorithm is presented, which is called an improved value iteration ADP algorithm, to obtain the optimal policy for discrete stochastic processes. In the improved value iteration ADP algorithm, for the first time we propose a new criteria to verify whether the obtained policy is stable or not for stochastic processes. By analyzing the convergence properties of the proposed algorithm, it is shown that the iterative value functions can converge to the optimum. In addition, our algorithm allows the initial value function to be an arbitrary positive semi-definite function. Finally, two simulation examples are presented to validate the effectiveness of the developed method.

Data-Driven Nonlinear Near-Optimal Regulation Based on Multi-Dimensional Taylor Network Dynamic Programming

Using the data-driven control formulation, an iterative dynamic programming approach which is based on a multi-dimensional Taylor network is established to design the near optimal regulation of discrete-time nonlinear systems. For discrete-time general nonlinear systems, the iterative adaptive dynamic programming algorithm is developed and proved to guarantee the property of convergence and optimality. Three networks are constructed, namely, the identification network, critic network and action network. Moreover, a globalized dual heuristic programming technique with detailed implementation is developed. The cost function and its derivative can be approximated by this novel architecture. Besides, without the consideration of the system dynamics, this technique can learn the near-optimal control law simultaneously and adaptively. In addition, this technique greatly improves the existing results of the iterative adaptive dynamic programming algorithm in terms of reducing the requirement of the control matrix. Furthermore, because of the approach that is based on the multi-dimensional Taylor network, the amount of calculation needed is also greatly reduced. The simulation experiment is described to illustrate the effectiveness of the data-driven optimal regulation method proposed in this paper.

Data-Based Nonaffine Optimal Tracking Control Using Iterative DHP Approach

Abstract In this paper, a data-based optimal tracking control approach is developed by involving the iterative dual heuristic dynamic programming algorithm for nonaffine systems. In order to gain the steady control corresponding to the desired trajectory, a novel strategy is established with regard to the unknown system function. Then, according to the iterative adaptive dynamic programming algorithm, the updating formula of the costate function and the new optimal control policy for unknown nonaffine systems are provided to solve the optimal tracking control problem. Moreover, three neural networks are used to facilitate the implementation of the proposed algorithm. In order to improve the accuracy of the steady control corresponding to the desired trajectory, we employ a model network to directly approximate the unknown system function instead of the error dynamics. Finally, the effectiveness of the proposed method is demonstrated through a simulation example.