Adaptive Dynamic Programming Approach Research Articles

Adaptive cruise control via adaptive dynamic programming with experience replay

The adaptive cruise control (ACC) problem can be transformed to an optimal tracking control problem for complex nonlinear systems. In this paper, a novel highly efficient model-free adaptive dynamic programming (ADP) approach with experience replay technology is proposed to design the ACC controller. Experience replay increases the data efficiency by recording the available driving data and repeatedly presenting them to the learning procedure of the acceleration controller in the ACC system. The learning framework that combines ADP with experience replay is described in detail. The distinguishing feature of the algorithm is that when estimating parameters of the critic network and the actor network with gradient rules, the gradients of historical data and current data are used to update parameters concurrently. It is proved with Lyapunov theory that the weight estimation errors of the actor network and the critic network are uniformly ultimately bounded under the novel weight update rules. The learning performance of the ACC controller implemented by this ADP algorithm is clearly demonstrated that experience replay can increase data efficiency significantly, and the approximate optimality and adaptability of the learned control policy are tested with typical driving scenarios.

Data-Driven Distributed Output Consensus Control for Partially Observable Multiagent Systems.

This paper is concerned with a class of optimal output consensus control problems for discrete linear multiagent systems with the partially observable system state. Since the optimal control policy depends on the full system state which is not accessible for a partially observable system, traditionally, distributed observers are employed to recover the system state. However, in many situations, the accurate model of a real-world dynamical system might be difficult to obtain, which makes the observer design infeasible. Furthermore, the optimal consensus control policy cannot be analytically solved without system functions. To overcome these challenges, we propose a data-driven adaptive dynamic programming approach that does not require the complete system inner state. The key idea is to use the input and output sequence as an equivalent representation of the underlying state. Based on this representation, an adaptive dynamic programming algorithm is developed to generate the optimal control policy. For the implementation of this algorithm, we design a neural network-based actor-critic structure to approximate the local performance indices and the control polices. Two numerical simulations are used to demonstrate the effectiveness of our method.

Linear-quadratic optimal control for unknown mean-field stochastic discrete-time system via adaptive dynamic programming approach

We develop an adaptive dynamic programming (ADP) approach to deal with the linear-quadratic (LQ) optimal control problem with unknown discrete-time mean-field stochastic system in this paper. At first, the mean-field stochastic LQ problem is transformed into the deterministic case by system transition. Secondly, a value iteration ADP approach is proposed and convergence proof is also discussed. Once again, in order to achieve the iteration method without any knowledge of the dynamics, back propagation neural network (BPNN) is applied to design model network, critic network and action network to ensure unknown system model, value function and control strategy, respectively. At last, it is demonstrated that the ADP approach is valid through simulation.

Connected cruise control with delayed feedback and disturbance: An adaptive dynamic programming approach

SummaryThis paper studies the connected cruise control problem for a platoon of human‐operated and autonomous vehicles. The autonomous vehicles can receive motional data, ie, headway and velocity information from other vehicles by wireless vehicle‐to‐vehicle communication. The use of wireless communications in information exchange between vehicles inevitably causes input delay in the platooning system. Meanwhile, unpredictable behaviors of the leading vehicle constitute exogenous disturbance for the system. An adaptive optimal control problem with input delay and disturbance is formulated, and a novel data‐driven control solution is proposed such that each vehicle in the platoon can achieve safe distance and desired velocity. By adopting an adaptive dynamic programming technique with sampled‐data system theory, a data‐driven adaptive optimal control approach is proposed for autonomous vehicles by the learning strategies of policy iteration without the accurate knowledge of the dynamics of all human drivers and vehicles. The efficacy of the proposed controller is substantiated by rigorous analysis and validated by simulation results in different scenarios.

Boundary Control of 2-D Burgers' PDE: An Adaptive Dynamic Programming Approach.

In this paper, an adaptive dynamic programming-based near optimal boundary controller is developed for partial differential equations (PDEs) modeled by the uncertain Burgers' equation under Neumann boundary condition in 2-D. Initially, Hamilton-Jacobi-Bellman equation is derived in infinite-dimensional space. Subsequently, a novel neural network (NN) identifier is introduced to approximate the nonlinear dynamics in the 2-D PDE. The optimal control input is derived by online estimation of the value function through an additional NN-based forward-in-time estimation and approximated dynamic model. Novel update laws are developed for estimation of the identifier and value function online. The designed control policy can be applied using a finite number of actuators at the boundaries. Local ultimate boundedness of the closed-loop system is studied in detail using Lyapunov theory. Simulation results confirm the optimizing performance of the proposed controller on an unstable 2-D Burgers' equation.

Data-based approximate optimal control for nonzero-sum games of multi-player systems using adaptive dynamic programming

This paper investigates the non-zerosum game issue for unknown nonlinear systems with multi-player by using data-based adaptive dynamic programming (ADP) methods. It is known that the traditional ADP approaches require accurate system models to compute the solutions of non-zerosum games. However, for the practical nonlinear systems, system models are difficult to be obtained and thus these methods will be invalid. To overcome this difficulty, we propose two neural-network-based identification schemes. Combined with the identification results, we design a data-based actor-critic algorithm to learn and approximate the optimal solutions in real time. Subsequently, in order to reduce the computation burden of dual network algorithm, we develop a single network one. To test the feasibility and validity of our schemes, we provide two simulation examples including a linear one and a nonlinear one.

Data-based adaptive neural network optimal output feedback control for nonlinear systems with actuator saturation

This paper investigates the adaptive neural network optimal output feedback control design problem for nonlinear continuous-time systems with actuator saturation. The system dynamics and states of the controlled system are unknown. A neural network state observer is constructed to estimate the system states. This paper uses two neural networks, one is used to construct the neural network state observer, the other (critic neural network) is used to approximate the cost functions, which comprise an observer-critic architecture. In this architecture, the critic neural network weights are tuned based on both the current data and the previous data, thus the conditions of the persistent excitation in the previous literatures are relaxed. By utilizing adaptive dynamic programming approach, a new observer-based optimal control scheme is developed. It is proved that the proposed adaptive neural network output feedback optimal control scheme can ensure that the whole closed-loop system is stable. Moreover, the estimate errors of the critic neural network weights are asymptotically stable. A simulation example is given to validate the effectiveness of the proposed method.

Adaptive Critic Design with Local Gaussian Process Models

In this paper, local Gaussian process (GP) approximation is introduced to build the critic network of adaptive dynamic programming (ADP). The sample data are partitioned into local regions, and for each region, an individual GP model is utilized. The nearest local model is used to predict a given state-action point. With the two-phase value iteration method for a Gaussian-kernel (GK)-based critic network which realizes the update of the hyper-parameters and value functions simultaneously, fast value function approximation can be achieved. Combining this critic network with an actor network, we present a local GK-based ADP approach. Simulations were carried out to demonstrate the feasibility of the proposed approach.

A Hybrid-Adaptive Dynamic Programming Approach for the Model-Free Control of Nonlinear Switched Systems

This paper presents a hybrid adaptive dynamic programming (hybrid-ADP) approach for determining the optimal continuous and discrete control laws of a switched system online, solely from state observations. The new hybrid-ADP recurrence relationships presented are applicable to model-free control of switched hybrid systems that are possibly nonlinear. The computational complexity and convergence of the hybrid-ADP approach are analyzed, and the method is validated numerically showing that the optimal controller and value function can be learned iteratively online from state observations.

A climate change adaptive dynamic programming approach to optimize eucalypt stand management scheduling: a Portuguese application

The aim of this paper is to present approaches to optimize stand-level, short-rotation coppice management planning, taking into account uncertainty in stand growth due to climate change. The focus is on addressing growth uncertainty through a range of climate scenarios so that an adaptive capacity may be possible and the vulnerability of the stand to climate change may be reduced. The optimization encompasses finding both the harvest age in each cycle and the number of coppice cycles within a full rotation that maximize net present revenue. The innovation lies in the combination of the process-based model (Glob3PG) with two dynamic programming (DP) approaches. The former is able to project growth of eucalypt stands under climate change scenarios. The innovative approaches are thus influential to define the management policy (e.g., stool thinning, number of coppice cycles, and cycle length) that maximizes net present revenue taking into account uncertainty in forest growth due to climate change. In both approaches, the state of the system is defined by the number of years since plantation, whereas DP stages are defined by the cumulative number of harvests. The first approach proposes the optimal policy under each climate change scenario at each state. The second approach addresses further situations when the climate scenario is unknown at the beginning of the planning horizon. Both help address uncertainty in an adaptive framework, as a set of readily available options is proposed for each scenario. Results of an application to a typical Eucalyptus globulus Labill. stand in central Portugal are discussed.

Nonlinear and Adaptive Suboptimal Control of Connected Vehicles: A Global Adaptive Dynamic Programming Approach

This paper studies the cooperative adaptive cruise control (CACC) problem of connected vehicles with unknown nonlinear dynamics. Different from the present literature on CACC, data-driven feedforward and optimal feedback control policies are developed by global adaptive dynamic programming (GADP). Due to the presence of nonvanishing disturbance, a modified version of GADP is presented. Interestingly, the developed policy is guaranteed to globally stabilize the vehicular platoon system, and is robust to unmeasurable nonvanishing disturbance. Numerical simulation results are presented to validate the effectiveness of the developed approach.

A Review of Fuzzy Logic and Neural Network Based Intelligent Control Design for Discrete-Time Systems

Over the last few decades, the intelligent control methods such as fuzzy logic control (FLC) and neural network (NN) control have been successfully used in various applications. The rapid development of digital computer based control systems requires control signals to be calculated in a digital or discrete-time form. In this background, the intelligent control methods developed for discrete-time systems have drawn great attentions. This survey aims to present a summary of the state of the art of the design of FLC and NN-based intelligent control for discrete-time systems. For discrete-time FLC systems, numerous remarkable design approaches are introduced and a series of efficient methods to deal with the robustness, stability, and time delay of FLC discrete-time systems are recommended. Techniques for NN-based intelligent control for discrete-time systems, such as adaptive methods and adaptive dynamic programming approaches, are also reviewed. Overall, this paper is devoted to make a brief summary for recent progresses in FLC and NN-based intelligent control design for discrete-time systems as well as to present our thoughts and considerations of recent trends and potential research directions in this area.

Online finite-horizon optimal learning algorithm for nonzero-sum games with partially unknown dynamics and constrained inputs

In this paper, an online optimal learning algorithm based on adaptive dynamic programming (ADP) approach is designed to solve the finite-horizon optimal control for multi-player nonzero-sum games with partially unknown dynamics and constrained control inputs. Firstly, it is proved that the online policy iteration (PI) algorithm is equivalent to Newton׳s iteration. Secondly, the single neural networks (NNs) with time-varying activation functions for each player are used to approximate the time-varying solution to the coupled Hamilton–Jacobi–Bellman (HJB) equations in an online and forward-in-time manner. Control constraints are handled through non-quadratic functions. The convergence of NN-based online optimal learning algorithm for the multi-player nonzero-sum games is also proved. Finally, a simulation example illustrates the effectiveness of the proposed algorithm.

Infinite-time stochastic linear quadratic optimal control for unknown discrete-time systems using adaptive dynamic programming approach

In this paper, an adaptive dynamic programming (ADP) algorithm based on value iteration (VI) is proposed to solve the infinite-time stochastic linear quadratic (SLQ) optimal control problem for the linear discrete-time systems with completely unknown system dynamics. Firstly, the SLQ control problem is converted into the deterministic problem through system transformation and then an iterative ADP algorithm is introduced to solve the optimal control problem with convergence analysis. Secondly, for the implementation of the iteration algorithm, a neural network (NN) is used to identify the unknown system and then the other two NNs are employed to approximate the cost function and the control gain matrix. Lastly, the effectiveness of the iterative ADP approach is illustrated by two simulation examples.

QoS Differential Scheduling in Cognitive-Radio-Based Smart Grid Networks: An Adaptive Dynamic Programming Approach

As the next-generation power grid, smart grid will be integrated with a variety of novel communication technologies to support the explosive data traffic and the diverse requirements of quality of service (QoS). Cognitive radio (CR), which has the favorable ability to improve the spectrum utilization, provides an efficient and reliable solution for smart grid communications networks. In this paper, we study the QoS differential scheduling problem in the CR-based smart grid communications networks. The scheduler is responsible for managing the spectrum resources and arranging the data transmissions of smart grid users (SGUs). To guarantee the differential QoS, the SGUs are assigned to have different priorities according to their roles and their current situations in the smart grid. Based on the QoS-aware priority policy, the scheduler adjusts the channels allocation to minimize the transmission delay of SGUs. The entire transmission scheduling problem is formulated as a semi-Markov decision process and solved by the methodology of adaptive dynamic programming. A heuristic dynamic programming (HDP) architecture is established for the scheduling problem. By the online network training, the HDP can learn from the activities of primary users and SGUs, and adjust the scheduling decision to achieve the purpose of transmission delay minimization. Simulation results illustrate that the proposed priority policy ensures the low transmission delay of high priority SGUs. In addition, the emergency data transmission delay is also reduced to a significantly low level, guaranteeing the differential QoS in smart grid.

Policy iteration optimal tracking control for chaotic systems by using an adaptive dynamic programming approach**Project supported by the National Natural Science Foundation of China (Grant Nos. 61034002, 61233001, 61273140, 61304086, and 61374105) and the Beijing Natural Science Foundation, China (Grant No. 4132078).

A policy iteration algorithm of adaptive dynamic programming (ADP) is developed to solve the optimal tracking control for a class of discrete-time chaotic systems. By system transformations, the optimal tracking problem is transformed into an optimal regulation one. The policy iteration algorithm for discrete-time chaotic systems is first described. Then, the convergence and admissibility properties of the developed policy iteration algorithm are presented, which show that the transformed chaotic system can be stabilized under an arbitrary iterative control law and the iterative performance index function simultaneously converges to the optimum. By implementing the policy iteration algorithm via neural networks, the developed optimal tracking control scheme for chaotic systems is verified by a simulation.

Power System Stability Control for a Wind Farm Based on Adaptive Dynamic Programming

In this paper, a goal representation heuristic dynamic programming (GrHDP) based controller is developed for the doubly-fed induction generator based wind farm to improve the system transient stability under fault conditions. The proposed controller is based on adaptive dynamic programming (ADP) techniques to approximate the optimal control policy according to the interaction between the controller and the power plant. Compared to existing ADP approaches with one action network and one critic network, our GrHDP architecture introduces an additional network, i.e., the reference network, to form an internal goal/reward representation. This better mapping between the system state and the control action significantly improves the control performance. The effectiveness of the proposed approach is validated via two cases. The first case investigates a revised four-machine two-area system with high wind penetration and a static synchronous compensator. The second case is a practical size power system with wind farm in Liaoning Province in China. Detailed simulation analysis and comparative studies with traditional ADP approaches are presented to demonstrate the superior performance of our method.

Two-Phase Iteration for Value Function Approximation and Hyperparameter Optimization in Gaussian-Kernel-Based Adaptive Critic Design

Adaptive Dynamic Programming (ADP) with critic-actor architecture is an effective way to perform online learning control. To avoid the subjectivity in the design of a neural network that serves as a critic network, kernel-based adaptive critic design (ACD) was developed recently. There are two essential issues for a static kernel-based model: how to determine proper hyperparameters in advance and how to select right samples to describe the value function. They all rely on the assessment of sample values. Based on the theoretical analysis, this paper presents a two-phase simultaneous learning method for a Gaussian-kernel-based critic network. It is able to estimate the values of samples without infinitively revisiting them. And the hyperparameters of the kernel model are optimized simultaneously. Based on the estimated sample values, the sample set can be refined by adding alternatives or deleting redundances. Combining this critic design with actor network, we present a Gaussian-kernel-based Adaptive Dynamic Programming (GK-ADP) approach. Simulations are used to verify its feasibility, particularly the necessity of two-phase learning, the convergence characteristics, and the improvement of the system performance by using a varying sample set.

Global Optimal Output Regulation of Partially Linear Systems via Robust Adaptive Dynamic Programming

A non-model-based robust adaptive dynamic programming (RADP) approach is developed to solve the global optimal output regulation problem (GOORP) of partially linear systems with nonlinear dynamic uncertainties. By converting the output regulation problem into a global robust optimal stabilization problem, a robust optimal controller is designed by means of the RADP for the transformed system. Nonlinear small-gain theory is applied to conclude the global asymptotic stability for the closed-loop system and thus solves the original GOORP. Simulation results illustrate the effectiveness of the proposed methodology.

Neuro-optimal tracking control for a class of discrete-time nonlinear systems via generalized value iteration adaptive dynamic programming approach

In this paper, a novel value iteration adaptive dynamic programming (ADP) algorithm, called "generalized value iteration ADP" algorithm, is developed to solve infinite horizon optimal tracking control problems for a class of discrete-time nonlinear systems. The developed generalized value iteration ADP algorithm permits an arbitrary positive semi-definite function to initialize it, which overcomes the disadvantage of traditional value iteration algorithms. Convergence property is developed to guarantee that the iterative performance index function will converge to the optimum. Neural networks are used to approximate the iterative performance index function and compute the iterative control policy, respectively, to implement the iterative ADP algorithm. Finally, a simulation example is given to illustrate the performance of the developed algorithm.