Heuristic Dynamic Programming Algorithm Research Articles

Adaptive neural event‐triggered near‐optimal control for affined uncertain nonlinear discrete‐time system

AbstractA novel event‐triggered heuristic dynamic programming (HDP) algorithm is proposed for the near‐optimal control of uncertain discrete‐time nonlinear input‐affine systems. Based on input‐to‐state stability (ISS) analysis, a new event‐triggered mechanism (ETM) is designed. Under constant coefficients, a Lipschitz‐like assumption that forms the basis of the event‐triggering condition is considered to be conservative. To further reduce the conservativeness of the triggering condition and enlarge the average interevent time, an adaptive threshold parameter is utilized in the proposed ETM. In the HDP algorithm framework, model, critic, and action network are adopted to achieve state estimation, approximation to the optimal cost function, and solution to Hamilton–Jacobian–Bellman (HJB) equation. Under the proposed event‐triggered HDP algorithm, the closed system is proved to possess semiglobal uniform ultimate boundedness (SGUUB). Finally, by conducting simulation, it shows that on the premise of satisfying control performance, the event‐triggered strategy can realize reduction on the updating frequency of the controller.

Research on Production Scheduling of Industrial Big Data for Internet of Things Based on Dynamic Planning Algorithm

Abstract In this paper, the optimized Dual Heuristic Dynamic Programming (DHP) algorithm is studied in depth, compared and analyzed with the other three algorithms through experimental simulation, and the DHP algorithm is applied to different industrial accurate scheduling tests to explore the loads of the industrial equipment during scheduling operation. The simulation results show that the dual-heuristic dynamic programming DHP algorithm has a significant decrease in both the mean waiting time (MWT) and the mean response time (MRT), where the highest peak value of the mean waiting time is only 0.06901 μs, and the highest peak value of the mean response time is 0.24493 μs. In industrial accurate production scheduling, the improved DHP algorithm performs the best for textile industry’’s task scheduling with the best performance. Specifically, the task scheduling completion time for the textile industry is 4522 seconds when the number of automated guided vehicles (AGVs) is the minimum value of 3. In comparison, the completion time is 8541 seconds when the number of AGVs is the maximum value of 15. The textile industry shows the fastest task completion time compared to the other industry types for different settings of the number of AGVs. In addition, the analysis of the operation of the equipment engines shows that although there are high and low torques during operation, the Torque of most of the engines stays within the range of 29 Nm to 30.7 Nm, which is generally stable and has little impact on the performance of the equipment. The improved dynamic programming algorithm proposed in this paper has effective application potential in industrial extensive data production scheduling, which can improve the efficiency and responsiveness of production scheduling. It has important practical significance for industrial production management.

Evolutionary Dynamics Optimal Research-Oriented Tumor Immunity Architecture.

The article is devoted to evolutionary dynamics optimal control-oriented tumor immune differential game system. First, the mathematical model covering immune cells and tumor cells considering the effects of chemotherapy drugs and immune agents. Second, the bounded optimal control problem covering is transformed into solving Hamilton-Jacobi-Bellman (HJB) equation considering the actual constraints and infinite-horizon performance index based on minimizing the amount of medication administered. Finally, approximate optimal control strategy is acquired through iterative-dual heuristic dynamic programming (I-DHP) algorithm avoiding dimensional disaster effectively and providing optimal treatment scheme for clinical applications.

Adaptive Critic for Event-Triggered Unknown Nonlinear Optimal Tracking Design With Wastewater Treatment Applications.

In this article, an event-based near-optimal tracking control algorithm is developed for a class of nonaffine systems. First, in order to gain the tracking control strategy, the costate function is established through the iterative dual heuristic dynamic programming (DHP) algorithm. Then, the event-based control method is employed to improve the utilization efficiency of resources and ensure that the closed-loop system has an excellent control performance. Meanwhile, the input-to-state stability (ISS) is proven for the event-based tracking plant. In addition, three kinds of neural networks are used in the event-based DHP algorithm, which aims to identify the nonaffine nonlinear system, estimate the costate function, and approximate the tracking control law. Finally, a numerical experimental simulation is conducted to verify the effectiveness of the proposed scheme. Moreover, in order to further validate the feasibility, the algorithm is applied to the wastewater treatment plant to effectively control the concentrations of dissolved oxygen and nitrate nitrogen.

Neuro-Optimal Trajectory Tracking With Value Iteration of Discrete-Time Nonlinear Dynamics.

In this article, a novel neuro-optimal tracking control approach is developed toward discrete-time nonlinear systems. By constructing a new augmented plant, the optimal trajectory tracking design is transformed into an optimal regulation problem. For discrete-time nonlinear dynamics, the steady control input corresponding to the reference trajectory is given. Then, the value-iteration-based tracking control algorithm is provided and the convergence of the value function sequence is established. Therein, the approximation error between the iterative value function and the optimal cost is estimated. The uniformly ultimately bounded stability of the closed-loop system is also discussed in detail. Moreover, the iterative heuristic dynamic programming (HDP) algorithm is implemented by involving the critic and action components, where some new updating rules of the action network are provided. Finally, two examples are used to demonstrate the optimality of the present controller as well as the effectiveness of the proposed method.

Data-driven control for dynamic quantized nonlinear systems with state constraints based on barrier functions

In this study, the data-driven control problem is investigated for discrete-time nonlinear systems with state constraints and dynamic quantization. A dynamic quantizer is utilized to encode the control signal to relieve the impact of inherent quantization errors in comparison with traditional uniform quantizers. By employing three sets of neural networks (NNs), a data-based goal representation heuristic dynamic programming algorithm with the concurrent learning technique is offered to achieve the near-optimal control objective without model requirement. Specifically, by introducing the relaxed barrier function into the cost function, a barrier-based control strategy is proposed to generate the repulsive control force when the state moves to the constraint boundary. Subsequently, the recorded historical data is adequately leveraged to build a new weight updating rule, and persistent excitation of weight parameter estimation is hence removed. Furthermore, the Lyapunov method is utilized to demonstrate that the weights estimation errors of NNs are bounded. Finally, numerical simulation and power system examples are employed to demonstrate the validity of the control strategy.

A data-driven energy management method for parallel PHEVs based on action dependent heuristic dynamic programming (ADHDP) model

Due to the strong self-learning ability and adaptability of neural network, adaptive dynamic programming (ADP) method is regarded as an effective method to improve the vehicle economy in the case of uncertain driving conditions. With the look-ahead information, a data-driven energy management method is proposed based on action dependent heuristic dynamic programming (ADHDP) algorithm. Firstly, based on statistical analysis of DP behavior, a reference SOC trajectory is generated to limit the electricity consumption, which is adjusted with the dynamically updated driving information. Then, a data-driven energy management control method is developed for HEV architectures, including information acquisition module, shift scheduling module, energy distribution module. The gear shift command is designed by enumeration and the power distribution is performed by multiple ADHDP models, the core of which is to determine the utility function, action network (ANN), critic network (CNN) and training process. To ensure practical driveability, the restrictions on gear shifting and engine starting-stopping are taken as additional conditions in ADHDP optimizing process. Finally, two case studies under different driving scenarios are given. Simulation results demonstrate that the proposed method gains a good performance in both optimality approximation and adaptivity to uncertain driving conditions.

Hierarchical energy management strategy based on adaptive dynamic programming for hybrid electric vehicles in car-following scenarios

In order to improve the driving safety and fuel economy of hybrid electric vehicles (HEVs) in car-following scenarios, this paper proposes a hierarchical energy management strategy (EMS) based on adaptive dynamic programming (ADP). In the upper-layer speed planning, the speed prediction model of the preceding vehicle is established utilizing a back-propagation neural network (BPNN), and the host vehicle speed is planned based on heuristic dynamic programming (HDP) real-time through the predicted preceding vehicle speed and the current state of the host vehicle. In the lower-layer energy management control, the EMS based on dual heuristic dynamic programming (DHP) is used to realize the power allocation of each power source according to the planned speed to reduce fuel consumption. In both HDP and DHP algorithms, BPNN is used to implement the critic network (CN) and action network (AN) to obtain a more accurate control variable and faster convergence speed. By comparing the results with various existing EMSs in the different driving cycles, the simulation results show that the proposed EMS ensures real-time planning speed and good following performance while reducing energy consumption.

Adaptive learning control for triggered switched systems based on unknown direction control gain function

This paper investigates event-triggered switched heuristic dynamic programming (HDP) control for switched systems based on an unknown direction control gain function (UDCGF). First, the Nussbaum gain function is introduced to describe the unknown control direction, which makes the control behaviour of switched systems more flexible. The proposed variable threshold event-triggering (VTET) condition can significantly reduce computation while maintaining a certain system performance. Then, a detailed Lyapunov stability analysis based on the new tight bound is given, which reflects the asymptotic stability of the switched systems under the proposed switched HDP control method. The system mode-matched action-critic neural networks are employed to approximate the optimal triggered control law and the optimal switching value function, respectively. Subsequently, the boundedness analysis of neural network (NN) weights and the convergence analysis of HDP algorithm are given. At last, a numerical example is provided to verify the effectiveness of the proposed method.

Optimal Leader-Following Consensus Control of Multi-Agent Systems: A Neural Network Based Graphical Game Approach

In this article, the optimal leader-following consensus control problem of multi-agent systems is solved using a novel neural network-based (NN-based) integrated heuristic dynamic programming (IHDP) algorithm under a distributed learning framework. The optimal controller is designed to find a Nash equilibrium of the multi-agent differential graphical game. According to the principle of optimality, this problem can be transformed into solving a set of coupled Hamilton-Jacobi-Bellman (HJB) equations, which are difficult to solve directly because of the nonlinearity and strong coupling of the equations. The proposed algorithm combines policy iteration (PI) and value iteration (VI), and the Nash equilibrium of performance index among agents is achieved distributedly, i.e., all agents update their control inputs only use local information. The convergence of the algorithm is rigorously proved. Neural network (NN) approximation implementation is given for the unknown structures of local value functions of agents. The effectiveness of the algorithm is verified by numerical simulation.

Novel optimal trajectory tracking for nonlinear affine systems with an advanced critic learning structure

In this paper, a critic learning structure based on the novel utility function is developed to solve the optimal tracking control problem with the discount factor of affine nonlinear systems. The utility function is defined as the quadratic form of the error at the next moment, which can not only avoid solving the stable control input, but also effectively eliminate the tracking error. Next, the theoretical derivation of the method under value iteration is given in detail with convergence and stability analysis. Then, the dual heuristic dynamic programming (DHP) algorithm via a single neural network is introduced to reduce the amount of computation. The polynomial is used to approximate the costate function during the DHP implementation. The weighted residual method is used to update the weight matrix. During simulation, the convergence speed of the given strategy is compared with the heuristic dynamic programming (HDP) algorithm. The experiment results display that the convergence speed of the proposed method is faster than the HDP algorithm. Besides, the proposed method is compared with the traditional tracking control approach to verify its tracking performance. The experiment results show that the proposed method can avoid solving the stable control input, and the tracking error is closer to zero than the traditional strategy.

Bio-inspired heuristic dynamic programming for high-precision real-time flow control in a multi-tributary river system

Rivers are a type of time-varying, highly nonlinear systems, especially with multiple tributaries. To address low precision and poor real-time flow control of multi-tributary river systems, a heuristic dynamic programming algorithm inspired by regulation mechanisms in the human body (Bio-HDP) is proposed. Bio-HDP consists of several control units including the model network (MNCU), the action network (ANCU), the critic network (CNCU), and the central coordination (CCCU), all of which are developed based on the interactions and mechanisms in human’s NEI system (nervous, endocrine, and immune systems). The MNCU approximates the actual system through learning actual data on the control object. The ANCU calculates the control variable according to the error between the current state of the system acquired from the MNCU and the set value. The CNCU is used to evaluate whether the current control variable obtained by the ANCU can meet the control requirements. Finally, the CCCU conducts coordinated regulation of the above three control units in real-time according to the current state of the system, and makes the control mechanism faster and more accurate. By incorporating a biological feedback mechanism into the MNCU, the ANCU, and the CNCU, we adopt a neural network with multi-level feedback for the control mechanism. We select the Chongyang River system in China as the control object to verify the effects of the Bio-HDP control algorithm. The results show that the Bio-HDP algorithm improves speed, adaptability, and accuracy of the algorithm in comparison to other algorithms, and exhibits timely and accurate control effects.

Torsional oscillation-considered mode transition coordinated control for a power-split PHEV based on action dependent heuristic dynamic programming

The mode transition process (MTP) from electric mode to hybrid electric mode (EM-to-HM) will cause the deterioration in occupant comfort of PHEV, to tickle this issue, a torsional oscillation-considered mode transition coordinated control strategy and a novel general evaluation index for MTP are developed in this research, the quality of mode transition and transient torsional oscillation of gears (TTOGs) during MTP are taken into consideration comprehensively. An action dependent heuristic dynamic programming algorithm which takes the vehicle jerk, friction loss and TTOGs as evaluation index is used to optimize the pressure curve of clutch oil and the compensation torque of motor in the entire EM-to-HM process. Finally, the simulation results and hardware-in-the-loop tests show that vehicle jerk and TTOGs are suppressed, and the driving comfort can be improved accordingly.

Intelligent optimal tracking with asymmetric constraints of a nonlinear wastewater treatment system

AbstractThe wastewater treatment is an effective method for alleviating the shortage of water resources. In this article, a data‐driven iterative adaptive tracking control approach is developed to improve the control performance of the dissolved oxygen concentration and the nitrate nitrogen concentration in the nonlinear wastewater treatment plant. First, the model network is established to obtain the steady control and evaluate the new system state. Then, a nonquadratic performance functional is provided to handle asymmetric control constraints. Moreover, the new costate function and the tracking control policy are derived by using the dual heuristic dynamic programming algorithm. In the present control scheme, two neural networks are constructed to approximate the costate function and the tracking control law. Finally, the feasibility of the proposed algorithm is confirmed by applying the designed strategy to the wastewater treatment plant.

Neural optimal tracking control of constrained nonaffine systems with a wastewater treatment application

In this paper, we aim to solve the optimal tracking control problem for a class of nonaffine discrete-time systems with actuator saturation. First, a data-based neural identifier is constructed to learn the unknown system dynamics. Then, according to the expression of the trained neural identifier, we can obtain the steady control corresponding to the reference trajectory. Next, by involving the iterative dual heuristic dynamic programming algorithm, the new costate function and the tracking control law are developed. Two other neural networks are used to estimate the costate function and approximate the tracking control law. Considering approximation errors of neural networks, the stability analysis of the proposed algorithm for the specific systems is provided by introducing the Lyapunov approach. Finally, via conducting simulation and comparison, the superiority of the developed optimal tracking method is confirmed. Moreover, the trajectory tracking performance of the wastewater treatment application is also involved for further verifying the proposed approach.

Event-triggered design for discrete-time nonlinear systems with control constraints

In order to solve the constrained-input problem and reduce the computing resources, a novel event-triggered optimal control method is proposed for a class of discrete-time nonlinear systems. In the proposed method, the event-triggered control policy is applied to the globalized dual heuristic dynamic programming (GDHP) algorithm. Compared with the traditional adaptive dynamic programming (ADP) control, the event-triggered GDHP control can reduce the computation while ensuring the system performance. In this paper, a non-quadratic function is given to code the control constraints and the trigger condition with the stability analysis is provided. Neural networks (NN) are constructed in the GDHP structure, where the model network is designed to identify the unknown nonlinear system, the critic network is used to learn the cost function and its partial derivative, and the action network is designed to obtain the approximate optimal control law. Three simulation examples are presented to demonstrate the performance of the proposed event-triggered design for constrained discrete-time nonlinear systems.

Data-Based Nonaffine Optimal Tracking Control Using Iterative DHP Approach

Abstract In this paper, a data-based optimal tracking control approach is developed by involving the iterative dual heuristic dynamic programming algorithm for nonaffine systems. In order to gain the steady control corresponding to the desired trajectory, a novel strategy is established with regard to the unknown system function. Then, according to the iterative adaptive dynamic programming algorithm, the updating formula of the costate function and the new optimal control policy for unknown nonaffine systems are provided to solve the optimal tracking control problem. Moreover, three neural networks are used to facilitate the implementation of the proposed algorithm. In order to improve the accuracy of the steady control corresponding to the desired trajectory, we employ a model network to directly approximate the unknown system function instead of the error dynamics. Finally, the effectiveness of the proposed method is demonstrated through a simulation example.

Protocols for Creating and Distilling Multipartite GHZ States With Bell Pairs

The distribution of high-quality Greenberger-Horne-Zeilinger (GHZ) states is at the heart of many quantum communication tasks, ranging from extending the baseline of telescopes to secret sharing. They also play an important role in error-correction architectures for distributed quantum computation, where Bell pairs can be leveraged to create an entangled network of quantum computers. We investigate the creation and distillation of GHZ states out of non-perfect Bell pairs over quantum networks. In particular, we introduce a heuristic dynamic programming algorithm to optimize over a large class of protocols that create and purify GHZ states. All protocols considered use a common framework based on measurements of non-local stabilizer operators of the target state (i.e., the GHZ state), where each non-local measurement consumes another (non-perfect) entangled state as a resource. The new protocols outperform previous proposals for scenarios without decoherence and local gate noise. Furthermore, the algorithms can be applied for finding protocols for any number of parties and any number of entangled pairs involved.

Convergence and numerical stability of action-dependent heuristic dynamic programming algorithms based on RLS learning for online DLQR optimal control

The development and the numerical stability analysis of a novel algorithm of approximate dynamic programming (ADP) based on RLS learning for approximating the optimal control solution online in real-time are the main issues of this paper. The approximate dynamic programming is a method developed to make possible the use of dynamic programming techniques in real-time, but this method has a reasonable mathematical complexity due to the size of the internal matrices of the algorithm and the need for inversion of some of them. Thus, focusing on improving numerical stability and computational cost of ADP algorithms, more specifically in the action-dependent heuristic dynamic programming and optimal control context, UDUT-type unitary transformations are integrated in actor-critic architectures, which produce algorithms with better specifications for implementation in real-world optimal control systems. The control and stabilisation of the inverted pendulum system on a motor driven cart is established as a study platform to evaluate the convergence and numerical stability for the estimated parameters of the proposed algorithm.

Convergence and numerical stability of action-dependent heuristic dynamic programming algorithms based on RLS learning for online DLQR optimal control

The development and the numerical stability analysis of a novel algorithm of approximate dynamic programming (ADP) based on RLS learning for approximating the optimal control solution online in real-time are the main issues of this paper. The approximate dynamic programming is a method developed to make possible the use of dynamic programming techniques in real-time, but this method has a reasonable mathematical complexity due to the size of the internal matrices of the algorithm and the need for inversion of some of them. Thus, focusing on improving numerical stability and computational cost of ADP algorithms, more specifically in the action-dependent heuristic dynamic programming and optimal control context, UDUT-type unitary transformations are integrated in actor-critic architectures, which produce algorithms with better specifications for implementation in real-world optimal control systems. The control and stabilisation of the inverted pendulum system on a motor driven cart is established as a study platform to evaluate the convergence and numerical stability for the estimated parameters of the proposed algorithm.