Actor-critic Neural Network Research Articles

Policy iteration and coupled Riccati solutions for dynamic graphical games

A novel online adaptive learning technique is developed to solve the dynamic graphical games in real-time. The players or agents exchange the information on a communication graph. Hamiltonian mechanics are used to derive the constrained minimum conditions for the graphical game. Novel coupled Riccati equations are developed for this type of games. Convergence of the adaptive learning technique is studied given the graph topology. Nash equilibrium solution for the graphical game is found by solving the underlying Hamilton-Jacobi-Bellman equations. Actor-Critic neural network structures are used to implement the adaptive learning solution using local information available to the players.

Policy iteration and coupled Riccati solutions for dynamic graphical games

A novel online adaptive learning technique is developed to solve the dynamic graphical games in real-time. The players or agents exchange the information on a communication graph. Hamiltonian mechanics are used to derive the constrained minimum conditions for the graphical game. Novel coupled Riccati equations are developed for this type of games. Convergence of the adaptive learning technique is studied given the graph topology. Nash equilibrium solution for the graphical game is found by solving the underlying Hamilton-Jacobi-Bellman equations. Actor-Critic neural network structures are used to implement the adaptive learning solution using local information available to the players.

Data‐driven optimal tracking control for a class of affine non‐linear continuous‐time systems with completely unknown dynamics

In this study, the optimal tracking control problem (OTCP) for affine non-linear continuous-time systems with completely unknown dynamics is addressed based on data by introducing the reinforcement learning (RL) technique. Unlike existing methods to the OTCP, the proposed data-driven policy iteration (PI) method does not need to have or identify any knowledge of the system dynamics, including both drift dynamics and input dynamics. To carry out the proposed method, the original OTCP is pre-processed to construct an augmented system composed of the error system dynamics and the desired trajectory dynamics. Then, based on the augmented system, a data-driven PI, which introduces discount factor to solve the OTCP, is implemented on an actor–critic neural network (NN) structure by only using system data rather than the exact knowledge of system dynamics. Two NNs are used in the structure to generate the optimal cost and optimal control policy, respectively, and the weights are updated by a least-square approach which minimises the residual errors. The proposed method is an off-policy RL method, where the data can be arbitrarily sampled on the state and input domain. Finally, simulation results are provided to show the effectiveness of the proposed method.

Reinforcement learning solution for HJB equation arising in constrained optimal control problem

The constrained optimal control problem depends on the solution of the complicated Hamilton–Jacobi–Bellman equation (HJBE). In this paper, a data-based off-policy reinforcement learning (RL) method is proposed, which learns the solution of the HJBE and the optimal control policy from real system data. One important feature of the off-policy RL is that its policy evaluation can be realized with data generated by other behavior policies, not necessarily the target policy, which solves the insufficient exploration problem. The convergence of the off-policy RL is proved by demonstrating its equivalence to the successive approximation approach. Its implementation procedure is based on the actor–critic neural networks structure, where the function approximation is conducted with linearly independent basis functions. Subsequently, the convergence of the implementation procedure with function approximation is also proved. Finally, its effectiveness is verified through computer simulations.

Reinforcement learning and neural networks for multi-agent nonzero-sum games of nonlinear constrained-input systems

This paper presents an online adaptive optimal control method based on reinforcement learning to solve the multi-agent nonzero-sum (NZS) differential games of nonlinear constrained-input continuous-time systems. A non-quadratic cost functional associated with each agent is employed to encode the saturation nonlinearity into the NZS game. The algorithm is implemented as a separate actor-critic neural network (NN) structure for every participant in the game, where adaptation of both NNs is performed simultaneously and continuously. The technique of concurrent learning is utilized to obtain novel update laws for the critic NN weights. That is, recorded data and current data are used concurrently for adaptation of the critic NN weights. This results in an algorithm where an easier and verifiable condition is sufficient for parameter convergence rather than the restrictive persistence of excitation (PE) condition. The stability of the closed-loop systems is guaranteed and the convergence to the Nash equilibrium solution of the game is shown. Simulation results show the effectiveness of the proposed method.

Reinforcement learning output feedback NN control using deterministic learning technique.

In this brief, a novel adaptive-critic-based neural network (NN) controller is investigated for nonlinear pure-feedback systems. The controller design is based on the transformed predictor form, and the actor-critic NN control architecture includes two NNs, whereas the critic NN is used to approximate the strategic utility function, and the action NN is employed to minimize both the strategic utility function and the tracking error. A deterministic learning technique has been employed to guarantee that the partial persistent excitation condition of internal states is satisfied during tracking control to a periodic reference orbit. The uniformly ultimate boundedness of closed-loop signals is shown via Lyapunov stability analysis. Simulation results are presented to demonstrate the effectiveness of the proposed control.

APPLICATION OF AN EVOLUTIONARY ACTOR–CRITIC REINFORCEMENT LEARNING METHOD FOR THE CONTROL OF A THREE-LINK MUSCULOSKELETAL ARM DURING A REACHING MOVEMENT

In this paper, the control of a planar three-link musculoskeletal arm by using a revolutionary actor–critic reinforcement learning (RL) method during a reaching movement to a stationary target is presented. The arm model used in this study included three skeletal links (wrist, forearm, and upper arm), three joints (wrist, elbow, and shoulder without redundancy), and six non-linear monoarticular muscles (with redundancy), which were based on the Hill model. The learning control system was composed of actor, critic, and genetic algorithm (GA) parts. Two single-layer neural networks were used for each part of the actor and critic. This learning control system was used to apply six activation commands to six monoarticular muscles at each instant of time. It also used a reinforcement (reward) feedback for the learning process and controlling the direction of arm movement. Also, the GA was implemented to select the best learning rates for actor–critic neural networks. The results showed that mean square error (MSE) and average episode time gradually decrease and average reward gradually increases to constant values during the learning of the control policy. Furthermore, when learning was complete, optimal values of learning rates were selected.

A two-layer networked learning control system using actor–critic neural network

Aiming at the fact that the controlled objects are becoming more and more complex, while the existing control strategies cannot fulfill the needs for higher quality control performance due to the limitations of the widely used single-layer networked control system architecture, a two-layer networked learning control system architecture is proposed. Under this architecture, independent local controllers are interconnected to form the lower layer, while a learning agent which communicates with the independent local controllers in the lower layer forms the upper layer. To implement such a system, a discard-packet strategy is firstly developed to deal with network-induced delay, data packet out-of-order and data packet loss. The cubic spline interpolator is then employed to compensate the lost data. Finally, the output of the learning agent using actor–critic neural network is used to dynamically tune the control signal of local controller. Control simulations of different ways for a nonlinear heating, ventilation and air-conditioning (HVAC) system are compared. Simulation results show that this new architecture can effectively improve the control performance.

SDI applications of neural network technology

The constrained optimal control problem depends on the solution of the complicated Hamilton–Jacobi–Bellman equation (HJBE). In this paper, a data-based off-policy reinforcement learning (RL) method is proposed, which learns the solution of the HJBE and the optimal control policy from real system data. One important feature of the off-policy RL is that its policy evaluation can be realized with data generated by other behavior policies, not necessarily the target policy, which solves the insufficient exploration problem. The convergence of the off-policy RL is proved by demonstrating its equivalence to the successive approximation approach. Its implementation procedure is based on the actor–critic neural networks structure, where the function approximation is conducted with linearly independent basis functions. Subsequently, the convergence of the implementation procedure with function approximation is also proved. Finally, its effectiveness is verified through computer simulations.