Actor-critic Reinforcement Learning Framework Research Articles

Optimal formation tracking control based on reinforcement learning for multi-UAV systems

This paper investigates the problem of optimal formation tracking control for multi-UAV systems with model uncertainty and external disturbances. Firstly, by combining the sliding mode method and a neural network, an adaptive sliding mode controller is derived that counteracts the effects of modeling uncertainty and external disturbance. Subsequently, the optimal formation tracking control problem of the original system is then converted to the optimal control problem of a nominal system, and an actor–critic reinforcement learning framework is built using adaptive neural network identifiers to recursively approximate the total optimal policy and cost function. The Lyapunov analysis method shows that the stability of the closed-loop system and the convergence of the estimation weights for the actor–critic network are guaranteed. Additionally, a formation tracking using virtual experiment platform for multi-UAV systems are constructed based on the Robot Operating System (ROS) and Gazebo simulator. Finally, virtual-reality experiments is performed to demonstrate the effectiveness of the proposed control scheme.

A deep actor critic reinforcement learning framework for learning to rank

In this paper, we propose a Deep Reinforcement learning based approach for Learning to rank task. Reinforcement Learning has been applied in the ranking task with good success, but the existing Policy Gradient based approaches suffer from noisy gradients and high variance, resulting in unstable learning. The natural policy gradient methods like REINFORCE perform Monte Carlo sampling, thus taking samples randomly, which leads to high variance. As the action space becomes large, i.e., with a very large number of documents, traditional RL techniques lack the complex model required in the scenario to deal with a large number of items. We propose a Deep Reinforcement learning based approach for learning to rank task in this paper to address these issues. By combining Deep learning with the Reinforcement Learning framework, our approach can learn a complex function as deep neural networks can provide significant function approximation. We used Actor-Critic framework where the critic network can reduce variance by utilizing techniques such as clipped delayed policy updates, clipped double q learning, etc. Also, due to the enormous space of the web, the most relevant results are needed to be returned for the corresponding query from within a large action space. Policy gradient algorithms have been effectively applied to problems in large action spaces(items) with deep neural networks as they don’t rely on finding value for each action(item) as in value-based methods. Further, we use an actor-network with a CNN layer in the ranking process to capture the sequential patterns among the documents. We utilize the TD3 method to train our Reinforcement Learning agent with a listwise loss function, which performs delayed policy updates resulting in value estimates with lower variance. To the best of our knowledge, this is the first Deep reinforcement learning method applied in Learning to Rank for document retrieval. We performed experiments on the various Letor datasets and showed that our method outperforms various state-of-the-art baselines.

FLoadNet: Load Balancing in Fog Networks With Cooperative Multiagent Using Actor–Critic Method

The growing demands of the Internet of Things (IoT) require a platform that supports real-time interactions and high availability of services to devices. In this context, the fog computing paradigm has emerged as an attractive solution for processing the data of IoT applications. Owing to the unpredictable traffic demands and resource heterogeneity in the fog environment, a smart workload distribution is essential to achieve high resource utilization and computing efficiency. To this end, this paper considers a joint link and server load balancing problem with multiple cooperative access points, (APs), in a combined edge-fog-cloud environment. The joint optimization problem is formulated as a stochastic game, and an actor-critic reinforcement learning framework, called FLoadNet, is proposed to optimize the joint policy of the multi-agents. FLoadNet consists of a centralized critic network, with parameter sharing and distributed individual actor networks in all the APs. Due to the learning dynamics and partially observable environment, we propose an extended critic network model, where cooperative APs learn to communicate among themselves while evaluating the value function. Unlike previous studies, the proposed critic network is designed to train both value and message functions, which is shown to significantly reduce the computational cost. The main goal of this work is to advance the development of efficient edge learning and the application of distributed learning algorithms specifically to fog network load balancing. The experimental results show that FLoadNet outperforms baseline load balancing methods.

Adaptive optimal process control with actor-critic design for energy-efficient batch machining subject to time-varying tool wear

Batch machining systems are essential for improving productivity and quality, but they consume considerable amounts of energy due to the continuous interaction with machine tools, workpieces, and cutting tools. In contrast to single-piece machining that has a short production cycle, the tool wear impacts in batch machining systems on energy consumption cannot be underestimated. However, few studies have focused on adaptive process control subject to time-varying tool wear because process optimization has always been previously considered a static problem. As an alternative to metaheuristic algorithms, reinforcement learning (RL) offers an attractive means for solving such a dynamic, high-dimensional, and high-coupling problem. In the case of turning cylindrical parts, an energy-efficient decision model is developed for the process control of pass operations of batch machining. The decision variables are decoupled by reformulating the problem as the Markov decision process, wherein the tool wear experiences dynamic changes. To solve the problem, an actor-critic RL framework with multi-constraint and multi-objective design is developed. Based on the framework, a dynamic process control method is proposed where the RL agent observes workpiece features, machining requirements, and tool wear states (inputs) and adaptively selects the control parameters such as cutting speed, feed rate, and cutting rate (outputs), with the aim to conserve energy. Two application tests and comparisons against metaheuristic methods are performed. The results indicate that the method can further reduce energy by over 20% compared with energy-efficient optimization ignoring tool wear effects. The learning efficiency of RL is about three times faster than that of metaheuristics. The online sampling time is less than 0.1 millisecond, which facilitates real-time control of process parameters.

Target Tracking Control of a Biomimetic Underwater Vehicle Through Deep Reinforcement Learning.

In this article, the underwater target tracking control problem of a biomimetic underwater vehicle (BUV) is addressed. Since it is difficult to build an effective mathematic model of a BUV due to the uncertainty of hydrodynamics, target tracking control is converted into the Markov decision process and is further achieved via deep reinforcement learning. The system state and reward function of underwater target tracking control are described. Based on the actor-critic reinforcement learning framework, the deep deterministic policy gradient actor-critic algorithm with supervision controller is proposed. The training tricks, including prioritized experience replay, actor network indirect supervision training, target network updating with different periods, and expansion of exploration space by applying random noise, are presented. Indirect supervision training is designed to address the issues of low stability and slow convergence of reinforcement learning in the continuous state and action space. Comparative simulations are performed to show the effectiveness of the training tricks. Finally, the proposed actor-critic reinforcement learning algorithm with supervision controller is applied to the physical BUV. Swimming pool experiments of underwater object tracking of the BUV are conducted in multiple scenarios to verify the effectiveness and robustness of the proposed method.

Reinforcement-Learning-Based Tracking Control with Fixed-Time Prescribed Performance for Reusable Launch Vehicle under Input Constraints

This paper proposes a novel reinforcement learning (RL)-based tracking control scheme with fixed-time prescribed performance for a reusable launch vehicle subject to parametric uncertainties, external disturbances, and input constraints. First, a fixed-time prescribed performance function is employed to restrain attitude tracking errors, and an equivalent unconstrained system is derived via an error transformation technique. Then, a hyperbolic tangent function is incorporated into the optimal performance index of the unconstrained system to tackle the input constraints. Subsequently, an actor-critic RL framework with super-twisting-like sliding mode control is constructed to establish a practical solution for the optimal control problem. Benefiting from the proposed scheme, the robustness of the RL-based controller against unknown dynamics is enhanced, and the control performance can be qualitatively prearranged by users. Theoretical analysis shows that the attitude tracking errors converge to a preset region within a preassigned fixed time, and the weight estimation errors of the actor-critic networks are uniformly ultimately bounded. Finally, comparative numerical simulation results are provided to illustrate the effectiveness and improved performance of the proposed control scheme.

Reinforcement Learning-Based Optimal Tracking Control of an Unknown Unmanned Surface Vehicle.

In this article, a novel reinforcement learning-based optimal tracking control (RLOTC) scheme is established for an unmanned surface vehicle (USV) in the presence of complex unknowns, including dead-zone input nonlinearities, system dynamics, and disturbances. To be specific, dead-zone nonlinearities are decoupled to be input-dependent sloped controls and unknown biases that are encapsulated into lumped unknowns within tracking error dynamics. Neural network (NN) approximators are further deployed to adaptively identify complex unknowns and facilitate a Hamilton-Jacobi-Bellman (HJB) equation that formulates optimal tracking. In order to derive a practically optimal solution, an actor-critic reinforcement learning framework is built by employing adaptive NN identifiers to recursively approximate the total optimal policy and cost function. Eventually, theoretical analysis shows that the entire RLOTC scheme can render tracking errors that converge to an arbitrarily small neighborhood of the origin, subject to optimal cost. Simulation results and comprehensive comparisons on a prototype USV demonstrate remarkable effectiveness and superiority.

Self-learning drift control of automated vehicles beyond handling limit after rear-end collision

Abstract Vehicles involved in traffic accidents generally experience divergent vehicle motion, which causes severe damage. This paper presents a self-learning drift-control method for the purpose of stabilizing a vehicle's yaw motions after a high-speed rear-end collision. The struck vehicle generally experiences substantial drifting and/or spinning after the collision, which is beyond the handling limit and difficult to control. Drift control of the struck vehicle along the original lane was investigated. The rear-end collision was treated as a set of impact forces, and the three-dimensional non-linear dynamic responses of the vehicle were considered in the drift control. A multi-layer perception neural network was trained as a deterministic control policy using the actor-critic reinforcement learning framework. The control policy was iteratively updated, initiating from a random parameterized policy. The results show that the self-learning controller gained the ability to eliminate unstable vehicle motion after data-driven training of about 60,000 iterations. The controlled struck vehicle was also able to drift back to its original lane in a variety of rear-end collision scenarios, which could significantly reduce the risk of a second collision in traffic.