Actor Critic Research Articles

A Parallel Framework of Adaptive Dynamic Programming Algorithm With Off-Policy Learning.

In this article, a model-free online adaptive dynamic programming (ADP) approach is developed for solving the optimal control problem of nonaffine nonlinear systems. Combining the off-policy learning mechanism with the parallel paradigm, multithread agents are employed to collect the transitions by interacting with the environment that significantly augments the number of sampled data. On the other hand, each thread agent explores the environment with different initial states under its own behavior policy that enhances the exploration capability and alleviates the correlation between the sampled data. After the policy evaluation process, only one step update is required for policy improvement based on the policy gradient method. The stability of the system under iterative control laws is guaranteed. Moreover, the convergence analysis is given to prove that the iterative Q-function is monotonically nonincreasing and finally converges to the solution of the Hamilton-Jacobi-Bellman (HJB) equation. For implementing the algorithm, the actor-critic (AC) structure is utilized with two neural networks (NNs) to approximate the Q-function and the control policy. Finally, the effectiveness of the proposed algorithm is verified by two numerical examples.

Missile-Target Situation Assessment Model Based on Reinforcement Learning

In situation assessment (SA) of missile versus target fighter, the traditional SA models generally have the characteristics of strong subjectivity and poor dynamic adaptability. This paper considers SA as an expectation of future returns and establishes a missile-target simulation battle model. The actor-critic (AC) algorithm in reinforcement learning (RL) is used to train the evaluation network, and a missile-target SA model is established in simulation battle training. Simulation and comparative experiments show that the model can effectively estimate the expected effect of missile attack under the current situation, and it provides an effective basis for missile attack decision.

Automatic Generation Control Based on Multiple Neural Networks With Actor-Critic Strategy.

As the conventional automatic generation control (AGC) is inadequate to deal with the strong random disturbance issues induced by the ever-increasing penetration of renewable energy to the power grids, this article proposes a deep-reinforcement-learning-based three-network double-delay actor-critic (TDAC) control strategy for AGC to handle the above problem, which is mainly developed by multiple neural networks to fit the system action strategies and evaluate the value. The proposed strategy can increase the exploration efficiency and the quality of AGC and improve the system control performance using the modified actor-critic (AC) method with incentive heuristic mechanism, while a novel iterative way of the value function is also used to reduce the bias of optimization effectively for achieving optimal coordinated control of the power grid. The simulations are provided in the work to show the control performance of the strategy. Compared with other smart methods, the simulation study demonstrates that TDAC has excellent exploratory stability and learning ability. Meanwhile, it also can improve the dynamic performance of the power system and achieve the regional optimal coordinated control.

Supervised-actor-critic reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units

BackgroundReinforcement learning (RL) provides a promising technique to solve complex sequential decision making problems in healthcare domains. Recent years have seen a great progress of applying RL in addressing decision-making problems in Intensive Care Units (ICUs). However, since the goal of traditional RL algorithms is to maximize a long-term reward function, exploration in the learning process may have a fatal impact on the patient. As such, a short-term goal should also be considered to keep the patient stable during the treating process.MethodsWe use a Supervised-Actor-Critic (SAC) RL algorithm to address this problem by combining the long-term goal-oriented characteristics of RL with the short-term goal of supervised learning. We evaluate the differences between SAC and traditional Actor-Critic (AC) algorithms in addressing the decision making problems of ventilation and sedative dosing in ICUs.ResultsResults show that SAC is much more efficient than the traditional AC algorithm in terms of convergence rate and data utilization.ConclusionsThe SAC algorithm not only aims to cure patients in the long term, but also reduces the degree of deviation from the strategy applied by clinical doctors and thus improves the therapeutic effect.

Fixed-Wing UAVs flocking in continuous spaces: A deep reinforcement learning approach

Fixed-Wing UAVs (Unmanned Aerial Vehicles) flocking is still a challenging problem due to the kinematics complexity and environmental dynamics. In this paper, we solve the leader–followers flocking problem using a novel deep reinforcement learning algorithm that can generate roll angle and velocity commands by training an end-to-end controller in continuous state and action spaces. Specifically, we choose CACLA (Continuous Actor–Critic Learning Automation) as the base algorithm and we use the multi-layer perceptron to represent both the actor and the critic. Besides, we further improve the learning efficiency by using the experience replay technique that stores the training data in the experience memory and samples from the memory as needed. We have compared the performance of the proposed CACER (Continuous Actor–Critic with Experience Replay) algorithm with benchmark algorithms such as DDPG and double DQN in numerical simulation, and we have demonstrated the performance of the learned optimal policy in semi-physical simulation without any parameter tuning.

A Decision-Making Strategy for Vehicle Autonomous Braking in Emergency via Deep Reinforcement Learning

Autonomous braking through vehicle precise decision-making and control to reduce accidents is a key issue, especially in the early diffusion phase of autonomous vehicle development. This paper proposes a deep reinforcement learning (DRL)-based autonomous braking decision-making strategy in an emergency situation. Three key influencing factors, including efficiency, accuracy and passengers' comfort, are fully considered and satisfied by the proposed strategy. First, the vehicle lane-changing process and the braking process are analyzed in detail, which include the critical factors in the design of the autonomous braking strategy. Second, we propose a DRL process that determines the optimal strategy for autonomous braking. Particularly, a multi-objective reward function is designed, which can compromise the rewards achieved of different brake moments, the degree of the accident, and the comfort of the passenger. Third, a typical actor-critic (AC) algorithm named deep deterministic policy gradient (DDPG) is adopted for solving the autonomous braking problem, which can improve the efficiency of the optimal strategy and be stable in continuous control tasks. Once the strategy is well trained, the vehicle can automatically take optimal braking behavior in an emergency to improve driving safety. Extensive simulations validate the effectiveness and efficiency of our proposal in terms of learning effectiveness, decision-making accuracy and driving safety.

Actor Critic Deep Reinforcement Learning for Neural Malware Control

In addition to using signatures, antimalware products also detect malicious attacks by evaluating unknown files in an emulated environment, i.e. sandbox, prior to execution on a computer's native operating system. During emulation, a file cannot be scanned indefinitely, and antimalware engines often set the number of instructions to be executed based on a set of heuristics. These heuristics only make the decision of when to halt emulation using partial information leading to the execution of the file for either too many or too few instructions. Also this method is vulnerable if the attackers learn this set of heuristics.
 Recent research uses a deep reinforcement learning (DRL) model employing a Deep Q-Network (DQN) to learn when to halt the emulation of a file. In this paper, we propose a new DRL-based system which instead employs a modified actor critic (AC) framework for the emulation halting task. This AC model dynamically predicts the best time to halt the file's execution based on a sequence of system API calls. Compared to the earlier models, the new model is capable of handling adversarial attacks by simulating their behaviors using the critic model. The new AC model demonstrates much better performance than both the DQN model and antimalware engine's heuristics. In terms of execution speed (evaluated by the halting decision), the new model halts the execution of unknown files by up to 2.5% earlier than the DQN model and 93.6% earlier than the heuristics. For the task of detecting malicious files, the proposed AC model increases the true positive rate by 9.9% from 69.5% to 76.4% at a false positive rate of 1% compared to the DQN model, and by 83.4% from 41.2% to 76.4% at a false positive rate of 1% compared to a recently proposed LSTM model.

Local motion simulation using deep reinforcement learning

AbstractTraditional local motion simulation focuses largely on avoiding collisions in the next frame. However, due to its lack of forward looking, the global trajectory of agents usually seems unreasonable. As a method of optimizing the overall reward, deep reinforcement learning (DRL) can better correct the problems that exist in the traditional local motion simulation method. In this article, we propose a local motion simulation method integrating optimal reciprocal collision avoidance (ORCA) and DRL, referred to as ORCA‐DRL. The main idea of ORCA‐DRL is to perform local collision avoidance detection via ORCA and smooth the trajectory at the same time via DRL. We use a deep neural network (DNN) as the state‐to‐action mapping function, where the state information is detected by virtual visual sensors and the action space includes two continuous spaces: speed and direction. To improve data utilization and speed up the training process, we use the proximal policy optimization based on the actor–critic (AC) framework to update the DNN parameters. Three scenes (circle, hallway, and crossing) are designed to evaluate the performance of ORCA‐DRL. The results reveal that, compared with the ORCA, our proposed ORCA‐DRL method can: (a) reduce the total number of frames, leading to less time for agents to reach their destination; and (b) effectively avoid local optima, evidenced by smoothed global trajectories.

Robust control under worst‐case uncertainty for unknown nonlinear systems using modified reinforcement learning

SummaryReinforcement learning (RL) is an effective method for the design of robust controllers of unknown nonlinear systems. Normal RLs for robust control, such as actor‐critic (AC) algorithms, depend on the estimation accuracy. Uncertainty in the worst case requires a large state‐action space, this causes overestimation and computational problems. In this article, the RL method is modified with the k‐nearest neighbor and the double Q‐learning algorithm. The modified RL does not need the neural estimator as AC and can stabilize the unknown nonlinear system under the worst‐case uncertainty. The convergence property of the proposed RL method is analyzed. The simulations and the experimental results show that our modified RLs are much more robust compared with the classic controllers, such as the proportional‐integral‐derivative, the sliding mode, and the optimal linear quadratic regulator controllers.

An actor-critic reinforcement learning-based resource management in mobile edge computing systems

Reinforcement learning (RL) as an effective tool has attracted great attention in wireless communication field nowadays. In this paper, we investigate the offloading decision and resource allocation problem in mobile edge computing (MEC) systems based on RL methods. Different from existing literature, our research focuses on improving mobile operators’ revenue by maximizing the amount of the offloaded tasks while decreasing the energy expenditure and time-delays. Considering the dynamic characteristics of wireless environment, the above problem is modeled as a Markov decision process (MDP). Since the action space of the MDP is multidimensional continuous variables mixed with discrete variables, traditional RL algorithms are powerless. Therefore, an actor-critic (AC) with eligibility traces algorithm is proposed to resolve the problem. The actor part introduces the parameterized normal distribution to generate the probabilities of continuous stochastic actions, and the critic part employs a linear approximator to estimate the value of states, based on which the actor part updates policy parameters in the direction of performance improvement. Furthermore, an advantage function is designed to reduce the variance of the learning process. Simulation results indicate that the proposed algorithm can find the best strategy to maximize the amount of the tasks executed by the MEC server while decreasing the energy consumption and time-delays.

Capsule Network Performance with Autonomous Navigation

Capsule Networks (CapsNets) have been proposed as an alternative to Convolutional Neural Networks (CNNs). This paper showcases how CapsNets are more capable than CNNs for autonomous agent exploration of realistic scenarios. In real world navigation, rewards external to agents may be rare. In turn, reinforcement learning algorithms can struggle to form meaningful policy functions. This paper's approach Capsules Exploration Module (Caps-EM) pairs a CapsNets architecture with an Advantage Actor Critic algorithm. Other approaches for navigating sparse environments require intrinsic reward generators, such as the Intrinsic Curiosity Module (ICM) and Augmented Curiosity Modules (ACM). Caps-EM uses a more compact architecture without need for intrinsic rewards. Tested using ViZDoom, the Caps-EM uses 44% and 83% fewer trainable network parameters than the ICM and Depth-Augmented Curiosity Module (D-ACM), respectively, for 1141% and 437% average time improvement over the ICM and D-ACM, respectively, for converging to a policy function across "My Way Home" scenarios.

Complete coverage path planning using reinforcement learning for Tetromino based cleaning and maintenance robot

Tiling robotics have been deployed in autonomous complete area coverage tasks such as floor cleaning, building inspection, and maintenance, surface painting. One class of tiling robotics, polyomino-based reconfigurable robots, overcome the limitation of fixed-form robots in achieving high-efficiency area coverage by adopting different morphologies to suit the needs of the current environment. Since the reconfigurable actions of these robots are produced by real-time intelligent decisions during operations, an optimal path planning algorithm is paramount to maximize the area coverage while minimizing the energy consumed by these robots. This paper proposes a complete coverage path planning (CCPP) model trained using deep blackreinforcement learning (RL) for the tetromino based reconfigurable robot platform called hTetro to simultaneously generate the optimal set of shapes for any pretrained arbitrary environment shape with a trajectory that has the least overall cost. To this end, a Convolutional Neural Network (CNN) with Long Short Term Memory (LSTM) layers is trained using Actor Critic Experience Replay (ACER) reinforcement learning algorithm. The results are compared with existing approaches which are based on the traditional tiling theory model, including zigzag, spiral, and greedy search schemes. The model is also compared with the Travelling salesman problem (TSP) based Genetic Algorithm (GA) and Ant Colony Optimization (ACO) schemes. The proposed scheme generates a path with lower cost while also requiring lesser time to generate it. The model is also highly robust and can generate a path in any pretrained arbitrary environments.

Interconnection and damping assignment control based on modified actor–critic algorithm with wavelet function approximation

Passivity-based control (PBC) is model-based, and as a result, it is affected by uncertainties. Besides, for some PBC designs, partial differential equations (PDEs) should be solved in order to obtain the parameters of the controller. Actor–critic (AC) algorithm has been used to regulate PBC parameters instantaneously and solve PDEs online. Despite the benefits of this algorithm, it cannot handle disturbance. The main purpose of this paper is to modify the AC algorithm with an online wavelet function approximation to improve its performance. The proposed method uses PBC and the modified AC algorithm to attenuate the effects of uncertainties. Moreover, the stability analysis is investigated for a nonlinear system. The results show that the new method can handle disturbance and uncertainties more proper than other methods.

Federated Cooperation and Augmentation for Power Allocation in Decentralized Wireless Networks

Emerging mobile edge techniques and applications such as Augmented Reality (AR)/Virtual Reality (VR), Internet of Things (IoT), and vehicle networking, result in an explosive growth of power and computing resource consumptions. In the meantime, the volume of data generated at the edge networks is also increasing rapidly. Under this circumstance, building energy-efficient and privacy-protected communications is imperative for 5G and beyond wireless communication systems. The recent emerging distributed learning methods such as federated learning (FL) perform well in improving resource efficiency while protecting user privacy with low communication overhead. Specifically, FL enables edge devices to learn a shared network model by aggregating local updates while keeping all the training processes on local devices. This paper investigates distributed power allocation for edge users in decentralized wireless networks with aim to maximize energy/spectrum efficiency while preventing privacy leakage based on a FL framework. Due to the dynamics and complexity of wireless networks, we adopt an on-line Actor-Critic (AC) architecture as the local training model, and FL performs cooperation for edge users by sharing the gradients and weightages generated in the Actor network. Moreover, in order to resolve the over-fitting problem caused by data leakages in Non-independent and identically distributed (Non-i.i.d) data environment, we propose a federated augmentation mechanism with Wasserstein Generative Adversarial Networks (WGANs) algorithm for data augmentation. Federated augmentation empowers each device to replenish the data buffer using a generative model of WGANs until accomplishing an i.i.d training dataset, which significantly reduces the communication overhead in distributed learning compared to direct data sample exchange method. Numerical results reveal that the proposed federated learning based cooperation and augmentation (FL-CA) algorithm possesses a good convergence property, high robustness and achieves better accuracy of power allocation strategy than other three benchmark algorithms.

CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION

Capsule Networks (CapsNets) have been proposed as an alternative to Convolutional Neural Networks (CNNs). This paper showcases how CapsNets are more capable than CNNs for autonomous agent exploration of realistic scenarios. In real world navigation, rewards external to agents may be rare. In turn, reinforcement learning algorithms can struggle to form meaningful policy functions. This paper’s approach Capsules Exploration Module (Caps-EM) pairs a CapsNets architecture with an Advantage Actor Critic algorithm. Other approaches for navigating sparse environments require intrinsic reward generators, such as the Intrinsic Curiosity Module (ICM) and Augmented Curiosity Modules (ACM). CapsEM uses a more compact architecture without need for intrinsic rewards. Tested using ViZDoom, the CapsEM uses 44% and 83% fewer trainable network parameters than the ICM and Depth-Augmented Curiosity Module (D-ACM), respectively, for 1141% and 437% average time improvement over the ICM and DACM, respectively, for converging to a policy function across My Way Home scenarios.

Multi-Agent Actor Critic for Channel Allocation in Heterogeneous Networks

Heterogeneous networks (HetNets) can equalize traffic loads and cut down the cost of deploying cells. Thus, it is regarded to be the significant technique of the next-generation communication networks. Due to the non-convexity nature of the channel allocation problem in HetNets, it is difficult to design an optimal approach for allocating channels. To ensure the user quality of service as well as the long-term total network utility, this article proposes a new method through utilizing multi-agent reinforcement learning. Moreover, for the purpose of solving computational complexity problem caused by the large action space, deep reinforcement learning is put forward to learn optimal policy. A nearly-optimal solution with high efficiency and rapid convergence speed could be obtained by this learning method. Simulation results reveal that this new method has the best performance than other methods.

Score Following as a Multi-Modal Reinforcement Learning Problem

Score following is the process of tracking a musical performance (audio) in a corresponding symbolic representation (score). While methods using computer-readable score representations as input are able to achieve reliable tracking results, there is little research on score following based on raw score images. In this paper, we build on previous work that formulates the score following task as a multi-modal Markov Decision Process (MDP). Given this formal definition, one can address the problem of score following with state-of-the-art deep reinforcement learning (RL) algorithms. In particular, we design end-to-end multi-modal RL agents that simultaneously learn to listen to music recordings, read the scores from images of sheet music, and follow the music along in the sheet. Using algorithms such as synchronous Advantage Actor Critic (A2C) and Proximal Policy Optimization (PPO), we reproduce and further improve existing results. We also present first experiments indicating that this approach can be extended to track real piano recordings of human performances. These audio recordings are made openly available to the research community, along with precise note-level alignment ground truth.

Object Detection with Deep Neural Networks for Reinforcement Learning in the Task of Autonomous Vehicles Path Planning at the Intersection

Among a number of problems in the behavior planning of an unmanned vehicle the central one is movement in difficult areas. In particular, such areas are intersections at which direct interaction with other road agents takes place. In our work, we offer a new approach to train of the intelligent agent that simulates the behavior of an unmanned vehicle, based on the integration of reinforcement learning and computer vision. Using full visual information about the road intersection obtained from aerial photographs, it is studied automatic detection the relative positions of all road agents with various architectures of deep neural networks (YOLOv3, Faster R-CNN, RetinaNet, Cascade R-CNN, Mask R-CNN, Cascade Mask R-CNN). The possibilities of estimation of the vehicle orientation angle based on a convolutional neural network are also investigated. Obtained additional features are used in the modern effective reinforcement learning methods of Soft Actor Critic and Rainbow, which allows to accelerate the convergence of its learning process. To demonstrate the operation of the developed system, an intersection simulator was developed, at which a number of model experiments were carried out.

Application of deep reinforcement learning to intrusion detection for supervised problems

The application of new techniques to increase the performance of intrusion detection systems is crucial in modern data networks with a growing threat of cyber-attacks. These attacks impose a greater risk on network services that are increasingly important from a social end economical point of view. In this work we present a novel application of several deep reinforcement learning (DRL) algorithms to intrusion detection using a labeled dataset. We present how to perform supervised learning based on a DRL framework.The implementation of a reward function aligned with the detection of intrusions is extremely difficult for Intrusion Detection Systems (IDS) since there is no automatic way to identify intrusions. Usually the identification is performed manually and stored in datasets of network features associated with intrusion events. These datasets are used to train supervised machine learning algorithms for classifying intrusion events. In this paper we apply DRL using two of these datasets: NSL-KDD and AWID datasets. As a novel approach, we have made a conceptual modification of the classic DRL paradigm (based on interaction with a live environment), replacing the environment with a sampling function of recorded training intrusions. This new pseudo-environment, in addition to sampling the training dataset, generates rewards based on detection errors found during training.We present the results of applying our technique to four of the most relevant DRL models: Deep Q-Network (DQN), Double Deep Q-Network (DDQN), Policy Gradient (PG) and Actor-Critic (AC). The best results are obtained for the DDQN algorithm.We show that DRL, with our model and some parameter adjustments, can improve the results of intrusion detection in comparison with current machine learning techniques. Besides, the classifier obtained with DRL is faster than alternative models. A comprehensive comparison of the results obtained with other machine learning models is provided for the AWID and NSL-KDD datasets, together with the lessons learned from the application of several design alternatives to the four DRL models.

Decentralized Nonlinear Adaptive Optimal Control Scheme for Enhancement of Power System Stability

An adaptive decentralized control scheme is proposed for real-time control of oscillatory dynamics and overall stability improvement of an interconnected power system. A standard framework of continuous-time (CT) infinite-horizon optimal control paradigm is defined and an extended online actor-critic (AC) algorithm based on policy iteration is used for its solution. The AC structure uses neural networks (NNs) whose weights are updated adaptively. The proof for the convergence of the scheme guarantees the system stability. The unobservable internal states of the synchronous machine are estimated using a numerically stable, swift and relatively accurate decentralized dynamic state estimator (DDSE) based on spherical-radial cubature rule. Applicability of the developed scheme has been demonstrated on a benchmark power system (IEEE 2.2, 16 machine 68 bus system) model via nonlinear time-domain simulations. Multi-processor technology based scaled laboratory setup was used for controller performance validation in real-time.