Policy Gradient Method Research Articles

Self-play reinforcement learning with comprehensive critic in computer games

Self-play reinforcement learning, where agents learn by playing with themselves, has been successfully applied in many game scenarios. However, the training procedure for self-play reinforcement learning is unstable and more sample-inefficient than (general) reinforcement learning, especially in imperfect information games. To improve the self-play training process, we incorporate a comprehensive critic into the policy gradient method to form a self-play actor-critic (SPAC) method for training agents to play computer games. We evaluate our method in four different environments in both competitive and cooperative tasks. The results show that the agent trained with our SPAC method outperforms those trained with deep deterministic policy gradient (DDPG) and proximal policy optimization (PPO) algorithms in many different evaluation approaches, which vindicate the effect of our comprehensive critic in the self-play training procedure.

Efficient Robot Skills Learning with Weighted Near-Optimal Experiences Policy Optimization

Autonomous learning of robotic skills seems to be more natural and more practical than engineered skills, analogous to the learning process of human individuals. Policy gradient methods are a type of reinforcement learning technique which have great potential in solving robot skills learning problems. However, policy gradient methods require too many instances of robot online interaction with the environment in order to learn a good policy, which means lower efficiency of the learning process and a higher likelihood of damage to both the robot and the environment. In this paper, we propose a two-phase (imitation phase and practice phase) framework for efficient learning of robot walking skills, in which we pay more attention to the quality of skill learning and sample efficiency at the same time. The training starts with what we call the first stage or the imitation phase of learning, updating the parameters of the policy network in a supervised learning manner. The training set used in the policy network learning is composed of the experienced trajectories output by the iterative linear Gaussian controller. This paper also refers to these trajectories as near-optimal experiences. In the second stage, or the practice phase, the experiences for policy network learning are collected directly from online interactions, and the policy network parameters are updated with model-free reinforcement learning. The experiences from both stages are stored in the weighted replay buffer, and they are arranged in order according to the experience scoring algorithm proposed in this paper. The proposed framework is tested on a biped robot walking task in a MATLAB simulation environment. The results show that the sample efficiency of the proposed framework is much higher than ordinary policy gradient algorithms. The algorithm proposed in this paper achieved the highest cumulative reward, and the robot learned better walking skills autonomously. In addition, the weighted replay buffer method can be made as a general module for other model-free reinforcement learning algorithms. Our framework provides a new way to combine model-based reinforcement learning with model-free reinforcement learning to efficiently update the policy network parameters in the process of robot skills learning.

Hybrid deep reinforcement learning based eco-driving for low-level connected and automated vehicles along signalized corridors

Eco-Driving has great potential in reducing the fuel consumption of road vehicles, especially under the connected and automated vehicles (CAVs) environment. Traditional model-based Eco-Driving methods usually require sophisticated models and cannot deal with complex driving scenarios. This paper proposes a hybrid reinforcement learning (RL) based Eco-Driving algorithm considering both the longitudinal acceleration/deceleration and the lateral lane-changing operations. A deep deterministic policy gradient (DDPG) algorithm is designed to learn the continuous longitudinal acceleration/deceleration to reduce the fuel consumption as well as to maintain acceptable travel time. Collecting the critic’s value of each single lane from DDPG and integrating the information of adjacent lanes, a deep Q-learning algorithm is developed to make the discrete lane-changing decision. Together, a hybrid deep Q-learning and policy gradient (HDQPG) method is developed for vehicles driving along multi-lane urban signalized corridors. The method can enable the controlled vehicle to learn well-established longitudinal fuel-saving strategies, and to perform appropriate lane-changing operations at proper times to avoid congested lanes. Numerical experiments show that HDQPG can reduce fuel consumption by up to 46% with marginal or no increase of travel times.

Two-Stage Volt/Var Control in Active Distribution Networks With Multi-Agent Deep Reinforcement Learning Method

The high penetration of intermittent renewable energy resources in active distribution networks (ADN) results in a great challenge for the conventional Volt-Var control (VVC). This article proposes a two-stage deep reinforcement learning (DRL)-based real-time VVC method to mitigate fast voltage violation while minimizing the network power loss. In the first stage, on-load tap changer (OLTC) and capacitor banks (CBs) are dispatched hourly based on the optimal power flow method. The optimization problem is formulated as a mixed-integer second-order cone programming (MISOCP) which can be effectively solved. In the second stage, the reactive power of photovoltaics (PVs) is regulated dynamically to mitigate fast voltage fluctuation based on the well-learned control strategy and local measurements. The real-time VVC problem is formulated and solved using a multi-agent deep deterministic policy gradient (MADDPG) method, which features offline centralized training and online decentralized application. Rather than using the critic network to evaluate the output of the actor-network, the gradient of the action-value function to action is derived analytically based on the voltage sensitivity method. The proposed approach is tested on the IEEE 33-bus distribution system and comparative simulation results show the enhanced control effect in mitigating voltage violations.

Workflow scheduling based on deep reinforcement learning in the cloud environment

As a convenient and economic computing model, cloud computing promotes the development of intelligence. Solving the workflow scheduling is a significant topic to promote the development of the cloud computing. In this work, an Actor-Critic architecture is utilized to solve this problem achieving the task executive time minimization under the task precedence constraint. It is similar to the list-based heuristic algorithm which includes the task prioritizing phase and task allocation phase. However, the results of the two phases interact with each other. In the task prioritizing phase, given a workflow represented as the data communication time matrix and task computation time matrix, a distribution over different task permutations by the improved Pointer network can be predicted. Then, the heuristic algorithm based on the HEFT achieves the task allocation to get the task executive time. Using negative task executive time as the reward signals, the model parameters by a policy gradient method in the first phase can be optimized. The simulation experiment is done from the task executive time, and the results shows that the workflow scheduling by the deep reinforcement learning is more effective comparing with other four single objective heuristic algorithms.

Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games

We consider a general-sum N-player linear-quadratic game with stochastic dynamics over a finite horizon and prove the global convergence of the natural policy gradient method to the Nash equilibrium. In order to prove convergence of the method we require a certain amount of noise in the system. We give a condition, essentially a lower bound on the covariance of the noise in terms of the model parameters, in order to guarantee convergence. We illustrate our results with numerical experiments to show that even in situations where the policy gradient method may not converge in the deterministic setting, the addition of noise leads to convergence.

Policy Gradient Importance Sampling for Bayesian Inference

In this paper, we propose a novel adaptive importance sampling (AIS) algorithm for probabilistic inference. The sampler learns a proposal distribution adaptation strategy by framing AIS as a reinforcement learning problem. Under this structure, the proposal distribution of the sampler is treated as an agent whose state is controlled using a parameterized policy. At each iteration of the algorithm, the agent earns a reward that is related to its contribution to the variance of the AIS estimator of the normalization constant of the target distribution. Policy gradient methods are employed to learn a locally optimal policy that maximizes the expected value of the sum of all rewards. Numerical simulations on two different examples demonstrate promising results for the future application of the proposed method to complex Bayesian models.

Trajectory Based Prioritized Double Experience Buffer for Sample-Efficient Policy Optimization

Reinforcement learning has recently made great progress in various challenging domains such as board game of Go and MOBA game of StarCraft II. Policy gradient based reinforcement learning method has become the mainstream due to its effectiveness and simplicity both in discrete and continuous scenarios. However, policy gradient methods commonly involve function approximation and work in an on-policy fashion, which leads to high variance and low sample efficiency. This paper introduces a novel policy gradient method to improve the sample efficiency via a pair of trajectory based prioritized replay buffers and reduce the variance in training with a target network whose weights are updated in a “soft” manner. We evaluate our method on the reinforcement learning suit of Open AI Gym tasks, and the results show that the proposed method can learn more steadily and achieve higher performance than existing methods.

Human locomotion with reinforcement learning using bioinspired reward reshaping strategies.

Recent learning strategies such as reinforcement learning (RL) have favored the transition from applied artificial intelligence to general artificial intelligence. One of the current challenges of RL in healthcare relates to the development of a controller to teach a musculoskeletal model to perform dynamic movements. Several solutions have been proposed. However, there is still a lack of investigations exploring the muscle control problem from a biomechanical point of view. Moreover, no studies using biological knowledge to develop plausible motor control models for pathophysiological conditions make use of reward reshaping. Consequently, the objective of the present work was to design and evaluate specific bioinspired reward function strategies for human locomotion learning within an RL framework. The deep deterministic policy gradient (DDPG) method for a single-agent RL problem was applied. A 3D musculoskeletal model (8 DoF and 22 muscles) of a healthy adult was used. A virtual interactive environment was developed and simulated using opensim-rl library. Three reward functions were defined for walking, forward, and side falls. The training process was performed with Google Cloud Compute Engine. The obtained outcomes were compared to the NIPS 2017 challenge outcomes, experimental observations, and literature data. Regarding learning to walk, simulated musculoskeletal models were able to walk from 18 to 20.5 m for the best solutions. A compensation strategy of muscle activations was revealed. Soleus, tibia anterior, and vastii muscles are main actors of the simple forward fall. A higher intensity of muscle activations was also noted after the fall. All kinematics and muscle patterns were consistent with experimental observations and literature data. Regarding the side fall, an intensive level of muscle activation on the expected fall side to unbalance the body was noted. The obtained outcomes suggest that computational and human resources as well as biomechanical knowledge are needed together to develop and evaluate an efficient and robust RL solution. As perspectives, current solutions will be extended to a larger parameter space in 3D. Furthermore, a stochastic reinforcement learning model will be investigated in the future in scope with the uncertainties of the musculoskeletal model and associated environment to provide a general artificial intelligence solution for human locomotion learning. Graphical abstract.

Deep Reinforcement Learning Based Energy Efficient Multi-UAV Data Collection for IoT Networks

Unmanned aerial vehicles (UAVs) are regarded as an emerging technology, which can be effectively utilized to perform the data collection tasks in the Internet of Things (IoT) networks. However, both the UAVs and the sensors in these networks are energy-limited devices, which necessitates an energy-efficient data collection procedure to ensure the network lifetime. In this paper, we propose a multi-UAV-assisted network, where the UAVs fly to the ground sensors and control the sensor's transmit power during the data collection time. Our goal is to minimize the total energy consumption of the UAVs and the sensors, which is needed to accomplish the data collection mission. We formulate this problem into three sub-problems of single UAV navigation, sensor power control as well as multi-UAV scheduling and model each part as a finite-horizon Markov Decision Process (MDP). We deploy deep reinforcement learning (DRL)-based frameworks to solve each part. Specifically, we use deep deterministic policy gradient (DDPG) method to generate the best trajectory for the UAVs in an obstacle-constraint environment, given its starting position and the target sensor. We also deploy DDPG to control the sensor's transmit power during data collection. To schedule activity plans for each UAV to visit the sensors, we propose a multi-agent deep Q-learning (DQL) approach by taking the total energy consumption of the UAVs on each path into account. Our simulations show that the UAVs can find a safe and optimal path for each of their trips. Continuous power control of the sensors achieves better performance over the fixed power approaches in terms of the total energy consumption during data collection. In addition, compared to the two commonly used baselines, our scheduling framework achieves better and near-optimal results.

Resilience Microgrid as Power System Integrity Protection Scheme Element With Reinforcement Learning Based Management

The microgrid is a solution for integrating renewable energy resources into the power system. However, overcoming the randomness of these nature-based resources requires a robust control system. Moreover, electricity market participation and ancillary service provision for the utility grid are other aspects, although intensify microgrid penetration makes its environment interactions more complex. Reinforcement learning is a technique vastly applied to such an intricate environment. Hence, in this paper, we deployed deep deterministic policy gradient and soft-actor critic methods to solve the high-dimensional, continuous, and stochastic problem of the microgrid's energy management system and compared the performance of two methods. Additionally, we developed the microgrid interactions with the utility grid as a participant of system integrity protection schema responding promptly to the utility grid protection requirements based on its reliable available resources. Moreover, we applied actual data of Gasa Island microgrid in Korea to prove the efficiency of proposed method.

Deep Learning-Based Energy Management of an All-Electric City Bus With Wireless Power Transfer

Fuel cell-based hybrid electric vehicles are one of the most promising options to achieve zero-emission city buses. Efficient Energy Management (EM) plays a critical role to make such buses more efficient and practical. In this research, an available all-electric bus consisting of fuel cell (FC) and battery is considered and the efficiency of adding a Wireless Power Transfer (WPT) system to it is assessed. The proposed WPT system is only capable to receive energy in bus stations and use it to supply loads or charge the battery. To this end, the actual data of a city bus, its route and load profile were collected and utilized to ensure a realistic assessment. A full mathematical model of the energy system as well as the constraints governing the management issue is extracted and a Deep Deterministic Policy Gradient (DDPG) method is used to optimally manage the energy flows for the entire journey. All models are implemented in MATLAB software and the efficiency of the proposed system is investigated from economic and technical aspects. The results illustrate a high efficiency for the proposed WPT technique to be used in actual all-electric city buses.

A Hybrid Tracking Control Strategy for Nonholonomic Wheeled Mobile Robot Incorporating Deep Reinforcement Learning Approach

Tracking control is an essential capability for nonholonomic wheeled mobile robots (NWMR) to achieve autonomous navigation. This paper presents a novel hybrid control strategy combined mode-based control and actor-critic based deep reinforcement learning method. Based on the Lyapunov method, a kinematics control law named given control is obtained with pose errors. Then, the tracking control problem is converted to a finite Markov decision process, in which the defined state contains current tracking errors, given control inputs and one-step errors. After training with deep deterministic policy gradient method, the action named acquired control inputs is capable of compensating the existing errors. Thus, the hybrid control strategy is obtained under velocity constraint, acceleration constraint and bounded uncertainty. A cumulative error is also defined as a criteria to evaluate tracking performance. The comparison results in simulation demonstrate that our proposed method have an obviously advantage on both tracking accuracy and training efficiency.

Simultaneous Process Design and Control Optimization using Reinforcement Learning

Abstract The performance of a chemical plant is highly affected by its design and control. A design cannot be accurately evaluated without its controls and vice versa. To optimally address design and control simultaneously, one must formulate a bi-level mixed-integer nonlinear program with a dynamic optimization problem as the inner problem; this is intractable. However, by computing an optimal policy using reinforcement learning, a controller with a closed-form expression can be computed and embedded into the mathematical program. In this work, an approach that uses a policy gradient method to compute the optimal policy, which is then embedded into the mathematical program is proposed. The approach is tested in a tank design case study and the performance of the controller is evaluated. It is shown that the proposed approach outperforms current state-of-the-art control strategies. This opens a whole new range of possibilities to address the simultaneous design and control of engineering systems.

Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon

We explore reinforcement learning methods for finding the optimal policy in the linear quadratic regulator (LQR) problem. In particular we consider the convergence of policy gradient methods in the...

Policy gradient methods for free-electron laser and terahertz source optimization and stabilization at the FERMI free-electron laser at Elettra

In this article we report on the application of a model-free reinforcement learning method to the optimization of accelerator systems. We simplify a policy gradient algorithm to accelerator control from sophisticated algorithms that have recently been demonstrated to solve complex dynamic problems. After outlining a theoretical basis for the functioning of the algorithm, we explore the small hyperparameter space to develop intuition about said parameters using a simple number-guess environment. Finally, we demonstrate the algorithm optimizing both a free-electron laser and an accelerator-based terahertz source in-situ. The algorithm is applied to different accelerator control systems and optimizes the desired signals in a few hundred steps without any domain knowledge using up to five control parameters. In addition, the algorithm shows modest tolerance to accelerator fault conditions without any special preparation for such conditions.

Метод синтеза нейронных регуляторов для линейных объектов

The article proposes a method for the synthesis of a neural controller for closed-loop systems with linear objects. The scientific novelty of the proposed method lies in the fact that the neural controller, to the input of which the object state vector is fed, must be trained to stabilize in one of the possible desired values, and to ensure regulation in other desired values. For objects with an inaccessible state vector, it is possible to use the estimation vector of the object state vector. It is proposed to proportionally decrease/increase the signal of the state vector and increase/decrease the control signal formed by the neural regulator. Also, other advantages of the proposed method include: 1) the absence of the need for training on several desired values, which greatly simplifies and accelerates the training of the neural network, and also eliminates control errors in the range of values for which the neural controller was not trained; 2) the possibility of learning from an initially unstable state of a closed-loop system. The proposed method for the synthesis of a neural controller for a closed-loop system with a linear object was tested on the example of the synthesis of a controller for an object 1/s 3, which is unstable. A neural network is used as a regulator, which is proposed to be trained using one of the reinforcement learning methods (in the article, the Deterministic Policy Gradient method allowed us to obtain the best results). The resulting graphs of transient processes allow us to conclude about its successful application. The article ends with conclusions and considerations about further lines of research, which include the quality of the transient process and the possibility of adjusting it by changing the reward function, which will allow setting the graphs of transient processes.

Reinforcement learning-based hybrid spectrum resource allocation scheme for the high load of URLLC services

Ultra-reliable and low-latency communication (URLLC) in mobile networks is still one of the core solutions that require thorough research in 5G and beyond. With the vigorous development of various emerging URLLC technologies, resource shortages will soon occur even in mmWave cells with rich spectrum resources. As a result of the large radio resource space of mmWave, traditional real-time resource scheduling decisions can cause serious delays. Consequently, we investigate a delay minimization problem with the spectrum and power constraints in the mmWave hybrid access network. To reduce the delay caused by high load and radio resource shortage, a hybrid spectrum and power resource allocation scheme based on reinforcement learning (RL) is proposed. We compress the state space and the action space by temporarily dumping and decomposing the action. The multipath deep neural network and policy gradient method are used, respectively, as the approximater and update method of the parameterized policy. The experimental results reveal that the RL-based hybrid spectrum and the power resource allocation scheme eventually converged after a limited number of iterative learnings. Compared with other schemes, the RL-based scheme can effectively guarantee the URLLC delay constraint when the load does not exceed 130%.

Multi-Agent Deep Reinforcement Learning-Based Trajectory Planning for Multi-UAV Assisted Mobile Edge Computing

An unmanned aerial vehicle (UAV)-aided mobile edge computing (MEC) framework is proposed, where several UAVs having different trajectories fly over the target area and support the user equipments (UEs) on the ground. We aim to jointly optimize the geographical fairness among all the UEs, the fairness of each UAV’ UE-load and the overall energy consumption of UEs. The above optimization problem includes both integer and continues variables and it is challenging to solve. To address the above problem, a multi-agent deep reinforcement learning based trajectory control algorithm is proposed for managing the trajectory of each UAV independently, where the popular Multi-Agent Deep Deterministic Policy Gradient (MADDPG) method is applied. Given the UAVs’ trajectories, a low-complexity approach is introduced for optimizing the offloading decisions of UEs. We show that our proposed solution has considerable performance over other traditional algorithms, both in terms of the fairness for serving UEs, fairness of UE-load at each UAV and energy consumption for all the UEs.

MobileVisFixer: Tailoring Web Visualizations for Mobile Phones Leveraging an Explainable Reinforcement Learning Framework.

We contribute MobileVisFixer, a new method to make visualizations more mobile-friendly. Although mobile devices have become the primary means of accessing information on the web, many existing visualizations are not optimized for small screens and can lead to a frustrating user experience. Currently, practitioners and researchers have to engage in a tedious and time-consuming process to ensure that their designs scale to screens of different sizes, and existing toolkits and libraries provide little support in diagnosing and repairing issues. To address this challenge, MobileVisFixer automates a mobile-friendly visualization re-design process with a novel reinforcement learning framework. To inform the design of MobileVisFixer, we first collected and analyzed SVG-based visualizations on the web, and identified five common mobile-friendly issues. MobileVisFixer addresses four of these issues on single-view Cartesian visualizations with linear or discrete scales by a Markov Decision Process model that is both generalizable across various visualizations and fully explainable. MobileVisFixer deconstructs charts into declarative formats, and uses a greedy heuristic based on Policy Gradient methods to find solutions to this difficult, multi-criteria optimization problem in reasonable time. In addition, MobileVisFixer can be easily extended with the incorporation of optimization algorithms for data visualizations. Quantitative evaluation on two real-world datasets demonstrates the effectiveness and generalizability of our method.