Evolutionary Reinforcement Learning Research Articles

State-Dependent Maximum Entropy Reinforcement Learning for Robot Long-Horizon Task Learning

Task-oriented robot learning has shown significant potential with the development of Reinforcement Learning (RL) algorithms. However, the learning of long-horizon tasks for robots remains a formidable challenge due to the inherent complexity of tasks, typically comprising multiple diverse stages. Universal RL algorithms commonly encounter issues such as slow convergence or even failure to converge altogether when applied to such tasks. The reasons behind these challenges lie in the local optima trap or redundant exploration during the new stages or the junction of two continuous stages. To address these challenges, we propose a novel state-dependent maximum entropy (SDME) reinforcement learning algorithm. This algorithm effectively balances the trade-off between exploration and exploitation around three kinds of critical states arising from the unique nature of long-horizon tasks. We conducted experiments within an open-source simulation environment, focusing on two representative long-horizon tasks. The proposed SDME algorithm exhibits faster and more stable learning capabilities, requiring merely one-third of the number of learning samples necessary for baseline approaches. Furthermore, we assess the generalization ability of our method under randomly initialized conditions, and the results show that the success rate of the SDME algorithm is nearly twice that of the baselines. Our code will be available at https://github.com/Peter-zds/SDME.

Learning-Based Adaptive Optimal Control of Linear Time-Delay Systems: A Policy Iteration Approach

This paper studies the adaptive optimal control problem for a class of linear time-delay systems described by delay differential equations (DDEs). A crucial strategy is to take advantage of recent developments in reinforcement learning (RL) and adaptive dynamic programming (ADP) and develop novel methods to learn adaptive optimal controllers from finite samples of input and state data. In this paper, the data-driven policy iteration (PI) is proposed to solve the infinite-dimensional algebraic Riccati equation (ARE) iteratively in the absence of exact model knowledge. Interestingly, the proposed recursive PI algorithm is new in the present context of continuous-time time-delay systems, even when the model knowledge is assumed known. The efficacy of the proposed learning-based control methods is validated by means of practical applications arising from metal cutting and autonomous driving.

Adaptive Optimization in Evolutionary Reinforcement Learning Using Evolutionary Mutation Rates

Adaptive Optimization in Evolutionary Reinforcement Learning Using Evolutionary Mutation Rates

Editorial to the “Evolutionary Reinforcement Learning” Special Issue

Editorial to the “Evolutionary Reinforcement Learning” Special Issue

Solving an energy resource management problem with a novel multi-objective evolutionary reinforcement learning method

Microgrids have become popular candidates for integrating diverse energy sources into the power grid as means of reducing fossil fuel usage. Energy Resource Management (ERM) is a type of Unit Commitment problem, where a player operates a microgrid with diverse renewable generators integrated with an external supplier. Calculating the economic dispatch of each committed unit on a planning horizon is an NP-hard problem, and therefore, finding an exact solution is difficult. This paper presents a multi-objective solution to the ERM problem from the perspective of battery operation and external supplier dispatch. First, a novel multi-objective decision problem modeling is proposed that considers three objectives: cost, greenhouse gas emissions, and battery degradation. This framework involves a learning agent that controls the depth of discharge of a Lithium-Ion battery. To address the proposed problem, a new multi-objective algorithm called Multi-Objective Evolutionary Policy Search (MEPS) is introduced. The proposed algorithm uses NeuroEvolution of Augmenting Topologies structure to evolve artificial neural networks for estimating action-preference values considering multi-objective rewards. The MEPS performance is evaluated on both standard and newly-proposed benchmark problems, using the hypervolume as the evaluation metric. When compared to standard deep reinforcement learning, results showed that MEPS provides cost-effective, environmentally friendly, and efficient energy storage management solutions. Furthermore, MEPS effectively solves the proposed ERM problem by finding neural networks with a small number of nodes and connections, which are suitable for use in embedded control systems. Overall, MEPS proved to be a promising multi-objective approach in the transition to clean energy resources.

A survey on Evolutionary Reinforcement Learning algorithms

Reinforcement Learning (RL) has proven to be highly effective in various real-world applications. However, in certain scenarios, Evolutionary Algorithms (EAs) have been utilized as an alternative to RL algorithms. Recently, Evolutionary Reinforcement Learning algorithms (ERLs) have emerged as a promising solution that combines the advantages of both RL and EA. This paper presents a comprehensive survey that encompasses a majority of the studies in this exciting research area. We classify these ERLs according to the EA used in their frameworks and analyze the strengths and limitations of various EA components and combination schemes. Additionally, we conduct several experiments to evaluate the performance of some representative ERLs. By categorizing the different approaches and assessing their effectiveness, the paper can assist researchers and practitioners in selecting the most suitable method for their particular application.

Evolutionary reinforcement learning algorithm for large-scale multi-agent cooperation and confrontation applications

Evolutionary reinforcement learning algorithm for large-scale multi-agent cooperation and confrontation applications

Explainable multi-agent deep reinforcement learning for real-time demand response towards sustainable manufacturing

The demand response (DR) plays a significant role in manufacturing system energy management and sustainable industrial development. Current literature on DR management for manufacturing systems has mostly focused on day-ahead production scheduling, whose effectiveness is limited due to the lack of flexibility to control the production line in real time. The development of reinforcement learning (RL) possesses huge potential for real-time production control to address the flexibility issue. However, since production is the top priority for any manufacturing system, a trustful and explainable RL for manufacturing system energy management that can ensure the production requirements is necessary for this application. This study proposes an explainable multi-agent deep RL method, where the analytical manufacturing system model is applied to decompose the system-level energy management objective and production requirement to the agent level. Based on the decomposed task, the agent can then form a safe action subset that is interpretable to achieve the original system-level production requirement while learning to reduce energy costs under DR. The proposed RL method, which is referred to as decomposed multi-agent deep Q-network (DMADQN), is applied to control a section of an automotive assembly line using one year of DR electricity price data to validate its performance. Results show that the proposed method ensures the achievement of the production requirement while providing better DR energy management performance in both RL training and testing phases. In addition, the proposed approach can outperform the day-ahead scheduling approach and save up to an additional 30.7% of energy costs under dynamic DR.

Multi-agent evolution reinforcement learning method for machining parameters optimization based on bootstrap aggregating graph attention network simulated environment

Improving machining quality and production efficiency is the focus of the manufacturing industry. How to obtain efficient machining parameters under multiple constraints such as machining quality is a severe challenge for manufacturing industry. In this paper, a multi-agent evolutionary reinforcement learning method (MAERL) is proposed to optimize the machining parameters for high quality and high efficiency machining by combining the graph neural network and reinforcement learning. Firstly, a bootstrap aggregating graph attention network (Bagging-GAT) based roughness estimation method for machined surface is proposed, which combines the structural knowledge between machining parameters and vibration features. Secondly, a mathematical model of machining parameters optimization problem is established, which is formalized into Markov decision process (MDP), and a multi-agent reinforcement learning method is proposed to solve the MDP problem, and evolutionary learning is introduced to improve the stability of multi-agent training. Finally, a series of experiments were carried out on the commutator production line, and the results show that the proposed Bagging-GAT-based method can improve the prediction effect by about 25% in the case of small samples, and the MAERL-based optimization method can better deal with the coupling problem of reward function in the optimization process. Compared with the classical optimization method, the optimization effect is improved by 13% and a lot of optimization time is saved.

Simulated and Real Robotic Reach, Grasp, and Pick-and-Place Using Combined Reinforcement Learning and Traditional Controls

The majority of robots in factories today are operated with conventional control strategies that require individual programming on a task-by-task basis, with no margin for error. As an alternative to the rudimentary operation planning and task-programming techniques, machine learning has shown significant promise for higher-level task planning, with the development of reinforcement learning (RL)-based control strategies. This paper reviews the implementation of combined traditional and RL control for simulated and real environments to validate the RL approach for standard industrial tasks such as reach, grasp, and pick-and-place. The goal of this research is to bring intelligence to robotic control so that robotic operations can be completed without precisely defining the environment, constraints, and the action plan. The results from this approach provide optimistic preliminary data on the application of RL to real-world robotics.

Evolutionary Reinforcement Learning: A Survey

Reinforcement learning (RL) is a machine learning approach that trains agents to maximize cumulative rewards through interactions with environments. The integration of RL with deep learning has recently resulted in impressive achievements in a wide range of challenging tasks, including board games, arcade games, and robot control. Despite these successes, several critical challenges remain, such as brittle convergence properties caused by sensitive hyperparameters, difficulties in temporal credit assignment with long time horizons and sparse rewards, a lack of diverse exploration, particularly in continuous search space scenarios, challenges in credit assignment in multi-agent RL, and conflicting objectives for rewards. Evolutionary computation (EC), which maintains a population of learning agents, has demonstrated promising performance in addressing these limitations. This article presents a comprehensive survey of state-of-the-art methods for integrating EC into RL, referred to as evolutionary reinforcement learning (EvoRL). We categorize EvoRL methods according to key research areas in RL, including hyperparameter optimization, policy search, exploration, reward shaping, meta-RL, and multi-objective RL. We then discuss future research directions in terms of efficient methods, benchmarks, and scalable platforms. This survey serves as a resource for researchers and practitioners interested in the field of EvoRL, highlighting the important challenges and opportunities for future research. With the help of this survey, researchers and practitioners can develop more efficient methods and tailored benchmarks for EvoRL, further advancing this promising cross-disciplinary research field.

Intelligent Trajectory Design for Mobile Energy Harvesting and Data Transmission

Energy harvesting technology enables wireless sensor networks (WSNs) to be self-sustainable, for maintaining long-term key performance indicators, such as the data throughput and sensing coverage. Due to the highly dynamic and complex environment, energy sources (ES) cannot provide stable energy supply, which needs the efficient learning algorithm to enable system adaptations. This article reports on the development of reinforcement learning (RL) methodology to long-term data collection in self-sustainable WSNs. Specifically, we consider the WSN as a 2-D rectangular region, where a mobile sensor (MS) can harvest energy from ambient environments while transmitting the collected data to a fixed sink. Due to the changing environment and the mobility of the MS, the harvested energy by the MS at each slot presents spatiotemporal dynamics within the network, which severely affects the performance of data throughput from the MS to the sink. The MS’s trajectory is investigated to maximize the long-term average MS-to-sink data throughput. Due to the unknown energy arrival information as well as the locations of ESs, we formulate the problem as a Markov decision process, which is then solved with model-free RL. In particular, the deep deterministic policy gradient (DDPG) is applied to tackle the continuous and deterministic movement space. Results show that the MS can learn and optimize the moving trajectory by intelligently tracking the aggregated received energy over slots. Finally, the MS can identify and move to the optimal location where the maximized long-term average MS-to-sink data throughput is achieved. Extensive numerical evaluations are conducted to investigate the impact of various system parameters on the network performance.

Reinforcement Learning Control with Deep Deterministic Policy Gradient Algorithm for Multivariable pH Process

The pH treatment unit is widely used in various processes, such as wastewater treatment, pharmaceutical manufacturing, and fermentation. It is essential to get the on-specifications product. Thus, controlling pH is key management for accomplishing the manufacturing objective. However, the highly nonlinear pH characteristics of acid–base titration make pH regulation difficult. Applications of artificial intelligence for process control have progressed and gained popularity recently. The development of reinforcement learning (RL) control with a deep deterministic policy gradient (DDPG) algorithm to handle coupled pH and liquid level control in a continuous stirred tank reactor with a strong acid–base reaction is presented in this study. To validate the RL model, the reward functions are created individually for the level and pH controls. The grid search technique is deployed to optimize the hyperparameters of the RL controller models, including the number of nodes in the hidden layers and the number of episodes. The control performance of the proposed RL control system was compared with that of the proportional-integral controller in a servo-regulatory test. The simulation results show that the proposed RL controllers outperform the proportional-integral controllers in approaching setpoints faster, with better performance and less oscillation.

A surrogate-assisted controller for expensive evolutionary reinforcement learning

The integration of Reinforcement Learning (RL) and Evolutionary Algorithms (EAs) aims at simultaneously exploiting the sample efficiency as well as the diversity and robustness of the two paradigms. Recently, hybrid learning frameworks based on this principle have achieved great success in robot control tasks. However, in these methods, policies from the genetic population are evaluated via interactions with the real environments, severely restricting their applicability when such interactions are prohibitively costly. In this work, we propose Surrogate-assisted Controller (SC), a generic module that can be applied on top of existing hybrid frameworks to alleviate the computational burden of expensive fitness evaluation. The key to our approach is to leverage the critic network that is implemented in existing hybrid frameworks as a novel surrogate model, making it possible to estimate the fitness of individuals without environmental interactions. In addition, two model management strategies with the elite protection mechanism are introduced in SC to control the workflow, leading to a fast and stable optimization process. In the empirical studies, we combine SC with two state-of-the-art evolutionary reinforcement learning approaches to highlight its functionality and effectiveness. Experiments on six challenging continuous control benchmarks from the OpenAI Gym platform show that SC can not only significantly reduce the cost of interaction with the environment, but also bring better sample efficiency and dramatically boost the learning progress of the original hybrid framework.

A novelty-search-based evolutionary reinforcement learning algorithm for continuous optimization problems

A novelty-search-based evolutionary reinforcement learning algorithm for continuous optimization problems

Evolutionary reinforcement learning multi-agents system for intelligent traffic light control: new approach and case of study

<span>Due to the rapid growth of urban vehicles, traffic congestion has become more serious. The signalized intersections are used all over the world and still established in the new construction. This paper proposes a self-adapted approach, called evolutionary reinforcement learning multi-agents system (ERL-MA), which combines computational intelligence and machine learning. The concept of this work is to build an intelligent agent capable of developing senior skills to manage the traffic light control system at any type of junction, using two powerful tools: learning from the confronted experience and the assumption using the randomization concept. The ERL-MA is an independent multi-agents system composed of two layers: the modeling and the decision layers. The modeling layer uses the intersection modeling using generalized fuzzy graph technique. The decision layer uses two methods: the novel greedy genetic algorithm (NGGA), and the Q-learning. In the Q-learning method, a multi Q-tables strategy and a new reward formula are proposed. The experiments used in this work relied on a real case of study with a simulation of one-hour scenario at Pasubio area in Italy. The obtained results show that the ERL-MA system succeeds to achieve competitive results comparing to urban traffic optimization by integrated automation (UTOPIA) system using different metrics.</span>

Dynamic Scheduling Algorithm Based on Evolutionary Reinforcement Learning for Sudden Contaminant Events Under Uncertain Environment

For sudden drinking water pollution event, reasonable opening or closing valves and hydrants in a water distribution network (WDN), which ensures the isolation and discharge of contaminant as soon as possible, is considered as an effective emergency measure. In this paper, we propose an emergency scheduling algorithm based on evolutionary reinforcement learning (ERL), which can train a good scheduling policy by the combination of the evolutionary computation (EC) and reinforcement learning (RL). Then, the optimal scheduling policy can guide the operation of valves and hydrants in real time based on sensor information, and protect people from the risk of contaminated water. Experiments verify our algorithm can achieve good results and effectively reduce the impact of pollution events.

Improving DFIG performance under fault scenarios through evolutionary reinforcement learning based control

The doubly fed induction generator (DFIG) usually experiences high rotor current and DC capacitor link voltage spikes during system fault events. In this paper, a novel data-driven approach is proposed to enhance DFIG performance under fault scenarios. An advanced reinforcement learning algorithm called guided surrogate-gradient-based evolution strategy (GSES) is used to control the DFIG power and capacitor DC-link voltage by adjusting the optimal reference signals. This controller is able to prevent the DFIG rotor from over-current risk and maintain grid-connected operation. The proposed GSES-based control algorithm was evaluated through simulations on a 3.6-MW DFIG in the PSCAD/EMTDC software. Results have validated the effectiveness of the proposed GSES-based control algorithm in improving DFIG performance under various fault scenarios.

An Intelligent Penetration Test Simulation Environment Construction Method Incorporating Social Engineering Factors

The penetration test has many repetitive operations and requires advanced expert knowledge, therefore, the manual penetration test is inefficient. With the development of reinforcement learning, the intelligent penetration test has been a research hotspot. However, the existing intelligent penetration test simulation environments only focus on the exploits of target hosts by the penetration tester agent’s actions while ignoring the important role of social engineering in the penetration test in reality. In addition, the construction of the existing penetration test simulation environment is based on the traditional network graph model without integrating security factors and attributes, and it is difficult to express the interaction between the penetration tester and the target network. This paper constructs an improved network graph model for penetration test (NMPT), which integrates the relevant security attributes of the penetration test. The NMPT model lays the foundation for extending the penetration tester’s social engineering actions. Then, we propose an intelligent penetration test method that incorporates social engineering factors (SE-AIPT) based on the Markov Decision Process. We adopt several mainstream reinforcement learning algorithms to train attack agents. The experiments show that the SE-AIPT method could vividly model the penetration tester agent’s social engineering actions, which effectively improves the reality of the simulation environment. Moreover, the penetration tester agent shows superior effects in the attack path discovery in the intelligent penetration test simulation environment constructed by the SE-AIPT method.

Domain Knowledge-Based Evolutionary Reinforcement Learning for Sensor Placement.

Reducing pollutant detection time based on a reasonable sensor combination is desirable. Clean drinking water is essential to life. However, the water supply network (WSN) is a vulnerable target for accidental or intentional contamination due to its extensive geographic coverage, multiple points of access, backflow, infrastructure aging, and designed sabotage. Contaminants entering WSN are one of the most dangerous events that may cause sickness or even death among people. Using sensors to monitor the water quality in real time is one of the most effective ways to minimize negative consequences on public health. However, it is a challenge to deploy a limited number of sensors in a large-scale WSN. In this study, the sensor placement problem (SPP) is modeled as a sequential decision optimization problem, then an evolutionary reinforcement learning (ERL) algorithm based on domain knowledge is proposed to solve SPP. Extensive experiments have been conducted and the results show that our proposed algorithm outperforms meta-heuristic algorithms and deep reinforcement learning (DRL).