Traditional Reinforcement Learning Research Articles

On Transforming Reinforcement Learning With Transformers: The Development Trajectory.

Transformers, originally devised for natural language processing (NLP), have also produced significant successes in computer vision (CV). Due to their strong expression power, researchers are investigating ways to deploy transformers for reinforcement learning (RL), and transformer-based models have manifested their potential in representative RL benchmarks. In this paper, we collect and dissect recent advances concerning the transformation of RL with transformers (transformer-based RL (TRL)) to explore the development trajectory and future trends of this field. We group the existing developments into two categories: architecture enhancements and trajectory optimizations, and examine the main applications of TRL in robotic manipulation, text-based games (TBGs), navigation, and autonomous driving. Architecture enhancement methods consider how to apply the powerful transformer structure to RL problems under the traditional RL framework, facilitating more precise modeling of agents and environments compared to traditional deep RL techniques. However, these methods are still limited by the inherent defects of traditional RL algorithms, such as bootstrapping and the "deadly triad". Trajectory optimization methods treat RL problems as sequence modeling problems and train a joint state-action model over entire trajectories under the behavior cloning framework; such approaches are able to extract policies from static datasets and fully use the long-sequence modeling capabilities of transformers. Given these advancements, the limitations and challenges in TRL are reviewed and proposals regarding future research directions are discussed. We hope that this survey can provide a detailed introduction to TRL and motivate future research in this rapidly developing field.

Learning to search for parking like a human: A deep inverse reinforcement learning approach

The prevalence of on-street parking searches in urban downtown areas has led to significant externalities such as congestion, pollution, and collisions. Understanding the intricacies of parking search behavior is crucial for developing effective management strategies to mitigate these issues. Parking search is inherently a complex, sequential decision-making process, influenced by diverse driver preferences and dynamic urban environments. This study introduces a deep inverse reinforcement learning (DIRL) approach to model drivers’ parking search behavior. First, we constructed a high-fidelity parking simulation platform using Unity3D to replicate an urban road network, enabling the collection of 987 valid trajectories. We modeled the parking search process as a Markov decision process (MDP), with meticulously designed state-action pairs for accurate representation. Then, a maximum entropy-based DIRL model was developed to learn the reward function and search-for-parking policies of drivers. The experimental results demonstrate that the maximum entropy DIRL model significantly outperforms the traditional maximum entropy inverse reinforcement learning model, achieving a 19.0% improvement in accurately capturing final parking states and a 13.5% enhancement in characterizing overall trajectory distributions. Finally, we integrated these trained models into traditional traffic simulation systems to effectively observe the traffic state evolution with different parking search behaviors, providing valuable insights for optimizing urban traffic management strategies.

Optimizing energy efficiency, thermal comfort, and indoor air quality in HVAC systems using a robust DRL algorithm

This study presents a novel deep reinforcement learning (DRL) framework designed to optimize energy efficiency, thermal comfort, and indoor air quality, particularly focusing on CO2 levels, in Heating, Ventilation, and Air Conditioning (HVAC) systems. Existing DRL methods often face challenges in maintaining control effectiveness under environmental disturbances, leading to higher energy consumption and reduced system performance. To address these limitations, we propose the Decoupled Adversarial Long Short-Term Memory and Proximal Policy Optimization (DAL-PPO) algorithm, which integrates a Decoupled Adversarial Strategy (DAP) with Long Short-Term Memory networks (LSTM) to better handle temporal dynamics and disturbances. Using the EnergyPlus simulation tool and PyTorch, DAL-PPO was evaluated in a disturbed experimental setting, achieving a 15 % reduction in PMV and CO2 levels and an 8 % decrease in energy consumption compared to traditional reinforcement learning algorithms. These results highlight DAL-PPO's ability to maintain system stability and efficiency even under dynamic disturbances, offering a significant improvement in HVAC system control and contributing to the development of more energy-efficient and resilient buildings.

Research on the multi-area cooperative control method for novel power systems

As new energy sources increasingly penetrate novel power systems, the coordinated control of frequency regulation units has become more challenging, leading to the degraded performance of power grid control. The automatic generation control methods based on traditional reinforcement learning overly depend on exploration, making it difficult quickly obtain optimal solutions for multi-area cooperative control and resulting in suboptimal performance. To address this issue, this paper proposes a multi-area coordinated control method using a greedy actor-critic algorithm enhanced with expert experience replay. This method decouple the role of entropy for exploration and policy collapse, utilizing a proposal policy to secure high-value actions for policy updates. Thus the exploration and exploitation are balanced and the local optima is avoided. Additionally, the expert experience replay collects expert demonstration data with high-value to assist the learning of multi-agent. Therefore, the meaningless exploration in early training is reduced and the rapid attainment of optimal solutions is facilitated. This paper validates the proposed method through simulations on the model of the improved IEEE standard two-area load frequency control and the model of the Sichuan, Chongqing, and Hubei three-area. Compared with other various reinforcement learning methods, the proposed method is demonstrated with superior control performance.

GSADDQN: Combining GraphSAGE and reinforcement learning for routing optimization in software-defined optical transport network

As a result of rapid development of network communication technology, optical transport network (OTN) traffic has also experienced rapid growth in terms of information volume scale, traffic complexity, and spatiotemporal distribution dynamics. Because the OTN traffic demand has complex spatiotemporal fluctuations, the traditional deep reinforcement learning (DRL) algorithm has been applied to the routing optimization of software-defined OTNs. However, the traditional DRL algorithm has problems such as slow convergence, weak generalization ability, and load imbalance when performing routine tasks. To address these issues, we propose a graph sampling and aggregation (GraphSAGE)-based dueling deep Q-network (GSADDQN) algorithm for software-defined OTN routing optimization. First, we design a DRL-based routing decision model to find the best routing strategy for each optical network’s source–destination traffic demand. Second, considering the sparse connection characteristics of optical network nodes, we use sampling neighbors and a deep aggregation mechanism as the neural network model of the Dueling Deep Q-Network (Dueling DQN) algorithm so that the reinforcement learning agent can consciously aggregate important network information and improve the model’s convergence performance and generalization ability. Finally, we design simulation routing experiments based on Gym and evaluate the algorithm’s load balancing and generalization capabilities for different network topologies. The experimental results show that the GSADDQN algorithm has good convergence performance and load balancing ability in routing optimization of optical transmission networks and can generalize new network structures, maintaining good decision-making ability even during network node failures.

Efficient Robot Manipulation via Reinforcement Learning with Dynamic Movement Primitives-Based Policy

Reinforcement learning (RL) that autonomously explores optimal control policies has become a crucial direction for developing intelligent robots while Dynamic Movement Primitives (DMPs) serve as a powerful tool for efficiently expressing robot trajectories. This article explores an efficient integration of RL and DMP to enhance the learning efficiency and control performance of reinforcement learning in robot manipulation tasks by focusing on the forms of control actions and their smoothness. A novel approach, DDPG-DMP, is proposed to address the efficiency and feasibility issues in the current RL approaches that employ DMP to generate control actions. The proposed method naturally integrates a DMP-based policy into the actor–critic framework of the traditional RL approach Deep Deterministic Policy Gradient (DDPG) and derives the corresponding update formulas to learn the networks that properly decide the parameters of DMPs. A novel inverse controller is further introduced to adaptively learn the translation from observed states into various robot control signals through DMPs, eliminating the requirement for human prior knowledge. Evaluated on five robot arm control benchmark tasks, DDPG-DMP demonstrates significant advantages in control performance, learning efficiency, and smoothness of robot actions compared to related baselines, highlighting its potential in complex robot control applications.

Path Planning for Robots Combined with Zero-Shot and Hierarchical Reinforcement Learning in Novel Environments

Path planning for robots based on reinforcement learning encounters challenges in integrating semantic information about environments into the training process. In unseen or complex environmental information, agents often perform sub-optimally and require more training time. In response to these challenges, this manuscript pioneers a framework integrating zero-shot learning combined with hierarchical reinforcement learning to enhance agent decision-making in complex environments. Zero-shot learning enables agents to infer correct actions for previously unseen objects or situations based on learned semantic associations. Subsequently, the path planning component utilizes hierarchical reinforcement learning with adaptive replay buffer, directed by the insights gained from zero-shot learning, to make decisions effectively. Two parts are trained separately, so zero-shot learning is available in different and unseen environments. Through simulation experiments, we compare the traditional hierarchical reinforcement learning method with the proposed method. The results prove that this structure can make full use of environmental information to generalize across unseen environments and plan collision-free paths.

Safe reinforcement learning-based control using deep deterministic policy gradient algorithm and slime mould algorithm with experimental tower crane system validation

This paper presents a novel optimal control approach resulting from the combination between the safe Reinforcement Learning (RL) framework represented by a Deep Deterministic Policy Gradient (DDPG) algorithm and a Slime Mould Algorithm (SMA) as a representative nature-inspired optimization algorithm. The main drawbacks of the traditional DDPG-based safe RL optimal control approach are the possible instability of the control system caused by randomly generated initial values of the controller parameters and the lack of state safety guarantees in the first iterations of the learning process due to (i) and (ii): (i) the safety constraints are considered only in the DDPG-based training process of the controller, which is usually implemented as a neural network (NN); (ii) the initial values of the weights and the biases of the NN-based controller are initialized with randomly generated values. The proposed approach mitigates these drawbacks by initializing the parameters of the NN-based controller using SMA. The fitness function of the SMA-based initialization process is designed to incorporate state safety constraints into the search process, resulting in an initial NN-based controller with embedded state safety constraints. The proposed approach is compared to the classical one using real-time experimental results and performance indices popular for optimal reference tracking control problems and based on a state safety score.

Offline Reinforcement Learning for Adaptive Control in Manufacturing Processes: A Press Hardening Case Study

Abstract This paper explores the application of offline reinforcement learning in batch manufacturing, with a specific focus on press hardening processes. Offline reinforcement learning presents a viable alternative to traditional control and reinforcement learning methods, which often rely on impractical real-world interactions or complex simulations and iterative adjustments to bridge the gap between simulated and real-world environments. We demonstrate how offline reinforcement learning can improve control policies by leveraging existing data, thereby streamlining the training pipeline and reducing reliance on high-fidelity simulators. Our study evaluates the impact of varying data exploration rates by creating five datasets with exploration rates ranging from ε=0 to ε=0.8. Using the conservative Q-learning algorithm, we train and assess policies against both a dynamic baseline and a static industry-standard policy. The results indicate that while offline reinforcement learning effectively refines behavior policies and enhances supervised learning methods, its effectiveness is heavily dependent on the quality and exploratory nature of the initial behavior policy.

Intelligent multi-robot collaborative transport system

The research on multi-robot systems has been divided into various fields, such as communication, navigation, task allocation, and collaborative transport. While significant progress has been made in each area, there has been limited research integrating these fields to build a fully autonomous multi-robot collaborative transport system. Therefore, we identify the key issues and propose a multi-robot collaborative transport system founded on ROS1 and conduct validation in a simulated environment, laying a solid foundation for the system to run on real robots. The primary contributions of this study include three key areas: (1) modeling and validating robot collaborative transport, (2) developing a visual task allocation system leveraging FastDDS service, and (3) resolving path collision issues in multi-robot navigation through both traditional methods and reinforcement learning techniques. Extensive experimental evaluations demonstrate that the proposed intelligent multi-robot collaborative transport system can autonomously navigate to target points for collaborative transport task. Performance assessments, based on the error between the target point and the object’s arrival point as well as the transport trajectory error, reveal that the system effectively completes the assigned tasks.

Research on path planning in UAV-assisted emergency communication

The rapid development of UAV communication technology makes it have application potential in wireless systems. However, for the optimization problem of UAV base station providing communication for ground personnel in the disaster area under special natural disasters, traditional deep reinforcement learning is used. Algorithms cannot solve such problems well. This article proposes the AD3QN algorithm combined with the attention mechanism, which can communicate with disaster victims on the ground better and more quickly, providing better information collection for rescue missions and completing the task of communication optimization. In the mission simulation environment, in order to make the environment more realistic, we innovatively designed the ground user distribution model, air-to-ground communication model, and electromagnetic interference model. Finally, the effect of the proposed algorithm is evaluated by comparing different deep reinforcement learning algorithms. The results show that the algorithm we proposed can provide communication services faster and has higher practicability in terms of algorithm.

CoSD: Balancing behavioral consistency and diversity in unsupervised skill discovery

In hierarchical reinforcement learning, unsupervised skill discovery holds promise for overcoming the challenge of sparse rewards commonly encountered in traditional reinforcement learning. However, previous unsupervised skill discovery methods have excelled at maximizing intrinsic rewards but have overly prioritized skill diversity. Unrestrained pursuit of diversity leads skills to concentrate attention on unexplored domains, overlooking the internal consistency of skills themselves, resulting in the state visit distribution of individual skills lacking concentration. To address this problem, the Constrained Skill Discovery (CoSD) algorithm is proposed to balance the diversity and behavioral consistency of skills. CoSD integrates both the forward and backward decomposition forms of mutual information and uses the maximum entropy policy to maximize the information-theoretic objective of skill learning while requiring that each skill maintain low state entropy internally, which enhances the behavioral consistency of the skills while pursuing the diversity of the skills and ensures that the learned skills have a high degree of stability. Experimental results demonstrated that, compared to other skill discovery methods based on mutual information, skills from CoSD exhibit a more concentrated state visit distribution, indicating higher behavioral consistency and stability. In some complex downstream tasks, these skills with higher behavioral consistency exhibit superior performance.

Adaptive scheduling for Internet of Vehicles using deconfounded graph transfer learning

With the rapid development of Internet of Things technology, the Internet of Vehicles has emerged as a significant application. The Internet of Vehicles connects vehicles with traffic infrastructure and other vehicles, enabling real-time information sharing and vehicle dispatching. However, in the Internet of Vehicles environment, traffic scheduling systems face challenges such as sparse raw data and environmental confounding bias, which limit the effectiveness of traditional graph-based reinforcement learning methods. In this paper, we propose a Cross-City Federated Continual Learning Framework for Spatiotemporal Graph Transfer Learning called CCFTL, which removes confounding effects on reinforcement learning and enhances few-shot transfer learning in data-scarce cities. Our approach aims to significantly improve the adaptability and scalability of cross-city vehicle networking technologies in ever-changing environments. This study acts as an innovative research of the deconfounding strategy with spatiotemporal graph meta-knowledge learning, which enables optimization of cross-city meta-knowledge transfer through a federated continual learning approach. This approach effectively reduces learning bias caused by regional differences, which enhances the model’s generalization and adaptability to complex, heterogeneous urban scenarios. Experimental results show that our framework significantly outperforms traditional methods, especially in improving the efficiency and accuracy of traffic dispatch systems in Internet of Vehicles environments with scarce data and the presence of confounding factors.

Leveraging large language models for comprehensive locomotion control in humanoid robots design

This paper investigates the utilization of large language models (LLMs) for the comprehensive control of humanoid robot locomotion. Traditional reinforcement learning (RL) approaches for robot locomotion are resource-intensive and rely heavily on manually designed reward functions. To address these challenges, we propose a method that employs LLMs as the primary designer to handle key aspects of locomotion control, such as trajectory planning, inverse kinematics solving, and reward function design. By using user-provided prompts, LLMs generate and optimize code, reducing the need for manual intervention. Our approach was validated through simulations in Unity, demonstrating that LLMs can achieve human-level performance in humanoid robot control. The results indicate that LLMs can simplify and enhance the development of advanced locomotion control systems for humanoid robots.

Noisy Dueling Double Deep Q-Network algorithm for autonomous underwater vehicle path planning.

How to improve the success rate of autonomous underwater vehicle (AUV) path planning and reduce travel time as much as possible is a very challenging and crucial problem in the practical applications of AUV in the complex ocean current environment. Traditional reinforcement learning algorithms lack exploration of the environment, and the strategies learned by the agent may not generalize well to other different environments. To address these challenges, we propose a novel AUV path planning algorithm named the Noisy Dueling Double Deep Q-Network (ND3QN) algorithm by modifying the reward function and introducing a noisy network, which generalizes the traditional D3QN algorithm. Compared with the classical algorithm [e.g., Rapidly-exploring Random Trees Star (RRT*), DQN, and D3QN], with simulation experiments conducted in realistic terrain and ocean currents, the proposed ND3QN algorithm demonstrates the outstanding characteristics of a higher success rate of AUV path planning, shorter travel time, and smoother paths.

A comprehensive review on safe reinforcement learning for autonomous vehicle control in dynamic environments

To operate safely in a dynamic environment, autonomous vehicles must possess the same level of predictive driving abilities as human drivers and must be capable of anticipating the future actions of other dynamic objects in the environment, especially those of neighboring vehicles. The development of safe autonomous vehicles (AVs) poses a challenging task as it requires algorithms that can make real-time decisions in unpredictable circumstances. Reinforcement learning (RL) presents a promising approach for AV control, as it utilizes trial and error to enable optimal decision-making. However, traditional RL algorithms are unsuitable for safety-critical applications, as they may explore unsafe actions, potentially resulting in accidents. Safe reinforcement learning (SRL) algorithms have been developed to address this issue, prioritizing safe and reliable decisions. These algorithms incorporate constraints to prevent unsafe actions or utilize techniques to estimate action risk and avoid actions deemed excessively risky. Despite computational challenges, SRL holds significant promise for AV control, and is likely to play a crucial role in developing safe and reliable systems. SRL methods are critical for the general adoption of autonomous vehicles by guaranteeing their safety and reliability. These algorithms utilize methods like uncertainty and risk estimation along with penalty functions, to avoid excessively risky actions and have the potential to significantly reduce accidents and build public trust in autonomous driving. However, there are challenges that need to be addressed, such as the dynamic nature of real-world traffic, high computational costs, and the diversity of road design; and these varying conditions make the designing, testing, and validating of SRL algorithms difficult. Despite these challenges, SRL presents a promising solution, through integrating new sensing technologies and machine learning techniques, to develop safe, efficient, and environmentally friendly transportation systems.

Optimized distributed formation control using identifier–critic–actor reinforcement learning for a class of stochastic nonlinear multi-agent systems

This article is to propose an adaptive reinforcement learning (RL)-based optimized distributed formation control for the unknown stochastic nonlinear single-integrator dynamic multi-agent system (MAS). For solving the issue of unknown dynamic, an adaptive identifier neural network (NN) is developed to determine the stochastic MAS under expectation sense. And then, for deriving the optimized formation control, the RL is putted into effect via constructing a pair of critic and actor NNs. With regard of the traditional RL optimal controls, their algorithm exists the inherent complexity, because their adaptive RL algorithm are derived from negative gradient of the square of Hamilton–Jacobi–Bellman (HJB) equation. As a result, these methods are difficultly extended to stochastic dynamical systems. However, since this adaptive RL laws are derived from a simple positive function rather than the square of HJB equation, it can make optimal control with simple algorithm. Therefore, this optimized formation scheme can be smoothly performed to the stochastic MAS. Finally, according to theorem proof and computer simulation, the optimized method can realize the required control objective.

Improved ACO algorithm fused with improved Q-Learning algorithm for Bessel curve global path planning of search and rescue robots

Addressing issues with traditional ant colony and reinforcement learning algorithms, such as low search efficiency and the tendency to produce insufficiently smooth paths that easily fall into local optima, this paper designs an improved ant colony optimization algorithm fusion with improved Q-Learning (IAC-IQL) algorithm for Bessel curve global path planning of search and rescue (SAR) robots. First, the heuristic function model in the ant colony algorithm is improved, the elite ant search strategy and the adaptive pheromone volatility factor strategy are introduced, and the initial path is searched in realize the motion environment with the help of the improved ant colony algorithm, and the initialized pheromone matrix is constructed. Second, the improved ant colony algorithm and Q-Learning (QL) algorithm are fused by utilizing the similarity between the pheromone matrix in the improved ant colony algorithm and the Q-matrix in the QL algorithm. A heuristic learning evaluation model is designed to dynamically adjust the learning factor and provide guidance for the search path. Additionally, a dynamic adaptive greedy strategy is introduced to balance the exploration and exploitation of the robot in the environment. Finally, the paths are smoothed using third-order Bessel curves to eliminate the problem of excessive steering angles. Through three sets of comparative simulation experiments conducted in Pycharm platform, the effectiveness, superiority, and practicality of the IAC-IQL algorithm were verified. The experimental results demonstrated that the IAC-IQL algorithm integrates the strong search capability of ant colony algorithm and the self-learning characteristics of QL algorithm. SAR robots equipped with the improved IAC-IQL algorithm exhibit significantly enhanced iterative search efficiency in grid simulation environment and image sampling simulation environment. The global path optimization indicators demonstrate high efficiency, and the paths are smoother.

Safe dynamic optimization of automatic generation control via imitation-based reinforcement learning

IntroductionThe increasing penetration of distributed generation (e.g., solar power and wind power) in the energy market has caused unpredictable disturbances in power systems and accelerated the application of intelligent control, such as reinforcement learning (RL), in automatic generation control (AGC). However, traditional RL cannot ensure constraint safety during training and frequently violates the constraints (e.g., frequency limitations), further threatening the safety and stability of grid operation.MethodsTo address the safety issue, we propose a novel safe RL framework that combines expert experiences with the RL controller to achieve imitation-based RL. This method allows an initialized safe policy by imitating expert experiences to prevent random explorations at the beginning. Specifically, we first formulate the AGC problem mathematically as a Markov decision process. Then, the imitation mechanism is developed atop a soft actor–critic RL algorithm.Results and discussionFinally, numerical studies are conducted with an IEEE 39-bus network, which show that the proposed method satisfies the frequency control performance standard better and improves the RL training efficiency.

Reinforcement learning-based drone simulators: survey, practice, and challenge

Recently, machine learning has been very useful in solving diverse tasks with drones, such as autonomous navigation, visual surveillance, communication, disaster management, and agriculture. Among these machine learning, two representative paradigms have been widely utilized in such applications: supervised learning and reinforcement learning. Researchers prefer to use supervised learning, mostly based on convolutional neural networks, because of its robustness and ease of use but yet data labeling is laborious and time-consuming. On the other hand, when traditional reinforcement learning is combined with the deep neural network, it can be a very powerful tool to solve high-dimensional input problems such as image and video. Along with the fast development of reinforcement learning, many researchers utilize reinforcement learning in drone applications, and it often outperforms supervised learning. However, it usually requires the agent to explore the environment on a trial-and-error basis which is high cost and unrealistic in the real environment. Recent advances in simulated environments can allow an agent to learn by itself to overcome these drawbacks, although the gap between the real environment and the simulator has to be minimized in the end. In this sense, a realistic and reliable simulator is essential for reinforcement learning training. This paper investigates various drone simulators that work with diverse reinforcement learning architectures. The characteristics of the reinforcement learning-based drone simulators are analyzed and compared for the researchers who would like to employ them for their projects. Finally, we shed light on some challenges and potential directions for future drone simulators.