Traditional Reinforcement Learning Research Articles

Noisy Dueling Double Deep Q-Network algorithm for autonomous underwater vehicle path planning.

How to improve the success rate of autonomous underwater vehicle (AUV) path planning and reduce travel time as much as possible is a very challenging and crucial problem in the practical applications of AUV in the complex ocean current environment. Traditional reinforcement learning algorithms lack exploration of the environment, and the strategies learned by the agent may not generalize well to other different environments. To address these challenges, we propose a novel AUV path planning algorithm named the Noisy Dueling Double Deep Q-Network (ND3QN) algorithm by modifying the reward function and introducing a noisy network, which generalizes the traditional D3QN algorithm. Compared with the classical algorithm [e.g., Rapidly-exploring Random Trees Star (RRT*), DQN, and D3QN], with simulation experiments conducted in realistic terrain and ocean currents, the proposed ND3QN algorithm demonstrates the outstanding characteristics of a higher success rate of AUV path planning, shorter travel time, and smoother paths.

A comprehensive review on safe reinforcement learning for autonomous vehicle control in dynamic environments

To operate safely in a dynamic environment, autonomous vehicles must possess the same level of predictive driving abilities as human drivers and must be capable of anticipating the future actions of other dynamic objects in the environment, especially those of neighboring vehicles. The development of safe autonomous vehicles (AVs) poses a challenging task as it requires algorithms that can make real-time decisions in unpredictable circumstances. Reinforcement learning (RL) presents a promising approach for AV control, as it utilizes trial and error to enable optimal decision-making. However, traditional RL algorithms are unsuitable for safety-critical applications, as they may explore unsafe actions, potentially resulting in accidents. Safe reinforcement learning (SRL) algorithms have been developed to address this issue, prioritizing safe and reliable decisions. These algorithms incorporate constraints to prevent unsafe actions or utilize techniques to estimate action risk and avoid actions deemed excessively risky. Despite computational challenges, SRL holds significant promise for AV control, and is likely to play a crucial role in developing safe and reliable systems. SRL methods are critical for the general adoption of autonomous vehicles by guaranteeing their safety and reliability. These algorithms utilize methods like uncertainty and risk estimation along with penalty functions, to avoid excessively risky actions and have the potential to significantly reduce accidents and build public trust in autonomous driving. However, there are challenges that need to be addressed, such as the dynamic nature of real-world traffic, high computational costs, and the diversity of road design; and these varying conditions make the designing, testing, and validating of SRL algorithms difficult. Despite these challenges, SRL presents a promising solution, through integrating new sensing technologies and machine learning techniques, to develop safe, efficient, and environmentally friendly transportation systems.

Leveraging large language models for comprehensive locomotion control in humanoid robots design

This paper investigates the utilization of large language models (LLMs) for the comprehensive control of humanoid robot locomotion. Traditional reinforcement learning (RL) approaches for robot locomotion are resource-intensive and rely heavily on manually designed reward functions. To address these challenges, we propose a method that employs LLMs as the primary designer to handle key aspects of locomotion control, such as trajectory planning, inverse kinematics solving, and reward function design. By using user-provided prompts, LLMs generate and optimize code, reducing the need for manual intervention. Our approach was validated through simulations in Unity, demonstrating that LLMs can achieve human-level performance in humanoid robot control. The results indicate that LLMs can simplify and enhance the development of advanced locomotion control systems for humanoid robots.

Improved ACO algorithm fused with improved Q-Learning algorithm for Bessel curve global path planning of search and rescue robots

Addressing issues with traditional ant colony and reinforcement learning algorithms, such as low search efficiency and the tendency to produce insufficiently smooth paths that easily fall into local optima, this paper designs an improved ant colony optimization algorithm fusion with improved Q-Learning (IAC-IQL) algorithm for Bessel curve global path planning of search and rescue (SAR) robots. First, the heuristic function model in the ant colony algorithm is improved, the elite ant search strategy and the adaptive pheromone volatility factor strategy are introduced, and the initial path is searched in realize the motion environment with the help of the improved ant colony algorithm, and the initialized pheromone matrix is constructed. Second, the improved ant colony algorithm and Q-Learning (QL) algorithm are fused by utilizing the similarity between the pheromone matrix in the improved ant colony algorithm and the Q-matrix in the QL algorithm. A heuristic learning evaluation model is designed to dynamically adjust the learning factor and provide guidance for the search path. Additionally, a dynamic adaptive greedy strategy is introduced to balance the exploration and exploitation of the robot in the environment. Finally, the paths are smoothed using third-order Bessel curves to eliminate the problem of excessive steering angles. Through three sets of comparative simulation experiments conducted in Pycharm platform, the effectiveness, superiority, and practicality of the IAC-IQL algorithm were verified. The experimental results demonstrated that the IAC-IQL algorithm integrates the strong search capability of ant colony algorithm and the self-learning characteristics of QL algorithm. SAR robots equipped with the improved IAC-IQL algorithm exhibit significantly enhanced iterative search efficiency in grid simulation environment and image sampling simulation environment. The global path optimization indicators demonstrate high efficiency, and the paths are smoother.

Safe dynamic optimization of automatic generation control via imitation-based reinforcement learning

IntroductionThe increasing penetration of distributed generation (e.g., solar power and wind power) in the energy market has caused unpredictable disturbances in power systems and accelerated the application of intelligent control, such as reinforcement learning (RL), in automatic generation control (AGC). However, traditional RL cannot ensure constraint safety during training and frequently violates the constraints (e.g., frequency limitations), further threatening the safety and stability of grid operation.MethodsTo address the safety issue, we propose a novel safe RL framework that combines expert experiences with the RL controller to achieve imitation-based RL. This method allows an initialized safe policy by imitating expert experiences to prevent random explorations at the beginning. Specifically, we first formulate the AGC problem mathematically as a Markov decision process. Then, the imitation mechanism is developed atop a soft actor–critic RL algorithm.Results and discussionFinally, numerical studies are conducted with an IEEE 39-bus network, which show that the proposed method satisfies the frequency control performance standard better and improves the RL training efficiency.

Reinforcement learning-based drone simulators: survey, practice, and challenge

Recently, machine learning has been very useful in solving diverse tasks with drones, such as autonomous navigation, visual surveillance, communication, disaster management, and agriculture. Among these machine learning, two representative paradigms have been widely utilized in such applications: supervised learning and reinforcement learning. Researchers prefer to use supervised learning, mostly based on convolutional neural networks, because of its robustness and ease of use but yet data labeling is laborious and time-consuming. On the other hand, when traditional reinforcement learning is combined with the deep neural network, it can be a very powerful tool to solve high-dimensional input problems such as image and video. Along with the fast development of reinforcement learning, many researchers utilize reinforcement learning in drone applications, and it often outperforms supervised learning. However, it usually requires the agent to explore the environment on a trial-and-error basis which is high cost and unrealistic in the real environment. Recent advances in simulated environments can allow an agent to learn by itself to overcome these drawbacks, although the gap between the real environment and the simulator has to be minimized in the end. In this sense, a realistic and reliable simulator is essential for reinforcement learning training. This paper investigates various drone simulators that work with diverse reinforcement learning architectures. The characteristics of the reinforcement learning-based drone simulators are analyzed and compared for the researchers who would like to employ them for their projects. Finally, we shed light on some challenges and potential directions for future drone simulators.

A train trajectory optimization method based on the safety reinforcement learning with a relaxed dynamic reward

Train trajectory optimization (TTO) is an effective way to address energy consumption in rail transit. Reinforcement learning (RL), an excellent optimization method, has been used to solve TTO problems. Although traditional RL algorithms use penalty functions to restrict the random exploration behavior of agents, they cannot fully guarantee the safety of the process and results. This paper proposes a proximal policy optimization based safety reinforcement learning framework (S-PPO) for the train trajectory optimization, including a safe action rechoosing mechanism (SARM) and a relaxed dynamic reward mechanism (RDRM) combining a relaxed sparse reward and a dynamic dense reward. SARM guarantees that the new states generated by the agent consistently adhere to the environmental security constraints, thereby enhancing sampling efficiency and facilitating algorithm convergence. RDRM makes it easier for agents to obtain successful samples by relaxing time constraints, which also offers a better balance between exploration and exploitation. The experimental results show that S-PPO can significantly improve performance and obtain better train operation trajectories than soft constraint methods, and the convergence process is smoother. Finally, it was demonstrated that S-PPO exhibits good adaptability across various speed limit tracks.

A Multiproject and Multilevel Plan Management Model Based on a Hybrid Program Evaluation and Review Technique and Reinforcement Learning Mechanism

It is very difficult for manufacturing enterprises to achieve automatic coordination of multiproject and multilevel planning when they are unable to make large-scale resource adjustments. In addition, planning and coordination work mostly relies on human experience, and inaccurate planning often occurs. This article innovatively proposes the PERT-RP-DDPGAO algorithm, which effectively combines the program evaluation and review technique (PERT) and deep deterministic policy gradient (DDPG) technology. Innovatively using matrix computing, the resource plan (RP) itself is used for the first time as an intelligent agent for reinforcement learning, achieving automatic coordination of multilevel plans. Through experiments, this algorithm can achieve automatic planning and has interpretability in management theory. To solve the problem of continuous control, the second half of the new algorithm adopts the DDPG algorithm, which has advantages in convergence and response speed compared to traditional reinforcement learning algorithms and heuristic algorithms. The response time of this algorithm is 3.0% lower than the traditional deep Q-network (DQN) algorithm and more than 8.4% shorter than the heuristic algorithm.

Mars Exploration: Research on Goal-Driven Hierarchical DQN Autonomous Scene Exploration Algorithm

In the non-deterministic, large-scale navigation environment under the Mars exploration mission, there is a large space for action and many environmental states. Traditional reinforcement learning algorithms that can only obtain rewards at target points and obstacles will encounter the problems of reward sparsity and dimension explosion, making the training speed too slow or even impossible. This work proposes a deep layered learning algorithm based on the goal-driven layered deep Q-network (GDH-DQN), which is more suitable for mobile robots to explore, navigate, and avoid obstacles without a map. The algorithm model is designed in two layers. The lower layer provides behavioral strategies to achieve short-term goals, and the upper layer provides selection strategies for multiple short-term goals. Use known position nodes as short-term goals to guide the mobile robot forward and achieve long-term obstacle avoidance goals. Hierarchical execution not only simplifies tasks but also effectively solves the problems of reward sparsity and dimensionality explosion. In addition, each layer of the algorithm integrates a Hindsight Experience Replay mechanism to improve performance, make full use of the goal-driven function of the node, and effectively avoid the possibility of misleading the agent by complex processes and reward function design blind spots. The agent adjusts the number of model layers according to the number of short-term goals, further improving the efficiency and adaptability of the algorithm. Experimental results show that, compared with the hierarchical DQN method, the navigation success rate of the GDH-DQN algorithm is significantly improved, and it is more suitable for unknown scenarios such as Mars exploration.

Optimization Effectiveness of Multi-Intelligence Consistent Energy System Based on Digital Advertising and Data Analysis

This paper introduces a novel approach to address the issue of overestimation of state-action values in reinforcement learning algorithms based on the Q-framework. It integrates a dual-layer Q-learning algorithm with a grey wolf intelligent optimization method, enabling rapid search for optimal allocations in unknown search spaces. This integration results in the development of a multi-agent collaborative Area-Grid Coordination (A-GC) strategy, termed the grey wolf double Q (GWDQ) strategy, tailored for multi-area energy interconnection scenarios. The proposed GWDQ strategy is evaluated through simulation experiments on comprehensive energy system models, including mixed gas turbine systems, Combined Cooling, Heating, and Power (CCHP) systems, and a multi-area energy-interconnected Northeast power grid model. A centralized architecture is established to analyze the optimization effects of digital advertising. The performance of the GWDQ strategy is compared with traditional reinforcement learning algorithms through simulation and empirical data validation. Results indicate that the GWDQ strategy exhibits stronger learning capabilities, improved stability, and enhanced control performance compared to traditional methods. It demonstrates superior optimization for digital advertising and enables swift acquisition of optimal coordination in A-GC processes across multiple regions. Additionally, the paper analyzes the environmental and economic impacts of the proposed strategy.

Endo-atmospheric maneuver penetration strategy based on generative adversarial reinforcement learning

An intelligent endo-atmospheric penetration strategy based on generative adversarial reinforcement learning is proposed in this manuscript. Firstly, attack and defense adversarial models are established, and missile maneuver penetration problem is transformed into an optimal control problem, considering penetration, handover position and mid-terminal guidance velocity constraints. Then, Radau Pseudospectral method is adopted to generate data samples considering random perturbations. Furthermore, Generative Adversarial Imitation Learning Combined with Deep Deterministic Policy Gradient method (GAIL-DDPG) is designed, with internal process reward signals constructed to tackle long-term sparse reward in missile manuver penetration problem. Finally, penetration strategy is trained and verified. Simulation shows that using generative adversarial reinforcement learning, with sample library to learn expert experience in training early stage, the proposed method can quickly converge. Also, performance is further optimized with reinforcement learning exploration strategy in the later stage of training. Simulation shows that the proposed method has better engineering application ability compared with traditional reinforcement learning method.

Optimization of swimming mode for elongated undulating fin using multi-agent deep deterministic policy gradient

Optimizing speed and propulsive efficiency are the most crucial survival skills for biomimetic robots. This paper investigates a swimming mode controller inspired by the black Knifefish to govern the fast-swimming gait with high propulsive efficiency for an elongated undulating fin robot. The proposed swimming mode controller is composed of a couple of Hopf oscillator-based central pattern generators (CPG) to generate the moving gait of robotic fish and a novel variant of Reinforcement Learning (RL) known as Multi-Agent Deep Deterministic Policy Gradient (MA-DDPG) for optimizing the propulsive efficiency. The proposed swimming controller facilitates the autonomous optimization of the oscillatory amplitude of the robotic fish to improve its propulsive efficiency. The proposed MA-DDPG demonstrates an aptitude for functioning within mixed cooperative-competitive environments. Furthermore, it effectively mitigates the drawback of zero amplitude in the updating process of conventional reinforcement learning (RL) methodologies. These findings highlight the potential utility of the MA-DDPG in optimizing the performance of multi-agent systems in dynamic, real-world scenarios. The simulation results show that the undulating fin robot reaches a maximum thrust of 0.9 N with a propulsive efficiency of 12.48 %, which is higher than that of traditional reinforcement learning methods.

Optimization of Single-user Task Migration based on Improved DDPG

Aiming at the problems of slow convergence and unstable convergence of traditional reinforcement learning algorithms in minimizing computational cost on edge servers with random task arrivals and time-varying wireless channels, an improved DDPG algorithm (IDDPG) was proposed. The Critic network structure of DDPG was replaced by the Dueling structure, which converged faster by splitting the state value function into an advantage function and a value function. The update frequency of the Critic network was adjusted to be higher than that of the Actor-network to make the overall training more stable. The Ornstein- Uhlenbeck noise was added to the actions selected through the Actor-network to improve the algorithm exploration ability, and the action noise size was set in segments to ensure the stability of convergence. Experimental results show that, compared with other algorithms, the IDDPG algorithm can better minimize the computational cost and has a certain improvement in the convergence speed and convergence stability.

An obstacle avoidance method for robotic arm based on reinforcement learning

Purpose Robotic arms play a crucial role in various industrial operations, such as sorting, assembly, handling and spraying. However, traditional robotic arm control algorithms often struggle to adapt when faced with the challenge of dynamic obstacles. This paper aims to propose a dynamic obstacle avoidance method based on reinforcement learning to address real-time processing of dynamic obstacles. Design/methodology/approach This paper introduces an innovative method that introduces a feature extraction network that integrates gating mechanisms on the basis of traditional reinforcement learning algorithms. Additionally, an adaptive dynamic reward mechanism is designed to optimize the obstacle avoidance strategy. Findings Validation through the CoppeliaSim simulation environment and on-site testing has demonstrated the method's capability to effectively evade randomly moving obstacles, with a significant improvement in the convergence speed compared to traditional algorithms. Originality/value The proposed dynamic obstacle avoidance method based on Reinforcement Learning not only accomplishes the task of dynamic obstacle avoidance efficiently but also offers a distinct advantage in terms of convergence speed. This approach provides a novel solution to the obstacle avoidance methods for robotic arms.

Generalization Enhancement of Visual Reinforcement Learning through Internal States.

Visual reinforcement learning is important in various practical applications, such as video games, robotic manipulation, and autonomous navigation. However, a major challenge in visual reinforcement learning is the generalization to unseen environments, that is, how agents manage environments with previously unseen backgrounds. This issue is triggered mainly by the high unpredictability inherent in high-dimensional observation space. To deal with this problem, techniques including domain randomization and data augmentation have been explored; nevertheless, these methods still cannot attain a satisfactory result. This paper proposes a new method named Internal States Simulation Auxiliary (ISSA), which uses internal states to improve generalization in visual reinforcement learning tasks. Our method contains two agents, a teacher agent and a student agent: the teacher agent has the ability to directly access the environment's internal states and is used to facilitate the student agent's training; the student agent receives initial guidance from the teacher agent and subsequently continues to learn independently. From another perspective, our method can be divided into two phases, the transfer learning phase and traditional visual reinforcement learning phase. In the first phase, the teacher agent interacts with environments and imparts knowledge to the vision-based student agent. With the guidance of the teacher agent, the student agent is able to discover more effective visual representations that address the high unpredictability of high-dimensional observation space. In the next phase, the student agent autonomously learns from the visual information in the environment, and ultimately, it becomes a vision-based reinforcement learning agent with enhanced generalization. The effectiveness of our method is evaluated using the DMControl Generalization Benchmark and the DrawerWorld with texture distortions. Preliminary results indicate that our method significantly improves generalization ability and performance in complex continuous control tasks.

Defining a Metric-Driven Approach for Learning Hazardous Situations

Artificial intelligence has brought many innovations to our lives. At the same time, it is worth designing robust safety machine learning (ML) algorithms to obtain more benefits from technology. Reinforcement learning (RL) being an important ML method is largely applied in safety-centric scenarios. In such a situation, learning safety constraints are necessary to avoid undesired outcomes. Within the traditional RL paradigm, agents typically focus on identifying states associated with high rewards to maximize its long-term returns. This prioritization can lead to a neglect of potentially hazardous situations. Particularly, the exploration phase can pose significant risks, as it necessitates actions that may have unpredictable consequences. For instance, in autonomous driving applications, an RL agent might discover routes that yield high efficiency but fail to account for sudden hazardous conditions such as sharp turns or pedestrian crossings, potentially leading to catastrophic failures. Ensuring the safety of agents operating in unpredictable environments with potentially catastrophic failure states remains a critical challenge. This paper introduces a novel metric-driven approach aimed at containing risk in RL applications. Central to this approach are two developed indicators: the Hazard Indicator and the Risk Indicator. These metrics are designed to evaluate the safety of an environment by quantifying the likelihood of transitioning from safe states to failure states and assessing the associated risks. The fact that these indicators are characterized by a straightforward implementation, a highly generalizable probabilistic mathematical foundation, and a domain-independent nature makes them particularly interesting. To demonstrate their efficacy, we conducted experiments across various use cases, showcasing the feasibility of our proposed metrics. By enabling RL agents to effectively manage hazardous states, this approach paves the way for a more reliable and readily implementable RL in practical applications.

Data-driven offline reinforcement learning approach for quadrotor’s motion and path planning

Non-learning based motion and path planning of an Unmanned Aerial Vehicle (UAV) is faced with low computation efficiency, mapping memory occupation and local optimization problems. This article investigates the challenge of quadrotor control using offline reinforcement learning. By establishing a data-driven learning paradigm that operates without real-environment interaction, the proposed workflow offers a safer approach than traditional reinforcement learning, making it particularly suited for UAV control in industrial scenarios. The introduced algorithm evaluates dataset uncertainty and employs a pessimistic estimation to foster offline deep reinforcement learning. Experiments highlight the algorithm’s superiority over traditional online reinforcement learning methods, especially when learning from offline datasets. Furthermore, the article emphasizes the importance of a more general behavior policy. In evaluations, the trained policy demonstrated versatility by adeptly navigating diverse obstacles, underscoring its real-world applicability.

Multi-UAV aided energy-aware transmissions in mmWave communication network: Action-branching QMIX network

Advancements in drone technology and high-frequency millimeter-wave communications are transforming unmanned-aerial-vehicles (UAV)-aided networks, expanding their potential across diverse applications. Despite the advantages of broad frequency bandwidth and enhanced line of sight connectivity in the UAV-aided millimeter-wave networks, it is challenging to provide high network performance because of the inherent limitations of limited UAV energy and millimeter-wave’s large path loss. This challenge becomes more important in dynamically changing multi-UAV environments. To address this challenge in multi-UAV networks, we propose a novel approach based on multi-agent deep reinforcement learning called action-branching QMIX. Our method determines nearly optimal codebook-based discrete beamforming vectors and UAV trajectories while maintaining a balance between communication efficiency and energy consumption. The proposed approach employs a new Long Short-Term Memory module to control long sequences effectively and enables it to adapt to changing environmental variables in real-time. We thoroughly evaluate the proposed control with a real-world measurement-based channel model. The evaluation confirms that the proposed control converges stably and consistently, and provides enhanced performance in terms of downlink data rate, success rate of reaching the destination, and service duration when compared to traditional benchmark multi-agent reinforcement learning schemes. These results emphasize the enhanced energy sustainability, robustness, and stability of the proposed approach in dynamically changing multi-UAV environments when compared to the existing benchmark algorithms.

Federated deep reinforcement learning for task offloading and resource allocation in mobile edge computing-assisted vehicular networks

Mobile edge computing (MEC) enables computation intensive applications in the Internet of Vehicles (IoV) to no longer be limited by device resources. However, the lack of an effective task scheduling strategy will seriously affect users’ quality of experience (QoE). In this paper, a task type-based task offloading and resource allocation strategy is proposed to reduce delay and energy consumption during task execution. First, we establish communication, computing, and system cost models based on task offloading schemes, and model the joint optimization problem of task offloading and resource allocation as a Markov decision process. The utility function is obtained based on the task completion rate and the system cost. Second, an algorithm framework based on multi-agent deep deterministic policy gradient (MADDPG) is designed to solve the difficulty that traditional single-agent reinforcement learning algorithms are difficult to converge in a dynamic environment. In distributed scenarios, the proposed framework can also reduce system costs while handling more tasks. Finally, federated learning is introduced in the training process to reduce the impact of non-IID data while protecting privacy. Simulation results show that the proposed algorithm can effectively improve system processing efficiency and reduce device energy consumption compared to the popular reinforcement learning algorithms.

Multi-UAV pursuit-evasion gaming based on PSO-M3DDPG schemes

The sample data for reinforcement learning algorithms often exhibit sparsity and instability, making the training results susceptible to falling into local optima. Mini-Max-Multi-agent Deep Deterministic Policy Gradient (M3DDPG) algorithm is a multi-agent reinforcement learning algorithm, which introduces the minimax theorem into Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. It also has unstable convergence caused by sparse sample data and randomization. However, the Particle Swarm Optimisation (PSO) algorithm, unlike traditional reinforcement learning methods, involves the construction of independent populations of policy networks to generate sample data, followed by training the reinforcement learning algorithm. PSO optimizes and updates the policy population based on a fitness function, aiming to enhance the efficiency and convergence speed of the algorithm in learning from the sample data. In order to address the multi-agent pursuit-evasion problem, we propose the PSO-M3DDPG algorithm, which combines the PSO algorithm with the M3DDPG algorithm. Through experimental simulations, the improved algorithm demonstrates superior training results and faster convergence speeds, thus validating its effectiveness.