Traditional Reinforcement Learning Algorithms Research Articles

Noisy Dueling Double Deep Q-Network algorithm for autonomous underwater vehicle path planning.

How to improve the success rate of autonomous underwater vehicle (AUV) path planning and reduce travel time as much as possible is a very challenging and crucial problem in the practical applications of AUV in the complex ocean current environment. Traditional reinforcement learning algorithms lack exploration of the environment, and the strategies learned by the agent may not generalize well to other different environments. To address these challenges, we propose a novel AUV path planning algorithm named the Noisy Dueling Double Deep Q-Network (ND3QN) algorithm by modifying the reward function and introducing a noisy network, which generalizes the traditional D3QN algorithm. Compared with the classical algorithm [e.g., Rapidly-exploring Random Trees Star (RRT*), DQN, and D3QN], with simulation experiments conducted in realistic terrain and ocean currents, the proposed ND3QN algorithm demonstrates the outstanding characteristics of a higher success rate of AUV path planning, shorter travel time, and smoother paths.

A comprehensive review on safe reinforcement learning for autonomous vehicle control in dynamic environments

To operate safely in a dynamic environment, autonomous vehicles must possess the same level of predictive driving abilities as human drivers and must be capable of anticipating the future actions of other dynamic objects in the environment, especially those of neighboring vehicles. The development of safe autonomous vehicles (AVs) poses a challenging task as it requires algorithms that can make real-time decisions in unpredictable circumstances. Reinforcement learning (RL) presents a promising approach for AV control, as it utilizes trial and error to enable optimal decision-making. However, traditional RL algorithms are unsuitable for safety-critical applications, as they may explore unsafe actions, potentially resulting in accidents. Safe reinforcement learning (SRL) algorithms have been developed to address this issue, prioritizing safe and reliable decisions. These algorithms incorporate constraints to prevent unsafe actions or utilize techniques to estimate action risk and avoid actions deemed excessively risky. Despite computational challenges, SRL holds significant promise for AV control, and is likely to play a crucial role in developing safe and reliable systems. SRL methods are critical for the general adoption of autonomous vehicles by guaranteeing their safety and reliability. These algorithms utilize methods like uncertainty and risk estimation along with penalty functions, to avoid excessively risky actions and have the potential to significantly reduce accidents and build public trust in autonomous driving. However, there are challenges that need to be addressed, such as the dynamic nature of real-world traffic, high computational costs, and the diversity of road design; and these varying conditions make the designing, testing, and validating of SRL algorithms difficult. Despite these challenges, SRL presents a promising solution, through integrating new sensing technologies and machine learning techniques, to develop safe, efficient, and environmentally friendly transportation systems.

Improved ACO algorithm fused with improved Q-Learning algorithm for Bessel curve global path planning of search and rescue robots

Addressing issues with traditional ant colony and reinforcement learning algorithms, such as low search efficiency and the tendency to produce insufficiently smooth paths that easily fall into local optima, this paper designs an improved ant colony optimization algorithm fusion with improved Q-Learning (IAC-IQL) algorithm for Bessel curve global path planning of search and rescue (SAR) robots. First, the heuristic function model in the ant colony algorithm is improved, the elite ant search strategy and the adaptive pheromone volatility factor strategy are introduced, and the initial path is searched in realize the motion environment with the help of the improved ant colony algorithm, and the initialized pheromone matrix is constructed. Second, the improved ant colony algorithm and Q-Learning (QL) algorithm are fused by utilizing the similarity between the pheromone matrix in the improved ant colony algorithm and the Q-matrix in the QL algorithm. A heuristic learning evaluation model is designed to dynamically adjust the learning factor and provide guidance for the search path. Additionally, a dynamic adaptive greedy strategy is introduced to balance the exploration and exploitation of the robot in the environment. Finally, the paths are smoothed using third-order Bessel curves to eliminate the problem of excessive steering angles. Through three sets of comparative simulation experiments conducted in Pycharm platform, the effectiveness, superiority, and practicality of the IAC-IQL algorithm were verified. The experimental results demonstrated that the IAC-IQL algorithm integrates the strong search capability of ant colony algorithm and the self-learning characteristics of QL algorithm. SAR robots equipped with the improved IAC-IQL algorithm exhibit significantly enhanced iterative search efficiency in grid simulation environment and image sampling simulation environment. The global path optimization indicators demonstrate high efficiency, and the paths are smoother.

A train trajectory optimization method based on the safety reinforcement learning with a relaxed dynamic reward

Train trajectory optimization (TTO) is an effective way to address energy consumption in rail transit. Reinforcement learning (RL), an excellent optimization method, has been used to solve TTO problems. Although traditional RL algorithms use penalty functions to restrict the random exploration behavior of agents, they cannot fully guarantee the safety of the process and results. This paper proposes a proximal policy optimization based safety reinforcement learning framework (S-PPO) for the train trajectory optimization, including a safe action rechoosing mechanism (SARM) and a relaxed dynamic reward mechanism (RDRM) combining a relaxed sparse reward and a dynamic dense reward. SARM guarantees that the new states generated by the agent consistently adhere to the environmental security constraints, thereby enhancing sampling efficiency and facilitating algorithm convergence. RDRM makes it easier for agents to obtain successful samples by relaxing time constraints, which also offers a better balance between exploration and exploitation. The experimental results show that S-PPO can significantly improve performance and obtain better train operation trajectories than soft constraint methods, and the convergence process is smoother. Finally, it was demonstrated that S-PPO exhibits good adaptability across various speed limit tracks.

A Multiproject and Multilevel Plan Management Model Based on a Hybrid Program Evaluation and Review Technique and Reinforcement Learning Mechanism

It is very difficult for manufacturing enterprises to achieve automatic coordination of multiproject and multilevel planning when they are unable to make large-scale resource adjustments. In addition, planning and coordination work mostly relies on human experience, and inaccurate planning often occurs. This article innovatively proposes the PERT-RP-DDPGAO algorithm, which effectively combines the program evaluation and review technique (PERT) and deep deterministic policy gradient (DDPG) technology. Innovatively using matrix computing, the resource plan (RP) itself is used for the first time as an intelligent agent for reinforcement learning, achieving automatic coordination of multilevel plans. Through experiments, this algorithm can achieve automatic planning and has interpretability in management theory. To solve the problem of continuous control, the second half of the new algorithm adopts the DDPG algorithm, which has advantages in convergence and response speed compared to traditional reinforcement learning algorithms and heuristic algorithms. The response time of this algorithm is 3.0% lower than the traditional deep Q-network (DQN) algorithm and more than 8.4% shorter than the heuristic algorithm.

Mars Exploration: Research on Goal-Driven Hierarchical DQN Autonomous Scene Exploration Algorithm

In the non-deterministic, large-scale navigation environment under the Mars exploration mission, there is a large space for action and many environmental states. Traditional reinforcement learning algorithms that can only obtain rewards at target points and obstacles will encounter the problems of reward sparsity and dimension explosion, making the training speed too slow or even impossible. This work proposes a deep layered learning algorithm based on the goal-driven layered deep Q-network (GDH-DQN), which is more suitable for mobile robots to explore, navigate, and avoid obstacles without a map. The algorithm model is designed in two layers. The lower layer provides behavioral strategies to achieve short-term goals, and the upper layer provides selection strategies for multiple short-term goals. Use known position nodes as short-term goals to guide the mobile robot forward and achieve long-term obstacle avoidance goals. Hierarchical execution not only simplifies tasks but also effectively solves the problems of reward sparsity and dimensionality explosion. In addition, each layer of the algorithm integrates a Hindsight Experience Replay mechanism to improve performance, make full use of the goal-driven function of the node, and effectively avoid the possibility of misleading the agent by complex processes and reward function design blind spots. The agent adjusts the number of model layers according to the number of short-term goals, further improving the efficiency and adaptability of the algorithm. Experimental results show that, compared with the hierarchical DQN method, the navigation success rate of the GDH-DQN algorithm is significantly improved, and it is more suitable for unknown scenarios such as Mars exploration.

Optimization Effectiveness of Multi-Intelligence Consistent Energy System Based on Digital Advertising and Data Analysis

This paper introduces a novel approach to address the issue of overestimation of state-action values in reinforcement learning algorithms based on the Q-framework. It integrates a dual-layer Q-learning algorithm with a grey wolf intelligent optimization method, enabling rapid search for optimal allocations in unknown search spaces. This integration results in the development of a multi-agent collaborative Area-Grid Coordination (A-GC) strategy, termed the grey wolf double Q (GWDQ) strategy, tailored for multi-area energy interconnection scenarios. The proposed GWDQ strategy is evaluated through simulation experiments on comprehensive energy system models, including mixed gas turbine systems, Combined Cooling, Heating, and Power (CCHP) systems, and a multi-area energy-interconnected Northeast power grid model. A centralized architecture is established to analyze the optimization effects of digital advertising. The performance of the GWDQ strategy is compared with traditional reinforcement learning algorithms through simulation and empirical data validation. Results indicate that the GWDQ strategy exhibits stronger learning capabilities, improved stability, and enhanced control performance compared to traditional methods. It demonstrates superior optimization for digital advertising and enables swift acquisition of optimal coordination in A-GC processes across multiple regions. Additionally, the paper analyzes the environmental and economic impacts of the proposed strategy.

Optimization of Single-user Task Migration based on Improved DDPG

Aiming at the problems of slow convergence and unstable convergence of traditional reinforcement learning algorithms in minimizing computational cost on edge servers with random task arrivals and time-varying wireless channels, an improved DDPG algorithm (IDDPG) was proposed. The Critic network structure of DDPG was replaced by the Dueling structure, which converged faster by splitting the state value function into an advantage function and a value function. The update frequency of the Critic network was adjusted to be higher than that of the Actor-network to make the overall training more stable. The Ornstein- Uhlenbeck noise was added to the actions selected through the Actor-network to improve the algorithm exploration ability, and the action noise size was set in segments to ensure the stability of convergence. Experimental results show that, compared with other algorithms, the IDDPG algorithm can better minimize the computational cost and has a certain improvement in the convergence speed and convergence stability.

An obstacle avoidance method for robotic arm based on reinforcement learning

Purpose Robotic arms play a crucial role in various industrial operations, such as sorting, assembly, handling and spraying. However, traditional robotic arm control algorithms often struggle to adapt when faced with the challenge of dynamic obstacles. This paper aims to propose a dynamic obstacle avoidance method based on reinforcement learning to address real-time processing of dynamic obstacles. Design/methodology/approach This paper introduces an innovative method that introduces a feature extraction network that integrates gating mechanisms on the basis of traditional reinforcement learning algorithms. Additionally, an adaptive dynamic reward mechanism is designed to optimize the obstacle avoidance strategy. Findings Validation through the CoppeliaSim simulation environment and on-site testing has demonstrated the method's capability to effectively evade randomly moving obstacles, with a significant improvement in the convergence speed compared to traditional algorithms. Originality/value The proposed dynamic obstacle avoidance method based on Reinforcement Learning not only accomplishes the task of dynamic obstacle avoidance efficiently but also offers a distinct advantage in terms of convergence speed. This approach provides a novel solution to the obstacle avoidance methods for robotic arms.

Federated deep reinforcement learning for task offloading and resource allocation in mobile edge computing-assisted vehicular networks

Mobile edge computing (MEC) enables computation intensive applications in the Internet of Vehicles (IoV) to no longer be limited by device resources. However, the lack of an effective task scheduling strategy will seriously affect users’ quality of experience (QoE). In this paper, a task type-based task offloading and resource allocation strategy is proposed to reduce delay and energy consumption during task execution. First, we establish communication, computing, and system cost models based on task offloading schemes, and model the joint optimization problem of task offloading and resource allocation as a Markov decision process. The utility function is obtained based on the task completion rate and the system cost. Second, an algorithm framework based on multi-agent deep deterministic policy gradient (MADDPG) is designed to solve the difficulty that traditional single-agent reinforcement learning algorithms are difficult to converge in a dynamic environment. In distributed scenarios, the proposed framework can also reduce system costs while handling more tasks. Finally, federated learning is introduced in the training process to reduce the impact of non-IID data while protecting privacy. Simulation results show that the proposed algorithm can effectively improve system processing efficiency and reduce device energy consumption compared to the popular reinforcement learning algorithms.

A review of research on reinforcement learning algorithms for multi-agents

In recent years, multi-agent reinforcement learning techniques have been widely used and evolved in the field of artificial intelligence. However, traditional reinforcement learning methods have limitations such as long training time, large sample data requirements, and highly delayed rewards. Therefore, this paper systematically and specifically studies the MARL algorithm. Firstly, this paper uses Citespace software to visually analyze the existing literature on multi-agent reinforcement learning and briefly indicates the research hotspots and key research directions in this field. Secondly, the applications of traditional reinforcement learning algorithms under two task objects, namely single-agent and multi-agent systems, are described in detail. Then, the paper highlights the diverse applications, challenges, and corresponding solutions of MARL algorithmic techniques in the field of MAS. Finally, the paper points out future research directions based on the existing limitations of the algorithm. Through this paper, readers will gain a systematic and in-depth understanding of MARL algorithms and how they can be utilized to better address the various challenges posed by MAS.

An Improved Dyna-Q Algorithm Inspired by the Forward Prediction Mechanism in the Rat Brain for Mobile Robot Path Planning.

The traditional Model-Based Reinforcement Learning (MBRL) algorithm has high computational cost, poor convergence, and poor performance in robot spatial cognition and navigation tasks, and it cannot fully explain the ability of animals to quickly adapt to environmental changes and learn a variety of complex tasks. Studies have shown that vicarious trial and error (VTE) and the hippocampus forward prediction mechanism in rats and other mammals can be used as key components of action selection in MBRL to support "goal-oriented" behavior. Therefore, we propose an improved Dyna-Q algorithm inspired by the forward prediction mechanism of the hippocampus to solve the above problems and tackle the exploration-exploitation dilemma of Reinforcement Learning (RL). This algorithm alternately presents the potential path in the future for mobile robots and dynamically adjusts the sweep length according to the decision certainty, so as to determine action selection. We test the performance of the algorithm in a two-dimensional maze environment with static and dynamic obstacles, respectively. Compared with classic RL algorithms like State-Action-Reward-State-Action (SARSA) and Dyna-Q, the algorithm can speed up spatial cognition and improve the global search ability of path planning. In addition, our method reflects key features of how the brain organizes MBRL to effectively solve difficult tasks such as navigation, and it provides a new idea for spatial cognitive tasks from a biological perspective.

Evaluating the stealth of reinforcement learning-based cyber attacks against unknown scenarios using knowledge transfer techniques

Organizations are vulnerable to cyber attacks as they rely on computer networks and the internet for communication and data storage. While Reinforcement Learning (RL) is a widely used strategy to simulate and learn from these attacks, RL-guided offensives against unknown scenarios often lead to early exposure due to low stealth resulting from mistakes during the training phase. To address this issue, this work evaluates if the use of Knowledge Transfer Techniques (KTT), such as Transfer Learning and Imitation Learning, reduces the probability of early exposure by smoothing mistakes during training. This study developed a laboratory platform and a method to compare RL-based cyber attacks using KTT for unknown scenarios. The experiments simulated 2 unknown scenarios using 4 traditional RL algorithms and 4 KTT. In the results, although some algorithms using KTT obtained superior results, they were not so significant for stealth during the initial epochs of training. Nevertheless, experiments also revealed that throughout the entire learning cycle, Trust Region Policy Optimization (TRPO) is a promising algorithm for conducting cyber offensives based on Reinforcement Learning.

Energy Management Strategy Based on Reinforcement Learning and Frequency Decoupling for Fuel Cell Hybrid Powertrain

This study presents a Two-Layer Deep Deterministic Policy Gradient (TL-DDPG) energy management strategy for Hydrogen fuel cell hybrid train, that aims to solve the problem that traditional reinforcement learning strategies require high initial values and are difficult to optimize global variables. Augmenting the optimization capabilities of the inner layer, a frequency decoupling algorithm integrates into the outer layer, furnishing a fitting initial value for strategy optimization. This addition aims to bolster the stability of fuel cell output, thereby enhancing the overall efficiency of the hybrid power system. In comparison with the traditional reinforcement learning algorithm, the proposed approach demonstrates notable improvements: a reduction in hydrogen consumption per 100 km by 16.3 kg, a 9.7% increase in the output power stability of the fuel cell, and a 1.8% enhancement in its efficiency.

Advancing spacecraft rendezvous and docking through safety reinforcement learning and ubiquitous learning principles

As spacecraft rendezvous and docking missions become increasingly complex, the need for advanced solutions has surged. In recent years, the application of reinforcement learning techniques to tackle spacecraft rendezvous guidance challenges has emerged as a prominent international trend. Vital to ensuring the secure rendezvous and docking of spacecraft is the task of obstacle avoidance. However, traditional reinforcement learning algorithms lack the ability to enforce safety constraints within the exploration space, which presents a formidable obstacle in the design of spacecraft rendezvous guidance strategies. In response to this challenge, a spacecraft rendezvous guidance methodology founded on safety reinforcement learning is proposed. Firstly, a Markov model is crafted for autonomous spacecraft rendezvous in scenarios involving collision avoidance. A reward system, contingent on obstacle warnings and collision avoidance constraints, is introduced to establish a safety reinforcement learning framework for devising spacecraft rendezvous guidance strategies. Secondly, within the framework of safety reinforcement learning, two deep reinforcement learning (DRL) algorithms, Proximal Policy Optimisation (PPO) and Deep Deterministic Policy Gradient (DDPG), are leveraged to generate these guidance strategies. Experimental findings validate the effectiveness of this approach in successfully executing obstacle avoidance and achieving rendezvous with remarkable precision. Furthermore, through an analysis of the performance and generalization capabilities of these two algorithms, the efficacy of the proposed methodology is further underscored. This fusion of advanced space guidance technology with the principles of Ubiquitous Learning marks a significant step forward in the quest for safer and more efficient spacecraft rendezvous and docking operations.

Learning strategies for underwater robot autonomous manipulation control

Autonomous manipulation operations represent the high intelligent coordination from robotic vision and control, it is also a symbol of the advances of robotic intelligence. The limitations of visual sensing and the increasingly complex experimental conditions make autonomous manipulation operations more difficult, particularly for deep reinforcement learning methods, which can enhance robotic control intelligence but require a lot of training process. Due to the high-dimensional continuous state space and continuous action space characteristics of underwater operations, this paper adopts a policy-based reinforcement learning method as the foundational approach. To address the issues of instability and low convergence efficiency in traditional policy-based reinforcement learning algorithms during the learning process, this paper proposes a novel policy learning method. This method adopts the Proximal Policy Optimization algorithm (PPOClip) and optimizes it through an actor-critic network. The aim is to improve the stability and effectiveness of convergence in the learning process. In the underwater training environment, a new reward shaping scheme has been designed to address the issue of reward sparsity during the training process. The manually crafted dense reward function is utilized as attractive and repulsive potential functions for goal manipulation and obstacle avoidance. On the highly complex underwater manipulation and training environment, transferred learning algorithm has been established to reduce the training times and compensate the differences between the simulation and experiment. Simulations and tank experiments have verified the performance of the proposed strategy learning method.

Duty Cycle Scheduling in Wireless Sensor Networks Using an Exploratory Strategy-Directed MADDPG Algorithm

This paper presents an in-depth study of the application of Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithms with an exploratory strategy for duty cycle scheduling (DCS) in the wireless sensor networks (WSNs). The focus is on optimizing the performance of sensor nodes in terms of energy efficiency and event detection rates under varying environmental conditions. Through a series of simulations, we investigate the impact of key parameters such as the sensor specificity constant α and the Poisson rate of events on the learning and operational efficacy of sensor nodes. Our results demonstrate that the MADDPG algorithm with an exploratory strategy outperforms traditional reinforcement learning algorithms, particularly in environments characterized by high event rates and the need for precise energy management. The exploratory strategy enables a more effective balance between exploration and exploitation, leading to improved policy learning and adaptation in dynamic and uncertain environments. Furthermore, we explore the sensitivity of different algorithms to the tuning of the sensor specificity constant α, revealing that lower values generally yield better performance by reducing energy consumption without significantly compromising event detection. The study also examines the algorithms' robustness against the variability introduced by different event Poisson rates, emphasizing the importance of algorithm selection and parameter tuning in practical WSN applications. The insights gained from this research provide valuable guidelines for the deployment of sensor networks in real-world scenarios, where the trade-off between energy consumption and event detection is critical. Our findings suggest that the integration of exploratory strategies in MADDPG algorithms can significantly enhance the performance and reliability of sensor nodes in WSNs.

Optimization of Image Transmission in Cooperative Semantic Communication Networks

In this paper, a semantic communication framework for image data transmission is developed. In the investigated framework, a set of servers cooperatively transmit image data to a set of users utilizing semantic communication techniques, which enable servers to transmit only the semantic information that accurately captures the meaning of images. To evaluate the performance of studied semantic communication system, a multimodal metric called image-to-graph semantic similarity (ISS) is proposed to measure the correlation between the extracted semantic information and the original image. To meet the ISS requirement of each user, each server must jointly determine the semantic information to be transmitted and the resource blocks (RBs) used for semantic information transmission. Due to the co-channel interference among users associated with different servers, each server must cooperate with other servers to find a globally optimal semantic oriented RB allocation. We formulate this problem as an optimization problem whose goal is to minimize the sum of the average transmission latency of each server while reaching the ISS requirement. To solve this problem, we propose a value decomposition based entropy-maximized multi-agent reinforcement learning (RL) algorithm. The proposed algorithm enables each server to coordinate with other servers in training stage and execute RB allocation in a distributed manner to approach to a globally optimal performance with less training iterations. Compared to traditional multi-agent RL algorithms, the proposed RL framework improves the exploration of valuable action of servers and the probability of finding a globally optimal RB allocation policy based on local observation of wireless and semantic communication environments. Simulation results show that the proposed algorithm can reduce the transmission delay by up to 16.1% and improve the convergence speed by up to 100% compared to the traditional multi-agent RL algorithms.

Electric vehicle charging navigation strategy in coupled smart grid and transportation network: A hierarchical reinforcement learning approach

Most of the existing electric vehicle (EV) charging navigation methods do not simultaneously take into account the electric vehicle charging destination optimization and path planning. Moreover, they are unable to provide online real-time decision-making under a variety of uncertain factors. To address these problems, this paper first establishes a bilevel stochastic optimization model for EV charging navigation considering various uncertainties, and then proposes an EV charging navigation method based on the hierarchical enhanced deep Q network (HEDQN) to solve the above stochastic optimization model in real-time. The proposed HEDQN contains two enhanced deep Q networks, which are utilized to optimize the charging destination and charging route path of EVs, respectively. Finally, the proposed method is simulated and validated in two urban transportation networks. The simulation results demonstrate that compared with the Dijkstra shortest path algorithm, single-layer deep reinforcement learning algorithm, and traditional hierarchical deep reinforcement learning algorithm, the proposed HEDQN algorithm can effectively reduce the total charging cost of electric vehicles and realize online real-time charging navigation of electric vehicles, that shows excellent generalization ability and scalability.

Successful application of predictive information in deep reinforcement learning control: A case study based on an office building HVAC system

Reinforcement Learning (RL), a promising algorithm for the operational control of Heating, Ventilation, and Air Conditioning (HVAC) systems, has garnered considerable attention and applications. However, traditional RL algorithms typically do not incorporate predictive information for future scenarios, and only a limited number of studies have examined the enhancement and impact of predictive information on RL algorithms. To address the issue of coupling RL and predictive information in HVAC system operation optimization, we employed an open-source framework to examine the impact of various predictive information strategies on RL outcomes. We propose a joint gated recurrent unit (GRU)-RL algorithm to handle situations where a time-series exists in state space. The results from four classic test cases demonstrate that the proposed GRU-RL method can reduce operating costs by approximately 14.5% and increase comfort performance by 88.4% in indoor comfort control and cost-management tasks. Moreover, the GRU-RL method outperformed the conventional DRL method and was merely augmented with prediction information. In indoor temperature regulation, the GRU-RL algorithm improves control efficacy by 14.2% compared to models without predictive information and offers an approximately 5% improvement over traditional network models. Finally, all models were made open source for easy replication and further research.