Deterministic Policy Gradient Algorithm Research Articles

Deep Reinforcement Learning for Ecological and Distributed Urban Traffic Signal Control with Multi-Agent Equilibrium Decision Making

Multi-Agent Reinforcement Learning (MARL) has shown strong advantages in urban multi-intersection traffic signal control, but it also suffers from the problems of non-smooth environment and inter-agent coordination. However, most of the existing research on MARL traffic signal control has focused on designing efficient communication to solve the environment non-smoothness problem, while neglecting the coordination between agents. In order to coordinate among agents, this paper combines MARL and the regional mixed-strategy Nash equilibrium to construct a Deep Convolutional Nash Policy Gradient Traffic Signal Control (DCNPG-TSC) model, which enables agents to perceive the traffic environment in a wider range and achieves effective agent communication and collaboration. Additionally, a Multi-Agent Distributional Nash Policy Gradient (MADNPG) algorithm is proposed in this model, which is the first time the mixed-strategy Nash equilibrium is used for the improvement in the Multi-Agent Deep Deterministic Policy Gradient algorithm traffic signal control strategy to provide the optimal signal phase for each intersection. In addition, the eco-mobility concept is integrated into MARL traffic signal control to reduce pollutant emissions at intersections. Finally, simulation results in synthetic and real-world traffic road networks show that DCNPG-TSC outperforms other state-of-the-art MARL traffic signal control methods in almost all performance metrics, because it can aggregate the information of neighboring agents and optimize the agent’s decisions through gaming to find an optimal joint equilibrium strategy for the traffic road network.

Joint Phase Shift Design and Resource Management for a Non-Orthogonal Multiple Access-Enhanced Internet of Vehicle Assisted by an Intelligent Reflecting Surface-Equipped Unmanned Aerial Vehicle

This paper integrates intelligent reflecting surfaces (IRS) with unmanned aerial vehicles (UAV) to enhance the transmission performance of the Internet of Vehicles (IoV) through non-orthogonal multiple access (NOMA). It focuses on strengthening the signals from cell edge vehicles (CEVs) to the base station by optimizing the wireless propagation environment via an IRS-equipped UAV. The primary goal is to maximize the sum data rate of CEVs while satisfying the constraint of the successive interference cancellation (SIC) decoding threshold. The challenge lies in the non-convex nature of jointly considering the power control, subcarrier allocation, and phase shift design, making the problem difficult to optimally solve. To address this, the problem is decomposed into two independent subproblems, which are then solved iteratively. Specifically, the optimal phase shift design is achieved using the deep deterministic policy gradient (DDPG) algorithm. Furthermore, the graph theory is applied to determine the subcarrier allocation policy and derive a closed-form solution for optimal power control. Finally, the simulation results show that the proposed joint phase shift and resource management scheme significantly enhances the sum data rate compared to the state-of-the-art schemes, thereby demonstrating the benefits of integrating the IRS-equipped UAV into NOMA-enhanced IoV.

Deep reinforcement learning-based reactive trajectory planning method for UAVs

In order to improve the ability of avoiding dynamic threats during the flight of unmanned aerial vehicles (UAVs), a deep reinforcement learning-based reactive trajectory planning method is proposed in this paper. Firstly, a constrained Rapidly-exploring Random Tree-Connect algorithm (C-RRT-Connect) is proposed as the basic algorithm of reactive trajectory planning to globally plan for avoiding static obstacles in the environment. The C-RRT-Connect algorithm introduces the idea of target attraction to constrain the optimal growth point in the RRT-Connect algorithm. Then, based on the global trajectory, the local optimization is carried out according to the dynamic threats detected by the UAV during the flight. According to the real-time relative state between the UAV and the detected dynamic threat, the reaction sampling points and directional coefficients for avoiding the corresponding dynamic threat are generated online via the action network trained with the depth deterministic policy gradient algorithm (DDPG). And then the local trajectory is adjusted to modify the flight trajectory of the UAV to achieve reactive obstacle avoidance. The simulation experiment firstly compares the global trajectory planning performance of C-RRT-Connect and RRT-Connect in static environment, and secondly compares the local trajectory planning performance of DDPG algorithm and the artificial potential field method in dynamic environment. The experimental results show that in static environment, C-RRT-Connect algorithm has faster searching speed, less invalid samples and higher searching trajectory quality than RRT-Connect algorithm; In a dynamic environment, DDPG algorithm reduces the average running time by about 26% compared with the artificial potential field method, and has a stronger ability to evade dynamic threats in real time.

Distributed auto disturbances rejection resilient control of permanent magnetic maglev trains based on the optimized deep deterministic policy gradient algorithm

Distributed auto disturbances rejection resilient control of permanent magnetic maglev trains based on the optimized deep deterministic policy gradient algorithm

A Pilot Study of Observation Poisoning on Selective Reincarnation in Multi-Agent Reinforcement Learning

This research explores the vulnerability of selective reincarnation, a concept in Multi-Agent Reinforcement Learning (MARL), in response to observation poisoning attacks. Observation poisoning is an adversarial strategy that subtly manipulates an agent’s observation space, potentially leading to a misdirection in its learning process. The primary aim of this paper is to systematically evaluate the robustness of selective reincarnation in MARL systems against the subtle yet potentially debilitating effects of observation poisoning attacks. Through assessing how manipulated observation data influences MARL agents, we seek to highlight potential vulnerabilities and inform the development of more resilient MARL systems. Our experimental testbed was the widely used HalfCheetah environment, utilizing the Independent Deep Deterministic Policy Gradient algorithm within a cooperative MARL setting. We introduced a series of triggers, namely Gaussian noise addition, observation reversal, random shuffling, and scaling, into the teacher dataset of the MARL system provided to the reincarnating agents of HalfCheetah. Here, the “teacher dataset” refers to the stored experiences from previous training sessions used to accelerate the learning of reincarnating agents in MARL. This approach enabled the observation of these triggers’ significant impact on reincarnation decisions. Specifically, the reversal technique showed the most pronounced negative effect for maximum returns, with an average decrease of 38.08% in Kendall’s tau values across all the agent combinations. With random shuffling, Kendall’s tau values decreased by 17.66%. On the other hand, noise addition and scaling aligned with the original ranking by only 21.42% and 32.66%, respectively. The results, quantified by Kendall’s tau metric, indicate the fragility of the selective reincarnation process under adversarial observation poisoning. Our findings also reveal that vulnerability to observation poisoning varies significantly among different agent combinations, with some exhibiting markedly higher susceptibility than others. This investigation elucidates our understanding of selective reincarnation’s robustness against observation poisoning attacks, which is crucial for developing more secure MARL systems and also for making informed decisions about agent reincarnation.

Reinforcement learning-based order-dispatching optimization in the ride-sourcing service

Since the emergence of ride-sourcing, the dispatching problem has been a hot research issue in the travel market. Previous studies designed various refined algorithms to improve the dispatching efficiency. However, the heterogeneity of passenger cancellation behavior and driver's work patterns is ignored in the modeling of these studies, which may result in the algorithms failing to entirely capture the randomness in the real world, thereby affecting dispatching efficiency. To fill this research gap, this paper employs a discrete choice model and Gaussian mixture model to model passenger cancellation behavior and extract driver working patterns, respectively. Furthermore, a cooperative deep deterministic policy gradient (DDPG) algorithm is proposed for dispatching. The algorithm constructs decision networks separately for drivers with different working patterns to learn the optimal dispatching decision and designs a centralized neural network to coordinate the dispatching of all vehicles to maximize the global benefit. A price incentive strategy is proposed to improve passenger loyalty by providing incentive prices. Moreover, based on the constructed discrete choice model, a simulation environment is designed to describe the cancellation behavior in the real world and support the training of the algorithm. Experimental results based on real data in Beijing show that the cooperative DDPG algorithm can improve the order response rate by 2% to 6% and platform revenue by 3% to 9% in multiple travel periods while providing a better travel experience to passengers. Additionally, the price incentive strategy also effectively reduces passenger cancellation.

Dynamic resource matching in manufacturing using deep reinforcement learning

Matching plays an important role in the logical allocation of resources across a wide range of industries. The benefits of matching have been increasingly recognized in manufacturing industries. In particular, capacity sharing has received much attention recently. In this paper, we consider the problem of dynamically matching demand-capacity types of manufacturing resources. We formulate the multi-period, many-to-many manufacturing resource-matching problem as a sequential decision process. The formulated manufacturing resource-matching problem involves large state and action spaces, and it is not practical to accurately model the joint distribution of various types of demands. To address the curse of dimensionality and the difficulty of explicitly modeling the transition dynamics, we use a model-free deep reinforcement learning approach to find optimal matching policies. Moreover, to tackle the issue of infeasible actions and slow convergence due to initial biased estimates caused by the maximum operator in Q-learning, we introduce two penalties to the traditional Q-learning algorithm: a domain knowledge-based penalty based on a prior policy and an infeasibility penalty that conforms to the demand-supply constraints. We establish theoretical results on the convergence of our domain knowledge-informed Q-learning providing performance guarantee for small-size problems. For large-size problems, we further inject our modified approach into the deep deterministic policy gradient (DDPG) algorithm, which we refer to as domain knowledge-informed DDPG (DKDDPG). In our computational study, including small- and large-scale experiments, DKDDPG consistently outperformed traditional DDPG and other RL algorithms, yielding higher rewards and demonstrating greater efficiency in time and episodes.

Multi-agent reinforcement learning based computation offloading and resource allocation for LEO Satellite edge computing networks

Due to the limitations caused by geographical conditions and economic requirements, it is difficult to provide computing services by terrestrial networks for mobile terminals in remote areas. To address this issue, mobile edge computing (MEC) servers can be deployed in the low earth orbit (LEO) satellites to act as a complement and accommodate the unserved terminals. However, offloading computing tasks to servers in satellites may increase the energy consumption of ground terminals. Considering the limited battery capacity of ground terminals, how to perform the computation offloading and resource allocation are key challenges in the LEO satellite edge computing networks. Therefore, in this paper, we investigate the energy minimization problem for LEO satellite edge computing networks, where a multi-agent deep reinforcement learning algorithm with global rewards is proposed to optimize the transmit power, CPU frequency, bit allocation, offloading decision and bandwidth allocation via a decentralized method. Simulation results show that our proposed algorithm can converge faster. Most importantly, compared with the random algorithm, the proximal policy optimization (PPO) algorithm, and the deep deterministic policy gradient (DDPG) algorithm, the ground terminals’ energy consumption can be effectively reduced by our proposed algorithm.

VizNav: A Modular Off-Policy Deep Reinforcement Learning Framework for Vision-Based Autonomous UAV Navigation in 3D Dynamic Environments

Unmanned aerial vehicles (UAVs) provide benefits through eco-friendliness, cost-effectiveness, and reduction of human risk. Deep reinforcement learning (DRL) is widely used for autonomous UAV navigation; however, current techniques often oversimplify the environment or impose movement restrictions. Additionally, most vision-based systems lack precise depth perception, while range finders provide a limited environmental overview, and LiDAR is energy-intensive. To address these challenges, this paper proposes VizNav, a modular DRL-based framework for autonomous UAV navigation in dynamic 3D environments without imposing conventional mobility constraints. VizNav incorporates the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with Prioritized Experience Replay and Importance Sampling (PER) to improve performance in continuous action spaces and mitigate overestimations. Additionally, VizNav employs depth map images (DMIs) to enhance visual navigation by accurately estimating objects’ depth information, thereby improving obstacle avoidance. Empirical results show that VizNav, by leveraging TD3, improves navigation, and the inclusion of PER and DMI further boosts performance. Furthermore, the deployment of VizNav across various experimental settings confirms its flexibility and adaptability. The framework’s architecture separates the agent’s learning from the training process, facilitating integration with various DRL algorithms, simulation environments, and reward functions. This modularity creates a potential to influence RL simulation in various autonomous navigation systems, including robotics control and autonomous vehicles.

Trajectory Tracking Control of Variable Sweep Aircraft Based on Reinforcement Learning

An incremental deep deterministic policy gradient (IDDPG) algorithm is devised for the trajectory tracking control of a four-wing variable sweep (FWVS) aircraft with uncertainty. The IDDPG algorithm employs the line-of-sight (LOS) method for path tracking, formulates a reward function based on position and attitude errors, and integrates long short-term memory (LSTM) units into IDDPG algorithm to enhance its adaptability to environmental changes during flight. Finally, environmental disturbance factors are introduced in simulation to validate the designed controller’s ability to track climbing trajectories of morphing aircraft in the presence of uncertainty.

Deep Reinforcement Learning-Based Task Offloading and Load Balancing for Vehicular Edge Computing

Vehicular edge computing (VEC) effectively reduces the computational burden on vehicles by offloading tasks from resource-constrained vehicles to edge nodes. However, non-uniformly distributed vehicles offloading a large number of tasks cause load imbalance problems among edge nodes, resulting in performance degradation. In this paper, we propose a deep reinforcement learning-based decision scheme for task offloading and load balancing with the optimization objective of minimizing the system cost considering the split offloading of tasks and the load dynamics of edge nodes. First, we model the mutual interaction between mobile vehicles and Mobile Edge Computing (MEC) servers using a Markov decision process. Second, the optimal task-offloading and resource allocation decision is obtained by utilizing the twin delayed deep deterministic policy gradient algorithm (TD3), and server load balancing is achieved through edge collaboration using a server selection algorithm based on the technique for order preference by similarity to the ideal solution (TOPSIS). Finally, we have conducted extensive simulation experiments and compared the results with several other baseline schemes. The proposed scheme can more effectively reduce the system cost and increase the system resource utilization.

Inspection Robot Navigation Based on Improved TD3 Algorithm.

The swift advancements in robotics have rendered navigation an essential task for mobile robots. While map-based navigation methods depend on global environmental maps for decision-making, their efficacy in unfamiliar or dynamic settings falls short. Current deep reinforcement learning navigation strategies can navigate successfully without pre-existing map data, yet they grapple with issues like inefficient training, slow convergence, and infrequent rewards. To tackle these challenges, this study introduces an improved two-delay depth deterministic policy gradient algorithm (LP-TD3) for local planning navigation. Initially, the integration of the long-short-term memory (LSTM) module with the Prioritized Experience Re-play (PER) mechanism into the existing TD3 framework was performed to optimize training and improve the efficiency of experience data utilization. Furthermore, the incorporation of an Intrinsic Curiosity Module (ICM) merges intrinsic with extrinsic rewards to tackle sparse reward problems and enhance exploratory behavior. Experimental evaluations using ROS and Gazebo simulators demonstrate that the proposed method outperforms the original on various performance metrics.

Energy-Delay Joint Optimization for Task Offloading in Digital Twin-Assisted Internet of Vehicles

Vehicle edge computing (VEC) provides efficient services for vehicles by offloading tasks to edge servers. Notably, extant research mainly employs methods such as deep learning and reinforcement learning to make resource allocation decisions, without adequately accounting for the ramifications of high-speed mobility of vehicles and the dynamic nature of the Internet of Vehicles (IoV) on the decision-making process. This paper endeavours to tackle the aforementioned issue through the introduction of a novel concept, namely, a digital twin-assisted IoV. Among them, the digital twin of IoV offers training data for computational offloading and content caching decisions, which allows edge servers to directly interact with the dynamic environment while capturing its dynamic changes in real-time. Through this collaborative endeavour, edge intelligent servers can promptly respond to vehicular requests and return results. We transform the dynamic edge computing problem into a Markov decision process (MDP), and then solve it with the twin delayed deep deterministic policy gradient (TD3) algorithm. Simulation experiments demonstrate the adaptability of our proposed approach in the dynamic environment while successfully enhancing the Quality of Service, that is, decreasing total delay and energy consumption.

A Multi-Objective Optimal Control Method for Navigating Connected and Automated Vehicles at Signalized Intersections Based on Reinforcement Learning

The emergence and application of connected and automated vehicles (CAVs) have played a positive role in improving the efficiency of urban transportation and achieving sustainable development. To improve the traffic efficiency at signalized intersections in a connected environment while simultaneously reducing energy consumption and ensuring a more comfortable driving experience, this study investigates a flexible and real-time control method to navigate the CAVs at signalized intersections utilizing reinforcement learning (RL). Initially, control of CAVs at intersections is formulated as a Markov Decision Process (MDP) based on the vehicles’ motion state and the intersection environment. Subsequently, a comprehensive reward function is formulated considering energy consumption, efficiency, comfort, and safety. Then, based on the established environment and the twin delayed deep deterministic policy gradient (TD3) algorithm, a control algorithm for CAVs is designed. Finally, a simulation study is conducted using SUMO, with Lankershim Boulevard as the research scenario. Results indicate that the proposed methods yield a 13.77% reduction in energy consumption and a notable 18.26% decrease in travel time. Vehicles controlled by the proposed method also exhibit smoother driving trajectories.

A Deep Reinforcement Learning Approach for Optimizing Inventory Management in the Agri-Food Supply Chain

This research aims to improve inventory management throughout the agri-food supply chain, through the use of Deep Reinforcement Learning algorithms. Three Deep Reinforcement Learning algorithms, including Deep Q-Networks, Deep Deterministic Policy Gradient, and Proximal Policy Optimization algorithms were implemented and tested in order to evaluate their ability to actively manage inventories and improve the performance of supply chains. The results of the experimental phase offered important information regarding the performance of each Deep Reinforcement Learning algorithm. The Deep Deterministic Policy Gradient algorithm was identified as a viable choice, offering the best results in terms of accuracy to set the optimal inventories for the supply chain and improving the efficiency of the supply chain. The percentage of cost efficiency improved from 92.5% to 95.8% in the case of the DDPG model. The inventory turnover was also improved, surpassing the level of 8.1 units from the original level of 7.3 units which means that the system converts the inventor into sales in less time. The metric for on-time delivery also beneficiated from several improvements, reaching 96.5% form the level of 93.2% return. The quality metrics also registered a significant improvement and reported level of 96.2% after the implementation of the DDPG algorithm and compared to the level of 94.6% prior to the implementation of the algorithm. These results suggest that using such a system will bring beneficial changes to the supply chain and will offer the possibility of implementing a data-driven inventory system based on Deep Reinforcement Learning.

AoI minimization of ambient backscatter-assisted EH-CRN with cooperative spectrum sensing

To evaluate the timeliness and freshness of information from energy-constrained devices, we minimize the weighted average age of information (AoI) of the ambient backscatter-assisted energy harvesting cognitive radio network (AB-EH-CRN), where secondary transmitters (STs) follow the non-orthogonal multiple access (NOMA) strategy to transmit status updates. The secondary receiver (SR) schedules the selected actions for STs to perform spectrum access, and receives status updates from them. We formulate the AoI minimization problem where the action selection and transmission power of STs are considered as variables. To exploit spectrum resources in the AB-EH-CRN with unknown signal-to-noise ratios (SNRs) at the STs, we propose the deep neural network (DNN)-based cooperative spectrum sensing (CSS), and adopt an energy threshold approach for STs to determine the energy allocation between spectrum sensing (SS) and packet transmission. Moreover, we address the AoI minimization problem by the proposed hybrid action and energy penalty deep deterministic policy gradient (HAEP-DDPG) algorithm, which adapts to the hybrid continuous and discrete action spaces. Simulation results validate the advantage of the proposed HAEP-DDPG algorithm compared with comparison schemes in terms of AoI.

Deep reinforcement learning algorithm based ramp merging decision model

On-ramp merge is a complex traffic scenario in autonomous driving. Because of the uncertainty of the driving environment, most rule-based models cannot solve such a problem. This paper designs a ramp merging decision model based on deep deterministic policy gradient algorithm (DDPG) to solve the vehicle merging problem. To address the problems of slow algorithm merging and poor robustness of previous deep reinforcement learning algorithms in the field of intelligent vehicle ramp merging leading to the low success rate of intelligent vehicle merging, first, we introduce a simple recurrent unit (SRU) for extracting intelligent vehicle states and environment features and use the DDPG algorithm for intelligent vehicle decision making. Second, the experience playback pool of DDPG algorithm is improved by using priority sampling instead of uniform sampling. Finally, a multi-objective reward function is set up during training, considering factors such as safety and efficiency. The simulation experiments show that the improved algorithm improves the merging speed of the model, reduces the collision rate, and enables the vehicle to make more reasonable decisions. In addition, the superiority of the method is demonstrated by comparing with the advanced method.

Fuzzy-based collective pitch control for wind turbine via deep reinforcement learning

Wind turbines (WTs) have highly nonlinear and uncertain dynamics due to aerodynamic complexity, mechanical factors, and fluctuations in wind conditions. Turbulence and wind shear add complexity to modelling, especially in constant power region (region 3). Thus, an effective control design demands a deep understanding of the nonlinearities and uncertainties. This paper suggests a novel model-free reinforcement learning (RL) collective pitch angle controller to operate efficiently in region 3. The proposed controller stabilizes generator speed, maximizes power output, and minimizes fluctuations while accommodating system uncertainties, nonlinearity, and pitch limits. The disparity between WT dynamics due to wind speed perturbations and uncertainties is measured using a gap-metric criterion. The controller design adopts a deep deterministic policy gradient (DDPG) algorithm to train six agents in a medium-fidelity WT environment at different mean wind speeds to ensure the controller's robustness. Initially, imitation learning is used for efficient sample collection to fasten training convergence. Afterwards, the agent learns by interacting with the environment. After the training, the pitch control outputs from multi-trained agents are processed by a fuzzy system to have smooth transitions under different operating conditions. The resulting fuzzy DDPG (F-DDPG) controller is deployed to obtain the optimal pitch control action. The performance of the proposed F-DDPG controller is compared to the gain-scheduled PI (GSPI), Linear-Quadratic-Regulator (LQR), and single-DDPG-agent controllers. The controllers are simulated in high-fidelity onshore and offshore 5-MW WT environments using the OpenFAST/MATLAB simulation tools. The results reveal the superiority of the proposed controller in generalizing its optimal performance in different operating conditions.

Investigating the Impact of Curriculum Learning on Reinforcement Learning for Improved Navigational Capabilities in Mobile Robots

This paper proposes a method for finding the shortest path of a mobile robot using deep reinforcement learning with utilizing Proximal policy optimization algorithm (PPO) enhanced with curriculum learning. By modelling the environment in 3D space using the Webots simulator, we extend the PPO algorithm's capabilities to handle continuous states from 8 IR sensors and control the velocities of two motors of E-puck robot. Our study uniquely integrates curriculum learning into the PPO framework, aiming to improve adaptability and training efficiency in complex environments. A comparative analysis is conducted between the modified PPO, the original PPO, and the deep deterministic policy gradient algorithm, highlighting the strengths of our approach The results demonstrate that our curriculum-augmented PPO algorithm not only accelerates the training process but also shows superior adaptability and generalization in new environments. This work underscores the significant potential of curriculum learning in enhancing the performance of deep reinforcement learning algorithms for robust and efficient robotic navigation.

Energy Management for Hybrid Energy Storage System in Electric Based on Deep Deterministic Policy Gradient

In this paper, an intelligent control system design scheme based on deep deterministic policy gradient (DDPG) algorithm is proposed for the complex continuous action space problem in the hybrid energy storage system of electric vehicles. Firstly, the basic principle and internal logic of DDPG algorithm are introduced, including key elements such as Actor-Critic architecture, experience playback, target network, reward signal, policy gradient and value function update. Then, how to apply the DDPG algorithm to the industrial control system is described in detail. The Actor network learns the optimal strategy, the Critic network evaluates the value of the state-action pair, and uses the experience playback and the target network to improve the system stability and performance. Finally, the effect of the intelligent control system based on DDPG algorithm in complex environment is verified by simulation experiments. The results show that the system can effectively optimize the control strategy, improve the response speed and stability of the system, and has a good engineering application prospect.