Deep Deterministic Policy Gradient Algorithm Research Articles

Collision avoidance decision-making strategy for multiple USVs based on Deep Reinforcement Learning algorithm

—To address the issue of flexible decision-making for Unmanned Surface Vehicles (USVs) in emergency collision avoidance scenarios, this paper proposes a Collision Avoidance Decision-making Strategy (CADMS) for multiple USVs based on Deep Reinforcement Learning (DRL). The novelty of this method is: (1) The Preferential Experience Replay Mechanism (PERM) and the Gate Recurrent Unit (GRU) network are employed to enhance the Deep Deterministic Policy Gradient (DDPG) algorithm. The PERM facilitates faster convergence by increasing the sampling probability, thereby aiding in data assimilation. Meanwhile, the GRU enables self-assessment of sensor input significance, leading to more accurate action prediction. (2) In the design of the reward function, the departure from COLREGs has been fully considered, including the two-ship encounter situations and the multi-ship encounter situations. This study also provides a comprehensive analysis of neural network training and generalization verification. The experimental results demonstrate the effectiveness of the proposed approach in effectively avoiding collisions with multiple USVs in complex water environments. It is noteworthy that in comparison to conventional collision avoidance methods, the proposed CADMS model exhibits a superior collision avoidance success rate and enables safer and more efficient decision-making behaviors when confronted with intricate encounter situations.

Path planning of mobile robot based on improved TD3 algorithm in dynamic environment

This paper proposes an improved TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm to address the flaws of low success rate and slow training speed, when using the original TD3 algorithm in mobile robot path planning in dynamic environment. Firstly, prioritized experience replay and transfer learning are introduced to enhance the learning efficiency, where the probability of beneficial experiences being sampled in the experience pool is increased, and the pre-trained model is applied in an obstacle-free environment as the initial model for training in a dynamic environment. Secondly, dynamic delay update strategy is devised and OU noise is added to improve the success rate of path planning, where the probability of missing high-quality value estimate is reduced through changing the delay update interval dynamically, and the correlated exploration of the mobile robot inertial navigation system in the dynamic environment is temporally improved. The algorithm is tested by simulation where the Turtlebot3 robot model as a training object, the ROS melodic operating system and Gazebo simulation software as an experimental environment. Meanwhile, the result shows that the improved TD3 algorithm has a 16.6 % increase in success rate and a 23.5 % reduction in algorithm training time. A generalization experiment was designed finally, and it indicates that superior generation performance has been acquired in mobile robot path planning with continuous action spaces through the improved TD3 algorithm.

Research on a Personalized Decision Control Algorithm for Autonomous Vehicles Based on the Reinforcement Learning from Human Feedback Strategy

To address the shortcomings of previous autonomous decision models, which often overlook the personalized features of users, this paper proposes a personalized decision control algorithm for autonomous vehicles based on RLHF (reinforcement learning from human feedback). The algorithm combines two reinforcement learning approaches, DDPG (Deep Deterministic Policy Gradient) and PPO (proximal policy optimization), and divides the control scheme into three phases including pre-training, human evaluation, and parameter optimization. During the pre-training phase, an agent is trained using the DDPG algorithm. In the human evaluation phase, different trajectories generated by the DDPG-trained agent are scored by individuals with different styles, and the respective reward models are trained based on the trajectories. In the parameter optimization phase, the network parameters are updated using the PPO algorithm and the reward values given by the reward model to achieve personalized autonomous vehicle control. To validate the control algorithm designed in this paper, a simulation scenario was built using CARLA_0.9.13 software. The results demonstrate that the proposed algorithm can provide personalized decision control solutions for different styles of people, satisfying human needs while ensuring safety.

Reinforcement learning-based heuristic planning for optimized energy management in power-split hybrid electric heavy duty vehicles

In this work, we systematically integrate relevant expertise, specifically on the optimal brake-specific fuel consumption (BSFC) curve, battery characteristics and terrain information into the development of an energy management plan for heavy-duty power-split hybrid electric vehicles. We utilize deep deterministic policy gradient (DDPG) algorithm as one of the most sophisticated reinforcement learning technique. Initially, we begin by explaining the vehicle configuration's system modeling. Subsequently, we then present an energy management approach based on deep Q-learning concepts. A novel algorithm, Deep Deterministic Policy Gradient, for energy management control, have been created to combat the “curse of dimensionality” in reinforcement learning. The new AMSGrad optimization technique is used by DDPG algorithm to update the weight of neural networks. We robustly train the proposed control system in realistic driving environment. The Knowledge Incorporation (KI) based DDPG based system is compared systematically to the conventional DDDPG methodology and the benchmark Dynamic Programming (DP) method, the latter of which usually uses the RMSProp Optimizer in its formulation. The results show that as compared to the typical DQL policy, deep reinforcement learning approaches, notably expert Knowledge incorporation KI-DDPG and terrain information with the AMSGrad optimizer, achieve faster training speeds and reduced fuel consumption while maintaining the terminal state of charge (SOC).

Optimal service caching, pricing and task partitioning in mobile edge computing federation

Mobile Edge Computing (MEC) federations aim to establish a joint edge service model between Edge Infrastructure Providers (EIPs) and clouds, facilitating the sharing and utilization of MEC services and resources. However, in such a hierarchical multi-EIP MEC federation environment, optimizing service caching, task partitioning, and pricing strategies to maximize the profits gained by EIPs and minimize the cost of mobile devices (MDs) is challenging. To address this challenge, This paper first formulates a two-stage multi-leader, multi-follower Stackelberg game between EIPs and MDs. Then, a new method combining a Stackelberg-based Multi-Agent Deep Deterministic Policy Gradient (STMADDPG) algorithm and a Transformer-based Popularity Prediction (TPP) model is devised to learn the optimal strategies. Extensive experiments with real-world datasets are conducted to demonstrate the excellent performance of the proposed STMADDPG-TPP method.

Dynamic resource matching in manufacturing using deep reinforcement learning

Matching plays an important role in the logical allocation of resources across a wide range of industries. The benefits of matching have been increasingly recognized in manufacturing industries. In particular, capacity sharing has received much attention recently. In this paper, we consider the problem of dynamically matching demand-capacity types of manufacturing resources. We formulate the multi-period, many-to-many manufacturing resource-matching problem as a sequential decision process. The formulated manufacturing resource-matching problem involves large state and action spaces, and it is not practical to accurately model the joint distribution of various types of demands. To address the curse of dimensionality and the difficulty of explicitly modeling the transition dynamics, we use a model-free deep reinforcement learning approach to find optimal matching policies. Moreover, to tackle the issue of infeasible actions and slow convergence due to initial biased estimates caused by the maximum operator in Q-learning, we introduce two penalties to the traditional Q-learning algorithm: a domain knowledge-based penalty based on a prior policy and an infeasibility penalty that conforms to the demand–supply constraints. We establish theoretical results on the convergence of our domain knowledge-informed Q-learning providing performance guarantee for small-size problems. For large-size problems, we further inject our modified approach into the deep deterministic policy gradient (DDPG) algorithm, which we refer to as domain knowledge-informed DDPG (DKDDPG). In our computational study, including small- and large-scale experiments, DKDDPG consistently outperformed traditional DDPG and other RL algorithms, yielding higher rewards and demonstrating greater efficiency in time and episodes.

Model-Free Optimal Vibration Control of a Nonlinear System Based on Deep Reinforcement Learning

Optimal control of nonlinear vibration requires precise knowledge of the system and the solution to Hamilton–Jacobi–Bellman (HJB) equation. However, in practical engineering applications, acquiring precise system parameters poses challenges, and the analytical solutions for the HJB equation are difficult to obtain. In this paper, two reinforcement learning algorithms, Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and Soft Actor-Critic (SAC) algorithm, are employed to train neural network-based optimal controllers for the van der Pol vibration system in the presence of unknown system parameters. To validate their performance, the controllers undergo testing in a series of experiments, including assessments of free vibration, frequency sweep excitation, and Gaussian noise excitation. The results indicate that both the TD3-trained and SAC-trained neural network-based controller are capable of proficiently suppress the vibration of the van der Pol oscillator. Additionally, these two model-free controllers can approximate the optimal control law which solved based on the dynamic model of the nonlinear system.

Deep Reinforcement Learning for Ecological and Distributed Urban Traffic Signal Control with Multi-Agent Equilibrium Decision Making

Multi-Agent Reinforcement Learning (MARL) has shown strong advantages in urban multi-intersection traffic signal control, but it also suffers from the problems of non-smooth environment and inter-agent coordination. However, most of the existing research on MARL traffic signal control has focused on designing efficient communication to solve the environment non-smoothness problem, while neglecting the coordination between agents. In order to coordinate among agents, this paper combines MARL and the regional mixed-strategy Nash equilibrium to construct a Deep Convolutional Nash Policy Gradient Traffic Signal Control (DCNPG-TSC) model, which enables agents to perceive the traffic environment in a wider range and achieves effective agent communication and collaboration. Additionally, a Multi-Agent Distributional Nash Policy Gradient (MADNPG) algorithm is proposed in this model, which is the first time the mixed-strategy Nash equilibrium is used for the improvement in the Multi-Agent Deep Deterministic Policy Gradient algorithm traffic signal control strategy to provide the optimal signal phase for each intersection. In addition, the eco-mobility concept is integrated into MARL traffic signal control to reduce pollutant emissions at intersections. Finally, simulation results in synthetic and real-world traffic road networks show that DCNPG-TSC outperforms other state-of-the-art MARL traffic signal control methods in almost all performance metrics, because it can aggregate the information of neighboring agents and optimize the agent’s decisions through gaming to find an optimal joint equilibrium strategy for the traffic road network.

Multi-agent reinforcement learning based computation offloading and resource allocation for LEO Satellite edge computing networks

Due to the limitations caused by geographical conditions and economic requirements, it is difficult to provide computing services by terrestrial networks for mobile terminals in remote areas. To address this issue, mobile edge computing (MEC) servers can be deployed in the low earth orbit (LEO) satellites to act as a complement and accommodate the unserved terminals. However, offloading computing tasks to servers in satellites may increase the energy consumption of ground terminals. Considering the limited battery capacity of ground terminals, how to perform the computation offloading and resource allocation are key challenges in the LEO satellite edge computing networks. Therefore, in this paper, we investigate the energy minimization problem for LEO satellite edge computing networks, where a multi-agent deep reinforcement learning algorithm with global rewards is proposed to optimize the transmit power, CPU frequency, bit allocation, offloading decision and bandwidth allocation via a decentralized method. Simulation results show that our proposed algorithm can converge faster. Most importantly, compared with the random algorithm, the proximal policy optimization (PPO) algorithm, and the deep deterministic policy gradient (DDPG) algorithm, the ground terminals’ energy consumption can be effectively reduced by our proposed algorithm.

Joint Phase Shift Design and Resource Management for a Non-Orthogonal Multiple Access-Enhanced Internet of Vehicle Assisted by an Intelligent Reflecting Surface-Equipped Unmanned Aerial Vehicle

This paper integrates intelligent reflecting surfaces (IRS) with unmanned aerial vehicles (UAV) to enhance the transmission performance of the Internet of Vehicles (IoV) through non-orthogonal multiple access (NOMA). It focuses on strengthening the signals from cell edge vehicles (CEVs) to the base station by optimizing the wireless propagation environment via an IRS-equipped UAV. The primary goal is to maximize the sum data rate of CEVs while satisfying the constraint of the successive interference cancellation (SIC) decoding threshold. The challenge lies in the non-convex nature of jointly considering the power control, subcarrier allocation, and phase shift design, making the problem difficult to optimally solve. To address this, the problem is decomposed into two independent subproblems, which are then solved iteratively. Specifically, the optimal phase shift design is achieved using the deep deterministic policy gradient (DDPG) algorithm. Furthermore, the graph theory is applied to determine the subcarrier allocation policy and derive a closed-form solution for optimal power control. Finally, the simulation results show that the proposed joint phase shift and resource management scheme significantly enhances the sum data rate compared to the state-of-the-art schemes, thereby demonstrating the benefits of integrating the IRS-equipped UAV into NOMA-enhanced IoV.

Distributed auto disturbances rejection resilient control of permanent magnetic maglev trains based on the optimized deep deterministic policy gradient algorithm

Distributed auto disturbances rejection resilient control of permanent magnetic maglev trains based on the optimized deep deterministic policy gradient algorithm

Evolving testing scenario generation and intelligence evaluation for automated vehicles

Interaction between the background vehicles (BVs) and automated vehicles (AVs) in scenario-based testing plays a critical role in evaluating the intelligence of the AVs. Current testing scenarios typically employ predefined or scripted BVs, which inadequately reflect the complexity of human-like social behaviors in real-world driving scenarios, and also lack a systematic metric for evaluating the comprehensive intelligence of AVs. Therefore, this paper proposes an evolving scenario generation method, employing deep reinforcement learning (DRL) to construct human-like BVs that interact with AVs, and this evolving scenario is designed to test and evaluate the intelligence of AVs. Firstly, a class of BV driver models with human-like competitive, mutual, and cooperative driving motivations is designed. Then, utilizing the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and an improved level-k training procedure, the three distinct driver models acquire game-based interactive driving policies. And these driver models are combined to generate evolving scenarios in which they can interact continuously and evolve diverse contents. Next, a framework including safety, driving efficiency, and interaction utility are presented to evaluate and quantify the intelligence performance of 3 systems under test (SUTs), indicating the effectiveness of the evolving scenario for intelligence testing. Finally, the complexity and fidelity of the proposed evolving testing scenario are validated. The results demonstrate that the proposed evolving scenario exhibits the highest level of complexity compared to other baseline scenarios and has more than 85% similarity to naturalistic driving data. This highlights the potential of the proposed method to facilitate the development and evaluation of high-level AVs in a realistic and challenging environment.

A Pilot Study of Observation Poisoning on Selective Reincarnation in Multi-Agent Reinforcement Learning

This research explores the vulnerability of selective reincarnation, a concept in Multi-Agent Reinforcement Learning (MARL), in response to observation poisoning attacks. Observation poisoning is an adversarial strategy that subtly manipulates an agent’s observation space, potentially leading to a misdirection in its learning process. The primary aim of this paper is to systematically evaluate the robustness of selective reincarnation in MARL systems against the subtle yet potentially debilitating effects of observation poisoning attacks. Through assessing how manipulated observation data influences MARL agents, we seek to highlight potential vulnerabilities and inform the development of more resilient MARL systems. Our experimental testbed was the widely used HalfCheetah environment, utilizing the Independent Deep Deterministic Policy Gradient algorithm within a cooperative MARL setting. We introduced a series of triggers, namely Gaussian noise addition, observation reversal, random shuffling, and scaling, into the teacher dataset of the MARL system provided to the reincarnating agents of HalfCheetah. Here, the “teacher dataset” refers to the stored experiences from previous training sessions used to accelerate the learning of reincarnating agents in MARL. This approach enabled the observation of these triggers’ significant impact on reincarnation decisions. Specifically, the reversal technique showed the most pronounced negative effect for maximum returns, with an average decrease of 38.08% in Kendall’s tau values across all the agent combinations. With random shuffling, Kendall’s tau values decreased by 17.66%. On the other hand, noise addition and scaling aligned with the original ranking by only 21.42% and 32.66%, respectively. The results, quantified by Kendall’s tau metric, indicate the fragility of the selective reincarnation process under adversarial observation poisoning. Our findings also reveal that vulnerability to observation poisoning varies significantly among different agent combinations, with some exhibiting markedly higher susceptibility than others. This investigation elucidates our understanding of selective reincarnation’s robustness against observation poisoning attacks, which is crucial for developing more secure MARL systems and also for making informed decisions about agent reincarnation.

Reinforcement learning-based order-dispatching optimization in the ride-sourcing service

Since the emergence of ride-sourcing, the dispatching problem has been a hot research issue in the travel market. Previous studies designed various refined algorithms to improve the dispatching efficiency. However, the heterogeneity of passenger cancellation behavior and driver's work patterns is ignored in the modeling of these studies, which may result in the algorithms failing to entirely capture the randomness in the real world, thereby affecting dispatching efficiency. To fill this research gap, this paper employs a discrete choice model and Gaussian mixture model to model passenger cancellation behavior and extract driver working patterns, respectively. Furthermore, a cooperative deep deterministic policy gradient (DDPG) algorithm is proposed for dispatching. The algorithm constructs decision networks separately for drivers with different working patterns to learn the optimal dispatching decision and designs a centralized neural network to coordinate the dispatching of all vehicles to maximize the global benefit. A price incentive strategy is proposed to improve passenger loyalty by providing incentive prices. Moreover, based on the constructed discrete choice model, a simulation environment is designed to describe the cancellation behavior in the real world and support the training of the algorithm. Experimental results based on real data in Beijing show that the cooperative DDPG algorithm can improve the order response rate by 2% to 6% and platform revenue by 3% to 9% in multiple travel periods while providing a better travel experience to passengers. Additionally, the price incentive strategy also effectively reduces passenger cancellation.

Dependency task offloading in semantic communications

In the realm of advanced wireless communications, the significance of semantic communication has become increasingly pronounced, particularly in its focus on the meaning and efficiency of information transmission. Given the voluminous nature of task data and the intricacy of task interconnections, executing semantic segmentation tasks directly on user-end devices often falls short due to limitations in processing power. Leveraging mobile edge computing, however, presents a strategic enhancement in performing semantic segmentation tasks by delegating computational workloads to proximal devices, thus diminishing latency and reducing energy expenditure. This document delineates a task offloading framework aimed at refining the management of interdependent tasks, thereby boosting the efficiency of processing complex tasks within environments of semantic communication. The framework abstracts tasks in the form of directed acyclic graphs (DAG) and approaches the optimization challenge through a Markov Decision Process (MDP), targeting an optimal balance between task completion, delay reduction, and energy savings. To adapt to fluctuating network conditions, a novel approach employing the Deep Deterministic Policy Gradient (DTO-DDPG) algorithm is introduced for optimization decision-making. Our empirical tests underscore the framework’s capacity to significantly elevate task completion rates while simultaneously decreasing both energy consumption and latency in scenarios of semantic communication.

VizNav: A Modular Off-Policy Deep Reinforcement Learning Framework for Vision-Based Autonomous UAV Navigation in 3D Dynamic Environments

Unmanned aerial vehicles (UAVs) provide benefits through eco-friendliness, cost-effectiveness, and reduction of human risk. Deep reinforcement learning (DRL) is widely used for autonomous UAV navigation; however, current techniques often oversimplify the environment or impose movement restrictions. Additionally, most vision-based systems lack precise depth perception, while range finders provide a limited environmental overview, and LiDAR is energy-intensive. To address these challenges, this paper proposes VizNav, a modular DRL-based framework for autonomous UAV navigation in dynamic 3D environments without imposing conventional mobility constraints. VizNav incorporates the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with Prioritized Experience Replay and Importance Sampling (PER) to improve performance in continuous action spaces and mitigate overestimations. Additionally, VizNav employs depth map images (DMIs) to enhance visual navigation by accurately estimating objects’ depth information, thereby improving obstacle avoidance. Empirical results show that VizNav, by leveraging TD3, improves navigation, and the inclusion of PER and DMI further boosts performance. Furthermore, the deployment of VizNav across various experimental settings confirms its flexibility and adaptability. The framework’s architecture separates the agent’s learning from the training process, facilitating integration with various DRL algorithms, simulation environments, and reward functions. This modularity creates a potential to influence RL simulation in various autonomous navigation systems, including robotics control and autonomous vehicles.

Trajectory Tracking Control of Variable Sweep Aircraft Based on Reinforcement Learning.

An incremental deep deterministic policy gradient (IDDPG) algorithm is devised for the trajectory tracking control of a four-wing variable sweep (FWVS) aircraft with uncertainty. The IDDPG algorithm employs the line-of-sight (LOS) method for path tracking, formulates a reward function based on position and attitude errors, and integrates long short-term memory (LSTM) units into IDDPG algorithm to enhance its adaptability to environmental changes during flight. Finally, environmental disturbance factors are introduced in simulation to validate the designed controller's ability to track climbing trajectories of morphing aircraft in the presence of uncertainty.

Deep Reinforcement Learning Lane-Changing Decision Algorithm for Intelligent Vehicles Combining LSTM Trajectory Prediction

Intelligent decisions for autonomous lane-changing in vehicles have consistently been a focal point of research in the industry. Traditional lane-changing algorithms, which rely on predefined rules, are ill-suited for the complexities and variabilities of real-world road conditions. In this study, we propose an algorithm that leverages the deep deterministic policy gradient (DDPG) reinforcement learning, integrated with a long short-term memory (LSTM) trajectory prediction model, termed as LSTM-DDPG. In the proposed LSTM-DDPG model, the LSTM state module transforms the observed values from the observation module into a state representation, which then serves as a direct input to the DDPG actor network. Meanwhile, the LSTM prediction module translates the historical trajectory coordinates of nearby vehicles into a word-embedding vector via a fully connected layer, thus providing predicted trajectory information for surrounding vehicles. This integrated LSTM approach considers the potential influence of nearby vehicles on the lane-changing decisions of the subject vehicle. Furthermore, our study emphasizes the safety, efficiency, and comfort of the lane-changing process. Accordingly, we designed a reward and penalty function for the LSTM-DDPG algorithm and determined the optimal network structure parameters. The algorithm was then tested on a simulation platform built with MATLAB/Simulink. Our findings indicate that the LSTM-DDPG model offers a more realistic representation of traffic scenarios involving vehicle interactions. When compared to the traditional DDPG algorithm, the LSTM-DDPG achieved a 7.4% increase in average single-step rewards after normalization, underscoring its superior performance in enhancing lane-changing safety and efficiency. This research provides new ideas for advanced lane-changing decisions in autonomous vehicles.

Deep Reinforcement Learning-Based Task Offloading and Load Balancing for Vehicular Edge Computing

Vehicular edge computing (VEC) effectively reduces the computational burden on vehicles by offloading tasks from resource-constrained vehicles to edge nodes. However, non-uniformly distributed vehicles offloading a large number of tasks cause load imbalance problems among edge nodes, resulting in performance degradation. In this paper, we propose a deep reinforcement learning-based decision scheme for task offloading and load balancing with the optimization objective of minimizing the system cost considering the split offloading of tasks and the load dynamics of edge nodes. First, we model the mutual interaction between mobile vehicles and Mobile Edge Computing (MEC) servers using a Markov decision process. Second, the optimal task-offloading and resource allocation decision is obtained by utilizing the twin delayed deep deterministic policy gradient algorithm (TD3), and server load balancing is achieved through edge collaboration using a server selection algorithm based on the technique for order preference by similarity to the ideal solution (TOPSIS). Finally, we have conducted extensive simulation experiments and compared the results with several other baseline schemes. The proposed scheme can more effectively reduce the system cost and increase the system resource utilization.

Energy-Delay Joint Optimization for Task Offloading in Digital Twin-Assisted Internet of Vehicles

Vehicle edge computing (VEC) provides efficient services for vehicles by offloading tasks to edge servers. Notably, extant research mainly employs methods such as deep learning and reinforcement learning to make resource allocation decisions, without adequately accounting for the ramifications of high-speed mobility of vehicles and the dynamic nature of the Internet of Vehicles (IoV) on the decision-making process. This paper endeavours to tackle the aforementioned issue through the introduction of a novel concept, namely, a digital twin-assisted IoV. Among them, the digital twin of IoV offers training data for computational offloading and content caching decisions, which allows edge servers to directly interact with the dynamic environment while capturing its dynamic changes in real-time. Through this collaborative endeavour, edge intelligent servers can promptly respond to vehicular requests and return results. We transform the dynamic edge computing problem into a Markov decision process (MDP), and then solve it with the twin delayed deep deterministic policy gradient (TD3) algorithm. Simulation experiments demonstrate the adaptability of our proposed approach in the dynamic environment while successfully enhancing the Quality of Service, that is, decreasing total delay and energy consumption.