Deep Deterministic Policy Gradient Algorithm Research Articles

Design of sliding mode controller for servo feed system based on generalized extended state observer with reinforcement learning

Nonlinear friction, system uncertainty, and external disturbances have a significant impact on the performance of high-precision servo feed systems. In order to achieve higher tracking accuracy, a sliding mode controller based on a generalized extended state observer with the double critic deep deterministic policy gradient algorithm is designed. Firstly, a flexible two mass drive model (FTMDM) is established for the two-axis differential micro-feed system (TDMS). Next, a generalized extended state observer (GESO) is designed to estimate matched interference and mismatched interference. And it is proved that the observation error of GESO is bounded. A sliding mode controller based on GESO is further proposed. The stability of the controller has been proven through Lyapunov theory, and the error is bounded and converges to zero in finite time. The tuning process of controller parameters is simplified by using quadratic optimal control principle. Furthermore, a double critic deep deterministic policy gradient algorithm (DCDDPG) is proposed to achieve dynamic optimization of parameters about GESO. The simulation results show that GESO with DCDDPG can reduce the observation error of step signal and sinusoidal signal, and improve the observation accuracy of nonlinear friction significantly. Finally, experimental results show that the proposed control method achieves more accurate position tracking performance on TDMS.

Adaptive MPC path-tracking controller based on reinforcement learning and preview-based PID controller

Path-tracking control is a crucial process for autonomous vehicles, ensuring that the vehicle drives safely along the reference path, and the suitable controller parameters ensure the accuracy and stability of this process. To enhance the adaptability of traditional path-tracking controller parameters, this paper proposes an adaptive model predictive control (MPC) controller based on a preview-based PID controller and deep deterministic policy gradient (DDPG) algorithm to achieve adaptive tuning of the controller parameters. Starting with the design of the dynamics tracking error model of the vehicle and the MPC controller. Based on the actor-critic reinforcement learning architecture, the DDPG agent is designed to tune the prediction horizon and weight matrix of the MPC controller. A preview-based PID controller is proposed to improve the efficiency and stability of reinforcement learning and compensate for the error in vehicle modeling. The improved algorithm performance is verified through the simulation scenarios of high-speed lane changing and accelerated overtaking scenarios constructed by MATLAB/Simulink. The results show that the improved algorithm significantly improves the adaptive ability of the traditional MPC controller to time-varying conditions and achieves higher tracking accuracy and stability.

Adaptive Deep Ant Colony Optimization–Asymmetric Strategy Network Twin Delayed Deep Deterministic Policy Gradient Algorithm: Path Planning for Mobile Robots in Dynamic Environments

Path planning is one of the main focal points and challenges in mobile robotics research. Traditional ant colony optimization (ACO) algorithms encounter issues such as low efficiency, slow convergence, and a tendency to become stuck in local optima and search stagnation when applied to complex dynamic environments. Addressing these challenges, this study introduces an adaptive deep ant colony optimization (ADACO) algorithm, which significantly improves efficiency and convergence speed through enhanced pheromone diffusion mechanisms and updating strategies, applied to global path planning. To adapt to dynamically changing environments and achieve more precise local path planning, an asymmetric strategy network TD3 algorithm (ATD3) is further proposed, which utilizes global path planning information within the strategy network only, creating a new hierarchical path planning algorithm—ADACO-ATD3. Simulation experiments demonstrate that the proposed algorithm significantly outperforms in terms of path length and number of iterations, effectively enhancing the mobile robot’s path planning performance in complex dynamic environments.

Reinforcement Learning-Based Optimal Hull Form Design with Variations in Fore and Aft Parts

Abstract With recent advancements in artificial intelligence technology, various studies are being conducted in the shipbuilding industry. Traditionally, hull form variation methods have relied on the intuition and expertise of designers, leading to inconsistent results and unintended changes in the ship's main dimensions depending on the designer's competence. Moreover, the iterative process of design variation and analysis to derive the optimal hull form is both costly and time-consuming. To address these issues, this study proposes an optimal hull design technique utilizing reinforcement learning, a type of unsupervised learning in machine learning. Reinforcement learning allows the model to learn from past policies by recording and accumulating the rewards associated with various actions taken by an agent in a specific environment. In this study, after calculating the main parameters of the ship, the agent defines a state representing hull information and performs local transformations of the bow and stern. The reward of reinforcement learning is defined as the change in total resistance due to the hull deformation, constrained by limiting the tolerance of the ship's prismatic coefficient (CP) and longitudinal center of buoyancy (LCB). In this study, the problem is solved by comparing the proximal policy optimization (PPO) algorithm and the deep deterministic policy gradient (DDPG) algorithm to find the best deep reinforcement learning (DRL) model for the hull optimization problem. The results were compared with the genetic algorithm and speed-constrained multi-objective particle swarm optimization (SMPSO), and the optimal hull resistance values were less different, but the time of the reinforcement learning model was five times shorter.

A multi-objective reinforcement learning framework for real-time drilling optimization based on symbolic regression and perception

In the context of the global energy transition, optimizing deep-water oil and gas drilling parameters is crucial for ensuring safety while improving efficiency. Traditional methods face limitations in highly dynamic and nonlinear drilling environments, struggling to balance speed and cost-effectiveness. Furthermore, these methods rely on real-time logging-while-drilling (LWD) data for decision-making, but delays in data collection and processing hinder timely adjustments of drilling parameters, affecting decision accuracy and responsiveness. This paper proposes a multi-objective drilling parameter optimization framework, incorporating symbolic regression, time-series networks, and Markov decision processes to precisely predict ROP, formation conditions, and optimize drilling parameters in real time. Key innovations include a multi-population evolutionary symbolic regression algorithm for constructing empirical equations, the integration of variational mode decomposition (VMD) and sample entropy for data preprocessing, and multi-head self-attention time-series networks to enhance prediction accuracy. Quantile regression further estimates the range of drilling parameter adjustments. Additionally, a drilling parameter optimization deep deterministic policy gradient (DPODDPG) algorithm was developed to automate real-time parameter adjustments. Empirical analysis on the Ledong 10-1 block in the South China Sea demonstrated significant improvements: ROP increased from 54.18 m/hr to 122.17 m/hr, mechanical specific energy (MSE) decreased from 100.82 MPa to 97.78 MPa, and cost per foot reduced from 121.16 × 102 CNY/m to 51.31 × 102 CNY/m. Compared to traditional methods, the proposed framework showed clear advantages in enhancing ROP, reducing MSE, and controlling costs, further validating its superiority in complex drilling environments. This method not only significantly improves drilling efficiency and economic benefits but also adapts to complex and changing drilling conditions, showing broad application potential, particularly in challenging deep-water oil and gas drilling operations, where it can provide more efficient and reliable optimization solutions.

Multi-Agent DRL for Air-to-Ground Communication Planning in UAV-Enabled IoT Networks.

In this paper, we present a novel method to enhance the sum-rate effectiveness in full-duplex unmanned aerial vehicle (UAV)-assisted communication networks. Existing approaches often couple uplink and downlink associations, resulting in suboptimal performance, particularly in dynamic environments where user demands and network conditions are unpredictable. To overcome these limitations, we propose a decoupling of uplink and downlink associations for ground-based users (GBUs), significantly improving network efficiency. We formulate a comprehensive optimization problem that integrates UAV trajectory design and user association, aiming to maximize the overall sum-rate efficiency of the network. Due to the problem's non-convexity, we reformulate it as a Partially Observable Markov Decision Process (POMDP), enabling UAVs to make real-time decisions based on local observations without requiring complete global information. Our framework employs multi-agent deep reinforcement learning (MADRL), specifically the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, which balances centralized training with distributed execution. This allows UAVs to efficiently learn optimal user associations and trajectory controls while dynamically adapting to local conditions. The proposed solution is particularly suited for critical applications such as disaster response and search and rescue missions, highlighting the practical significance of utilizing UAVs for rapid network deployment in emergencies. By addressing the limitations of existing centralized and distributed solutions, our hybrid model combines the benefits of centralized training with the adaptability of distributed inference, ensuring optimal UAV operations in real-time scenarios.

Energy management with adaptive moving average filter and deep deterministic policy gradient reinforcement learning for fuel cell hybrid electric vehicles

Fuel cell hybrid electric vehicles (FCHEV) with battery (BAT) and supercapacitor (SC) advance in flexible configuration and high energy efficiency. However, the complex coupling relationship among various power sources poses a severe challenge to the design of the energy management system (EMS), including multi-degrees of freedom power allocation, fuel economy, and power sources lifespan of the FCHEV. This paper proposes an EMS based on a dual-layer power distribution structure. In the upper layer, adaptive moving average filter (AMAF) is designed to separate different frequency power, where the energy supply of the SC is managed to attenuate fluctuating power and simplifies the optimization problem and reduces computational costs. The lower layer is constructed by the deep deterministic policy gradient (DDPG) algorithm, where fuel cell system (FCS) hydrogen consumption and degradation rewards are designed to simultaneously enhance fuel efficiency and degradation performance by regulating the FCS real-time power variation. The proposed strategy has been evaluated regarding FCHEV fuel economy and FCS durability under combined driving cycle simulation, which shows AMAF + DDPG strategy reduces fuel consumption by 7.24 % and 1.3 %, also the degradation reduces by 0.04 % and 0.02 % compared with different EMS. Simulation results demonstrate that AMAF + DDPG optimizes the output characteristics of power sources.

Joint Computation Offloading and Trajectory Optimization for Edge Computing UAV: A KNN-DDPG Algorithm

Unmanned aerial vehicles (UAVs) are widely used to improve the coverage and communication quality of wireless networks and assist mobile edge computing (MEC) due to their flexible deployments. However, the UAV-assisted MEC systems also face challenges in terms of computation offloading and trajectory planning in the dynamic environment. This paper employs deep reinforcement learning to jointly optimize the computation offloading and trajectory planning for UAV-assisted MEC system. Specifically, this paper investigates a general scenario where multiple pieces of user equipment (UE) offload tasks to a UAV equipped with a MEC server to collaborate on a complex job. By fully considering UAV and UE movement, computation offloading ratio, and blocked relations, a joint computation offloading and trajectory optimization problem is formulated to minimize the maximum computational delay. Due to the non-convex nature of the problem, it is converted into a Markov decision process, and solved by the deep deterministic policy gradient (DDPG) algorithm. To enhance the exploration capability and stability of DDPG, the K-nearest neighbor (KNN) algorithm is employed, namely KNN-DDPG. Moreover, the prioritized experience replay algorithm, where the constant learning rate is replaced by the decaying learning rate, is utilized to enhance the converge. To validate the effectiveness and superiority of the proposed algorithm, KNN-DDPG is compared with the benchmark DDPG algorithm. Simulation results demonstrate that KNN-DDPG can converge and achieve 3.23% delay reduction compared to DDPG.

Advanced security measures in coupled phase-shift STAR-RIS networks: A DRL approach

The rapid development of next-generation wireless networks has intensified the need for robust security measures, particularly in environments susceptible to eavesdropping. Simultaneous transmitting and reflecting reconfigurable intelligent surfaces (STAR-RIS) have emerged as a transformative technology, offering full-space coverage by manipulating electromagnetic wave propagation. However, the inherent flexibility of STAR-RIS introduces new vulnerabilities, making secure communication a significant challenge. To overcome these challenges, we propose a deep reinforcement learning (DRL) based secure and efficient beamforming optimization strategy, leveraging the deep deterministic policy gradient (DDPG) algorithm. By framing the problem as a Markov decision process (MDP), our approach enables the DDPG algorithm to learn optimal strategies for beamforming and transmission and reflection coefficients (TARCs) configurations. This method is specifically designed to optimize phase-shift coefficients within the STAR-RIS environment, effectively managing the coupled phase shifts and complex interactions that are critical for enhancing physical layer security (PLS). Through extensive simulations, we demonstrate that our DRL-based strategy not only outperforms traditional optimization techniques but also achieves real-time adaptive optimization, significantly improving both confidentiality and network efficiency. This research addresses key gaps in secure wireless communication and sets a new standard for future advancements in intelligent, adaptable network technologies.

Multi-UAV Cooperative Pursuit of a Fast-Moving Target UAV Based on the GM-TD3 Algorithm

Recently, developing multi-UAVs to cooperatively pursue a fast-moving target has become a research hotspot in the current world. Although deep reinforcement learning (DRL) has made a lot of achievements in the UAV pursuit game, there are still some problems such as high-dimensional parameter space, the ease of falling into local optimization, the long training time, and the low task success rate. To solve the above-mentioned issues, we propose an improved twin delayed deep deterministic policy gradient algorithm combining the genetic algorithm and maximum mean discrepancy method (GM-TD3) for multi-UAV cooperative pursuit of high-speed targets. Firstly, this paper combines GA-based evolutionary strategies with TD3 to generate action networks. Then, in order to avoid local optimization in the algorithm training process, the maximum mean difference (MMD) method is used to increase the diversity of the policy population in the updating process of the population parameters. Finally, by setting the sensitivity weights of the genetic memory buffer of UAV individuals, the mutation operator is improved to enhance the stability of the algorithm. In addition, this paper designs a hybrid reward function to accelerate the convergence speed of training. Through simulation experiments, we have verified that the training efficiency of the improved algorithm has been greatly improved, which can achieve faster convergence; the successful rate of the task has reached 95%, and further validated UAVs can better cooperate to complete the pursuit game task.

PID Controller Based on Improved DDPG for Trajectory Tracking Control of USV

When navigating dynamic ocean environments characterized by significant wave and wind disturbances, USVs encounter time-varying external interferences and underactuated limitations. This results in reduced navigational stability and increased difficulty in trajectory tracking. Controllers based on deterministic models or non-adaptive control parameters often fail to achieve the desired performance. To enhance the adaptability of USV motion controllers, this paper proposes a trajectory tracking control algorithm that calculates PID control parameters using an improved Deep Deterministic Policy Gradient (DDPG) algorithm. Firstly, the maneuvering motion model and parameters for USVs are introduced, along with the guidance law for path tracking and the PID control algorithm. Secondly, a detailed explanation of the proposed method is provided, including the state, action, and reward settings for training the Reinforcement Learning (RL) model. Thirdly, the simulations of various algorithms, including the proposed controller, are presented and analyzed for comparison, demonstrating the superiority of the proposed algorithm. Finally, a maneuvering experiment under wave conditions was conducted in a marine tank using the proposed algorithm, proving its feasibility and effectiveness. This research contributes to the intelligent navigation of USVs in real ocean environments and facilitates the execution of subsequent specific tasks.

A novel TD3 for solving multi-level imperfect maintenance optimization problem

Abstract As we all know, multi-level imperfect maintenance strategy is usually more effective than single-level maintenance strategy for the actual production machines. At the same time, in the previous multi-level maintenance strategies, the majority of maintenance models only consider the constant maintenance thresholds, while variable maintenance thresholds are usually ignored. Under these contexts, a novel multi-level imperfect maintenance model with variable preventive maintenance (PM) thresholds, variable overhaul maintenance (OM) thresholds and variable number of PMs in each OM cycle is established. In order to deal with the concerned problem, a novel twin delayed deep deterministic policy gradient (TD3) algorithm that is a kind of reinforcement learning is designed, renamed as NTD3. Finally, through numerical simulation, we can find that (1) the average improvements between the proposed maintenance strategy and other three traditional strategies in the average cost rate (ACR) are 11.50%, 595.91% and 5.16%, respectively; and (2) the average improvement between the proposed NTD3 and other random search method is 5.53%. Thus, the effectiveness of the proposed maintenance strategy and the superiority of proposed NTD3 are all demonstrated.

DT-Block: Adaptive vertical federated reinforcement learning scheme for secure and efficient communication in 6G

The necessities of security and data sharing have focused on federated learning because of using decentralized data sources. The existing works used federated learning for security, however, it still faces many challenges such as poor security and privacy, computational complexity, etc. In this research, we propose adaptive vertical federated learning using a reinforcement learning approach and blockchain. The proposed work includes three phases: user registration and authentication, machine learning-based client selection, and adaptive secure federated learning. Initially, all the users register their credentials to the cognitive agent, which generates a private key, public key, and random number using a Chaotic Isogenic Post Quantum Cryptography (CIPQC) algorithm. Second, optimal clients are selected for participating in federated learning which improves learning rate and reduces complexity. Here, optimal clients are selected by the Enhanced Multilayer Feed Forward Neural Network (EMFFN) algorithm by considering CSI, RSSI, bandwidth, energy, communication efficiency, and statistical efficiency. Finally, adaptive secure federated learning is performed by the Distributed Distributional Deep Deterministic Policy Gradient (D4PG) algorithm, where the local models are adaptively used by the private strategy based on its sensitivity. The aggregated global models are stored in DT-block (dendrimer tree-based blockchain) which stores the data in a dendrimer tree structure for increasing scalability and reducing search time during data retrieval. The simulation of this research is conducted by NS-3.26 network simulator and the performance of the proposed DT-Block model is estimated based on various performance metrics such as accuracy, delay, loss, f1-score, and security strength this demonstrated that the suggested effort produced better results both in terms of quantitative and qualitative aspects.

Multi-objective energy management strategy for a dual-stack fuel cell heavy-duty truck via deep reinforcement learning

The Proton Exchange Membrane Fuel Cell (PEMFC) system, as an efficient, zero-pollution emission, and high-efficiency energy conversion device, is particularly suitable for heavy-duty transportation. Considering the limited power of the current PEMFC system, this study focuses on a hybrid fuel cell heavy-duty truck with dual stacks. Considering the optimal operating range of the PEMFC system and the health status of the lithium-ion battery pack, an energy management strategy (EMS) based on the Deep Deterministic Policy Gradient (DDPG) algorithm is established. This strategy optimizes the hydrogen consumption cost, degradation costs of lithium-ion battery and fuel cells. Additionally, a dynamic programming (DP) and an equivalent consumption minimization strategy (ECMS) based EMSs are established. The power allocation and overall transportation costs under the CSC driving cycle are compared and verified for the three EMSs. The results show that the DDPG algorithm provides a more optimal power allocation for the dual-stack fuel cells, improving the overall system efficiency, enhancing the lifecycle of heavy-duty trucks, and significantly reducing overall travel costs.

Three-dimensional cooperative guidance with impact angle constraints via value-policy decomposed multi-agent reinforcement learning

For a three-dimensional impact angle-constrained time-coordination guidance problem, a value-policy decomposed multi-agent twin delayed deep deterministic (VPD-MATD3) policy gradient algorithm is proposed in this paper, and a temporal-spatial cooperative guidance law is designed consequently. The derived cooperative engagement kinematics is formulated as a value-policy decomposed partially observable Markov decision process. Value decomposition and policy decomposition are correspondingly proposed to address the two bottlenecks of reward coupling and policy coupling in 3D guidance, thus improving the training process. The training environment with multiple uncertainties, such as observation noise, maneuverability perturbation, autopilot time lag, switching communication topology, and random initialization, is subsequently constructed. Time-energy type reward functions are designed, incorporating energy consumption as a direct optimization indicator via a penalty term. A long short-term memory network layer is inserted inside the policy networks to suppress the negative effect of observation noise. The policy training curves verify the significant effect of the proposed algorithm in improving training convergence. The policy testing results illustrate the robustness and generalization of the VPD-MATD3 cooperative guidance law and its ability to cope with randomly switched communication topologies. Moreover, the comparative testing further reveals its time-energy suboptimal property.

Enhancing Cooperative Lane-Changing for Intelligent Connected Vehicles: A DDPG-Based Approach

Current research on lane-changing for intelligent connected vehicles (ICVs) primarily focuses on the control of a single ICV. Even studies on cooperative lane-changing among multiple ICVs often use fixed control algorithms, which are difficult to adapt to constantly changing traffic conditions. This paper proposes a method based on the deep deterministic policy gradient (DDPG) model. The DDPG algorithm has strong generalization capabilities, enabling it to handle complex traffic scenarios. Combined with the design of innovative reward functions, it successfully achieves the control of cooperative lane-changing for multiple ICVs. The novelty of this model is mainly reflected in the implementation of multiple ICVs control and cooperative strategies through the design of a speed reward function. Specifically, when two or more ICVs come within a 15-m range, a collaborative strategy is implemented. In this strategy, the original speeds of the vehicles involved are replaced with their average speed. This average speed is then used to compute the velocity reward. Moreover, an additional headway reward is introduced when two vehicles are on the same lane with a distance of less than 10[Formula: see text]m between them, further optimizing lane-changing decisions and enhancing driving efficiency. The innovative approach described in this paper has been validated through simulations in the highway-env simulation environment. The model’s effectiveness was tested under various conditions, including different road densities and numbers of ICVs. Results indicate that with an increasing number of episodes, the model consistently enhances both the fleet’s and each individual vehicle’s average speed and steps. It also ensures that the headway is maintained within a desired range of 10–20[Formula: see text]m. This paper showcases the potential applications of a new DDPG-based multi-networked vehicular cooperative lane-changing decision model across different scenarios. These findings provide important reference values for future research and development in intelligent transportation systems (ITS), particularly in optimizing vehicle cooperative behaviors and enhancing traffic efficiency. This paper emphasizes the model’s potential to significantly impact the management and operation of future vehicular networks, paving the way for more sophisticated and efficient transportation systems.

A multi-objective hierarchical deep reinforcement learning algorithm for connected and automated HEVs energy management

Connected and autonomous vehicles have offered unprecedented opportunities to improve fuel economy and reduce emissions of hybrid electric vehicle (HEV) in vehicular platoons. In this context, a hierarchical control strategy is put forward for connected HEVs. Firstly, we consider a deep deterministic policy gradient (DDPG) algorithm to compute the optimized vehicle speed using a trained optimal policy via vehicle-to-vehicle communication in the upper level. A multi-objective reward function is introduced, integrating vehicle fuel consumption, battery state-of-the-charge, emissions, and vehicle car-following objectives. Secondly, an adaptive equivalent consumption minimization strategy is devised to implement vehicle-level torque allocation in the platoon. Two drive cycles, HWFET and human-in-the-loop simulator driving cycles are utilized for realistic testing of the considered platoon energy management. It is shown that DDPG runs the engine more efficiently than the widely-implemented Q-learning and deep Q-network, thus showing enhanced fuel savings. Further, the contribution of this paper is to speed up the higher-level vehicular control application of deep learning algorithms in the connected and automated HEV platoon energy management applications.

Autonomous collision avoidance decision-making method for USV based on ATL-TD3 algorithm

To address the issue of decision-making for ship collision avoidance in complex encounter situations, an Autonomous Collision Avoidance Decision-making (ACAD) model based on Attention Long Short-Term Memory Twin Delayed Deep Deterministic Policy Gradient (ATL-TD3) algorithm for Unmanned Surface Vessel (USV) is proposed in this study. The novelty of this method is: (1) Incorporating LSTM and multi-head self-attention mechanism into the existing TD3 algorithm network enhances its focus on pertinent historical state information. Furthermore, the generalization ability of the trained model is further enhanced through optimization of the accumulated experience. (2) A comprehensive reward mechanism is designed, incorporating ship domain, arena, yaw, and other relevant factors, which solve the encounters defined in clauses 13–17 in Chapter 2 of COLREGs and immediate danger. The experiment involved a comparative analysis of the average reward curve and success rate during training under different environments. Additionally, 20 cases of Imazu classic encounters and real navigation data from the Suez Canal were utilized to validate the proposed algorithm. It is highlight that the ACAD model trained by the ATL-TD3 algorithm exhibits remarkable performance in the generalization verification, surpassing conventional collision avoidance methods.

Jointly Optimization of Delay and Energy Consumption for Multi-Device FDMA in WPT-MEC System.

With the rapid development of mobile edge computing (MEC) and wireless power transfer (WPT) technologies, the MEC-WPT system makes it possible to provide high-quality data processing services for end users. However, in a real-world WPT-MEC system, the channel gain decreases with the transmission distance, leading to "double near and far effect" in the joint transmission of wireless energy and data, which affects the quality of the data processing service for end users. Consequently, it is essential to design a reasonable system model to overcome the "double near and far effect" and make reasonable scheduling of multi-dimensional resources such as energy, communication and computing to guarantee high-quality data processing services. First, this paper designs a relay collaboration WPT-MEC resource scheduling model to improve wireless energy utilization efficiency. The optimization goal is to minimize the normalization of the total communication delay and total energy consumption while meeting multiple resource constraints. Second, this paper imports a BK-means algorithm to complete the end terminals cluster to guarantee effective energy reception and adapts the whale optimization algorithm with adaptive mechanism (AWOA) for mobile vehicle path-planning to reduce energy waste. Third, this paper proposes an immune differential enhanced deep deterministic policy gradient (IDDPG) algorithm to realize efficient resource scheduling of multiple resources and minimize the optimization goal. Finally, simulation experiments are carried out on different data, and the simulation results prove the validity of the designed scheduling model and proposed IDDPG.

Path Planning in Complex Environments Using Attention-Based Deep Deterministic Policy Gradient

The traditional Deep Deterministic Policy Gradient (DDPG) algorithm frequently exhibits a notable reduction in success rate when transferred to new environments after being trained in complex simulation settings. To address these issues, this paper adopts a Multi-Environment (Multi-Env) parallel training approach and integrates Multi-Head Attention (MHA) and Prioritized Experience Replay (PER) into the DDPG framework, optimizing the reward function to form the MAP-DDPG algorithm. This approach enhances the algorithm’s generalization capability and execution efficiency. Through comparative training and testing of the DDPG and MAP-DDPG algorithms in both simulation and real-world environments, the experimental results demonstrate that MAP-DDPG significantly improves generalization and execution efficiency over the DDPG algorithm. Specifically, in simulation environment tests, the MAP-DDPG algorithm achieved an average 30% increase in success rate and reduced the average time to reach the target point by 23.7 s compared to the DDPG algorithm. These results indicate that the MAP-DDPG algorithm significantly enhances path planning generalization and execution efficiency, providing a more effective solution for path planning in complex environments.