Deep Deterministic Policy Gradient Algorithm Research Articles

The Optimal Strategies of Maneuver Decision in Air Combat of UCAV Based on the Improved TD3 Algorithm

Nowadays, unmanned aerial vehicles (UAVs) pose a significant challenge to air defense systems. Unmanned combat aerial vehicles (UCAVs) have been proven to be an effective method to counter the threat of UAVs in application. Therefore, maneuver decision-making has become the crucial technology to achieve autonomous air combat for UCAVs. In order to solve the problem of maneuver decision-making, an autonomous model of UCAVs based on the deep reinforcement learning method was proposed in this paper. Firstly, the six-degree-of-freedom (DoF) dynamic model was built in three-dimensional space, and the continuous actions of tangential overload, normal overload, and roll angle were selected as the maneuver inputs. Secondly, to improve the convergence speed for the deep reinforcement learning method, the idea of “scenario-transfer training” was introduced into the twin delayed deep deterministic (TD3) policy gradient algorithm, the results showing that the improved algorithm could cut off about 60% of the training time. Thirdly, for the “nose-to-nose turns”, which is one of the classical maneuvers for experienced pilots, the optimal maneuver generated by the proposed method was analyzed. The results showed that the maneuver strategy obtained by the proposed method was highly consistent with that made by experienced fighter pilots. This is also the first time in a public article that compared the maneuver decisions made by the deep reinforcement learning method with experienced fighter pilots. This research can provide some meaningful references to generate autonomous decision-making strategies for UCAVs.

A Wide-Range TCSC Based ADN in Mountainous Areas Considering Hydropower-Photovoltaic-ESS Complementarity

Due to the radial network structures, small cross-sectional lines, and light loads characteristic of existing AC distribution networks in mountainous areas, the development of active distribution networks (ADNs) in these regions has revealed significant issues with integrating distributed generation (DGs) and consuming renewable energy. Focusing on this issue, this paper proposes a wide-range thyristor-controlled series compensation (TCSC)-based ADN and presents a deep reinforcement learning (DRL)-based optimal operation strategy. This strategy takes into account the complementarity of hydropower, photovoltaic (PV) systems, and energy storage systems (ESSs) to enhance the capacity for consuming renewable energy. In the proposed ADN, a wide-range TCSC connects the sub-networks where PV and hydropower systems are located, with ESSs configured for each renewable energy generation. The designed wide-range TCSC allows for power reversal and improves power delivery efficiency, providing conditions for the optimization operation. The optimal operation issue is formulated as a Markov decision process (MDP) with continuous action space and solved using the twin delayed deep deterministic policy gradient (TD3) algorithm. The optimal objective is to maximize the consumption of renewable energy sources (RESs) and minimize line losses by coordinating the charging/discharging of ESSs with the operation mode of the TCSC. The simulation results demonstrate the effectiveness of the proposed method.

Research on the Motion Control Strategy of a Lower-Limb Exoskeleton Rehabilitation Robot Using the Twin Delayed Deep Deterministic Policy Gradient Algorithm

The motion control system of a lower-limb exoskeleton rehabilitation robot (LLERR) is designed to assist patients in lower-limb rehabilitation exercises. This research designed a motion controller for an LLERR-based on the Twin Delayed Deep Deterministic policy gradient (TD3) algorithm to control the lower-limb exoskeleton for gait training in a staircase environment. Commencing with the establishment of a mathematical model of the LLERR, the dynamics during its movement are systematically described. The TD3 algorithm is employed to plan the motion trajectory of the LLERR’s right-foot sole, and the target motion curve of the hip (knee) joint is deduced inversely to ensure adherence to human physiological principles during motion execution. The control strategy of the TD3 algorithm ensures that the movement of each joint of the LLERR is consistent with the target motion trajectory. The experimental results indicate that the trajectory tracking errors of the hip (knee) joints are all within 5°, confirming that the LLERR successfully assists patient in completing lower-limb rehabilitation training in a staircase environment. The primary contribution of this study is to propose a non-linear control strategy tailored for the staircase environment, enabling the planning and control of the lower-limb joint motions facilitated by the LLERR.

Research on Obstacle Avoidance Strategy of Automated Heavy Vehicle Platoon in High-Speed Scenarios

Amidst the rapid progression of the road transport industry, the safety and efficiency of heavy-vehicle platoons have garnered significant attention. The study tackles the challenge of obstacle avoidance presented by vehicles owing to their considerable mass, delayed response times, and line-of-sight impediments, by introducing a cooperative obstacle avoidance system for heavy-vehicle platoons based on deep reinforcement learning. The system comprises three primary modules: perception, decision-making, and control. Initially, the perception module acquires real-time environmental data. Subsequently, the decision-making module formulates obstacle avoidance decisions based on the acquired data. Specifically, it implements a two-stage braking obstacle avoidance strategy under low collision risk scenarios, while employing a fifth-degree polynomial for planning and tracking obstacle avoidance paths under high collision risk conditions suitable for steering maneuvers. The control module utilizes the local multi-agent deep deterministic policy gradient (LADDPG) algorithm to train the heavy-vehicle platoon agents, ensuring the formation’s maintenance while mitigating collisions with other vehicles and obstacles. The effectiveness of the proposed system is substantiated through simulation experiments, demonstrating its adaptability to various traffic conditions, selection of suitable obstacle avoidance strategies, and significant enhancement of obstacle avoidance performance and heavy-vehicle platoon stability.

Data-driven two-stage scheduling of multi-energy systems for operational flexibility enhancement

The multi-energy system, encompassing electricity networks, district heating networks (DHNs), and hydrogen-enriched compressed natural gas (HCNG) networks, provides an alternative for enhancing operational flexibility. This paper investigates a data-driven two-stage scheduling strategy of multi-energy system to promote uncertain renewable energy integration and improve economic benefits. Dynamic models for DHNs and HCNG networks are established, and the flexibility of multi-energy system is quantified through distribution-level power aggregation. To compromise flexibility enhancement and operational cost reduction, the multiobjective day-ahead scheduling optimization is conducted based on adaptive batch-ParEGO method. Taking day-ahead scheduling as a baseline, a data-driven real-time dispatch method is proposed based on the deep deterministic policy gradient algorithm, which adaptively modifies the energy management scheme in smaller temporal dimension to address uncertainties on both the load and source sides. The performance of the proposed two-stage scheduling method is demonstrated through numerical tests.

Learning-based hierarchical cooperative eco-driving with traffic flow prediction for hybrid electric vehicles

The integration of autonomous driving and hybrid electric vehicle technologies presents a promising solution for achieving environmental sustainability. This paper introduces an innovative energy-efficient driving strategy for hybrid electric vehicles that incorporates real-time traffic flow prediction. The study delves into the impact of both lateral and longitudinal vehicle maneuvers on energy consumption within dynamic traffic environments, offering novel insights into optimizing energy utilization. Firstly, a multi-lane traffic flow state rolling predictor is constructed based on the Hankel dynamic mode decomposition algorithm. Subsequently, a vehicle longitudinal and lateral coordinated control strategy is established by integrating the prioritized experience replay double deep Q-network algorithm. Finally, a novel energy management strategy is proposed that leverages Simulink dynamic model and the deep deterministic policy gradient algorithm to address the vehicle dynamic decision-making planning results. Within a hierarchical cooperative optimization framework, this research comprehensively considers safety, comfort, traffic efficiency, and fuel economy. By introducing a novel hierarchical collaborative ecological driving framework, we have achieved a substantial improvement in environmental sustainability, with traffic efficiency increasing by 10.27%-14.41% and fuel economy rising by 9.44%-10.47%. Hardware-in-the-loop validation has confirmed the proposed approach’s real-time capabilities and promising practical applications.

Research on priority scheduling strategy for smoothing power fluctuations of microgrid tie‐lines based on PER‐DDPG algorithm

AbstractThe variability of renewable energy within microgrids (MGs) necessitates the smoothing of power fluctuations through the effective scheduling of internal power equipment. Otherwise, significant power variations on the tie‐line connecting the MG to the main power grid could occur. This study introduces an innovative scheduling strategy that utilizes a data‐driven approach, employing a deep reinforcement learning algorithm to achieve this smoothing effect. The strategy prioritizes the scheduling of MG's internal power devices, taking into account the stochastic charging patterns of electric vehicles. The scheduling optimization model is initially described as a Markov decision process with the goal of minimizing power fluctuations on the interconnection lines and operational costs of the MG. Subsequently, after preprocessing the historical operational data of the MG, an enhanced scheduling strategy is developed through a neural network learning process. Finally, the results from four scheduling scenarios demonstrate the significant impact of the proposed strategy. Comparisons of reward curves before and after data preprocessing underscore its importance. In contrast to optimization results from deep deterministic policy gradient, soft actor‐critic, and particle swarm optimization algorithms, the superiority of the deep deterministic policy gradient algorithm with the addition of a priority experience replay mechanism is highlighted.

Adaptive Control of Ships’ Oil-Fired Boilers Using Flame Image-Based IMC-PID and Deep Reinforcement Learning

The control system of oil-fired boiler units on ships plays a crucial role in reducing the emissions of atmospheric pollutants such as nitrogen oxides (NOx), sulfur dioxides (SO2), and carbon dioxide (CO2). Traditional control methods using conventional measurement sensors face limitations in real-time control due to response delays, which has led to the growing interest in combustion control methods using flame images. To ensure the precision of such combustion control systems, the system model must be thoroughly considered during controller design. However, finding the optimal tuning point is challenging due to the changes in the system model and nonlinearity caused by environmental variations. This study proposes a controller that integrates an internal model control (IMC)-based PID controller with the deep deterministic policy gradient (DDPG) algorithm of deep reinforcement learning to enhance the adaptability of image-based combustion control systems to environmental changes. The proposed controller adjusts the PID parameter values in real-time through the learning of the determination constant lambda (λ) of the IMC internal model. This approach reduces computational resources by shrinking the learning dimensions of the DDPG agent and limits transient responses through constrained learning of control parameters. Experimental results show that the proposed controller exhibited rapid adaptive performance in the learning process for the target oxygen concentration, achieving a reward value of −0.05 within just 105 episodes. Furthermore, when compared to traditional PID tuning methods, the proposed controller demonstrated superior performance, achieving a target value error of 0.0032 and a low overshoot range of 0.0498 to 0.0631, providing the fastest response speed and minimal oscillation. Additionally, experiments conducted on an actual operating ship verified the practical feasibility of this system, highlighting its potential for real-time control and pollutant reduction in marine applications.

Optimizing Energy Harvesting in Wireless Body Area Networks: A Deep Reinforcement Learning Approach to Dynamic Sampling

In energy-harvesting wireless body area networks (EH-WBAN), the self-sustainability of body sensor nodes without compromising the service quality criteria is very important. In remote vital signs monitoring applications, the sampling rate of body nodes in each period should be determined to achieve this goal. There are two fundamental challenges in determining the sampling rate: 1) In EH-WBAN, the rate of the harvestable energy is time-dependent and unpredictable. 2) The vital signs of the patient have different change rates relevant to the time. Therefore, a technique is required that can learn the changes of the harvestable energy and the change rates of vital signs simultaneously and specify the proper sampling rate for BNs. The reinforcement learning algorithms are among the efficient solutions for this problem because they are capable of learning environmental uncertainties. Previous RL-based methods for determining the BN's sampling rate have three fundamental problems: 1) focus on the optimization of the transmission energy and ignore the sensing energy, 2) only affect one of the two aspects of energy or sensed data in determining the sampling rate and 3) discretization of the problem space does not ensure the determination of the optimal sampling rate. Therefore, this paper proposes a method named “deep reinforcement learning-based dynamic sampling” (DRDS) which first formulates the sampling rate determination problem as a Markov decision process (MDP) and proposes a deep deterministic policy gradient algorithm (DDPG) to solve it. The proposed method considers both energy and data variability aspects in determining the sampling rate. Two parameters, the super-capacitor’s voltage level, and ambient light intensity, are considered for the energy aspect; for the data variability aspect, the long short-term memory (LSTM) algorithm is developed to predict the change rate of data in the next round. Simulations indicate that by preserving the data integrity, the proposed method can decrease the sampling rate and unnecessary data transmission by about 49.95% and 89.7%, respectively, compared with state-of-the-art methods, and allow the sensor to achieve self-sustainability.

Method for remaining useful life prediction of rolling bearings based on deep reinforcement learning.

In contemporary industrial systems, the prediction of remaining useful life (RUL) is recognized as a valuable maintenance strategy for health management due to its ability to monitor equipment operational status in real time and ensure the safety of industrial production. Current studies have largely concentrated on deep learning (DL) techniques, leading to a shortage of RUL prediction methods that utilize deep reinforcement learning (DRL). To further enhance application and research, this paper introduces a novel approach to RUL prediction based on DRL, specifically using a combination of Convolutional Neural Network-Bidirectional Long Short-Term Memory Network (CNN-BiLSTM) and the Deep Deterministic Policy Gradient (DDPG) algorithm. The proposed method reframes the conventional task of estimating RUL as a Markov decision process (MDP), effectively integrating the feature extraction capabilities of DL with the decision-making abilities of DRL. Initially, a hybrid CNN-BiLSTM is employed to establish an agent that can extract degradation features from raw signals. Subsequently, the DDPG algorithm within DRL is leveraged to develop the RUL prediction mechanism, completing the MDP by defining appropriate action spaces and reward functions. The agent, through repeated trials and optimization, learns to map the current operational state of the rolling bearing to its remaining service life. Validation analysis was performed on the intelligent maintenance systems (IMS) bearing dataset. The findings suggest that the DRL-based approach outperforms the current methodologies, demonstrating a superior performance in root mean square error (MSE) and MSE metrics. The predicted outcomes align more closely with the actual lifespan values.

Mapless Path Planning for Mobile Robot Based on Improved Deep Deterministic Policy Gradient Algorithm.

In the traditional Deep Deterministic Policy Gradient (DDPG) algorithm, path planning for mobile robots in mapless environments still encounters challenges regarding learning efficiency and navigation performance, particularly adaptability and robustness to static and dynamic obstacles. To address these issues, in this study, an improved algorithm frame was proposed that designs the state and action spaces, and introduces a multi-step update strategy and a dual-noise mechanism to improve the reward function. These improvements significantly enhance the algorithm's learning efficiency and navigation performance, rendering it more adaptable and robust in complex mapless environments. Compared to the traditional DDPG algorithm, the improved algorithm shows a 20% increase in the stability of the navigation success rate with static obstacles along with a 25% reduction in pathfinding steps for smoother paths. In environments with dynamic obstacles, there is a remarkable 45% improvement in success rate. Real-world mobile robot tests further validated the feasibility and effectiveness of the algorithm in true mapless environments.

An Alternative Reinforcement Learning (ARL) control strategy for data center air-cooled HVAC systems

Energy efficiency of data center is of great concern globally due to their large amount of energy consumption and the foreseeable growth in the demand of digital services in the future. Advanced control strategies are needed to reduce energy consumption. However, the optimization of data center HVAC control is a challenge task due to the complexity of the thermal dynamic models of buildings and uncertainties associated with both server loads and outdoor temperature. In this paper, we propose a new control strategy called Alternately-Reinforcement Learning-control(ARL), which realizes the alternating control of RL and proportional, integral and derivative (PID) for optimizing the control of air-cooled HVAC systems in data centers. The control object of the ARL is the speed set of the compressor and the condensing fan, and the control goal is to minimize energy consumption while maintaining temperature stability. The applied RL algorithm is Deep deterministic policy gradient (DDPG) and Proximal policy optimization (PPO). We pre-train the ARL strategy in offline environment firstly and deploy it in real data center environment for online testing. The test results show that the ARL has significant advantages in maintaining temperature stability while reducing energy consumption, and the PPO algorithm performs better than the DDPG algorithm. Compared with the PID algorithm, the PPO algorithm can save energy by 5.27%, and the temperature control effect can be improved by 3.27%,which indicates the feasibility of the ARL for implementation in real data centers.

Collaborative task decision-making of multi-UUV in dynamic environments based on deep reinforcement learning

ABSTRACT The problem of collaborative task decision-making for unmanned underwater vehicles (UUVs) in an unknown dynamic ocean environment is investigated. In view of the unknown marine environment, a partially observable Markov decision process is designed to achieve partially observable path planning for multi-UUV. Aiming at the problems of long planning time, large amount of data, and insufficient multi-task decision-making capability in the process of multi-UUV underwater collaborative operation, a multi-UUV collaborative task decision-making model based on the multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm for the dynamic environment is constructed. The implementation of the centralised training with distributed execution (CT-DE) training framework enables each UUV to possess autonomous decision-making capability and ensures task safety under weak communication conditions. The pre-training phase shows that the MATD3 algorithm outperforms the multi-agent deep deterministic policy gradient (MADDPG) algorithm for training multi-UUV autonomous decision-making scenarios in dynamic environments. The simulation experiment results verify that the autonomous decision-making method based on the MATD3 algorithm can effectively solve the multi-UUV collaborative task decision-making problem in unknown dynamic environments, and the task process satisfies the requirements of real-time, safety and economy.

Integrated energy cluster hierarchical regulation technology considering demand response

Integrated Energy Cluster (IEC), the regional aggregation of integrated energy systems (IES), has accumulated plenty of dispatchable resources with the development of energy market. This, while significantly providing system Demand Response (DR) potential, also complicates the interaction of the IEC with the main grid and increases the difficulty of system scheduling. To address this issue, this paper proposes a Reinforcement Learning-driven multi-agent hierarchical regulation framework that makes full use of DR to maximize the benefits of both IEC and main grid. Firstly, in the context of the DR market, a mechanism for IECs to bid in the real-time DR market is proposed. Furthermore, an "IEC-main network" hierarchical regulation model taking account of DR is established to minimize the IEC operation cost and maximize the societal benefit. Moreover, an optimization algorithm utilizing Deep Deterministic Policy Gradient (DDPG) with Multi-process (MP) and Priority Experience Replay (PER) mechanism is proposed to allow adaptability to high-latitude and large-scale applications. In case study, the proposed model and algorithm is tested on an 8-node system and a 24-node system. The result indicates that the hierarchical regulation model considering DR can improve the system economy by 2.59 % than that without DR and the improved DDPG algorithm can enhance training effectiveness in comparison with DDPG and PER-DDPG.

A soft actor-critic reinforcement learning framework for optimal energy management in electric vehicles with hybrid storage

The efficient energy management of electric vehicles (EVs) equipped with hybrid energy storage systems (HESS) poses a significant challenge due to its vast search space, numerous control variables, and intricate driving conditions. In response to this challenge, this paper introduces a novel deep reinforcement learning (DRL) algorithm, specifically the soft actor-critic (SAC), tailored for optimizing energy distribution within EVs equipped with battery-supercapacitor (SC) HESS. The proposed SAC-based energy management system (EMS) is designed to address inherent limitations present in most existing DRL algorithms, such as slower convergence rates, discretization errors, unstable training dynamics, and suboptimal optimization effects. The SAC-based EMS undergoes training in a continuous action space through self-play, utilizing a complex driving cycle and a novel reward function. The algorithm demonstrates its efficiency by rapidly maximizing cumulative rewards and refining its decision-making policies. Afterward, extensive experiments showcase the superiority of the proposed SAC-based EMS over rule-based (RB) techniques, deep deterministic policy gradient (DDPG), and battery-only configurations across various driving cycles. Notably, the approach effectively allocates high-power surges and braking energy to the SC, reducing the frequency of charge/discharge cycles and thereby extending the battery's lifespan. Additionally, the learned SAC-based EMS policy achieves substantial reductions in electricity consumption by 39.6 %, 47.87 %, and 45 % compared to battery-only configuration, and 42.56 %, 47.87 %, and 45.83 % compared to RB technique under the NYCC, LA92, and FTP cycles, respectively. In contrast, the DDPG algorithm tends to rely heavily on the SC to deliver the total power required, even under normal driving conditions.

Automated architectural spatial composition via multi-agent deep reinforcement learning for building renovation

This paper focuses on utilizing deep reinforcement learning technology to explore how to achieve automated architectural space composition under established built environment conditions to address the extensive demands of old building renovation. A reinforcement learning platform has been develop0065d, centered around the multi-agent deep deterministic policy gradient (MADDPG) algorithm. The paper leverages functional blocks as agents within Grasshopper to simulate existing architectural structures and their interactions, establishing a task-based reward system to guide the design process. Furthermore, data exchange between algorithmic and modeling software is facilitated through UDP communication. Training results using typical frame structure spaces indicate a significant increase in cumulative rewards around the 650,000th training step, effectively achieving the given design tasks. The paper demonstrates the potential of automated architectural space composition methods, based on deep reinforcement learning, to enhance design efficiency, optimize spatial arrangements, and promote human-machine collaborative design in building renovations.

Anti-Jamming Resource-Allocation Method in the EH-CIoT Network through LWDDPG Algorithm.

In the Energy-Harvesting (EH) Cognitive Internet of Things (EH-CIoT) network, due to the broadcast nature of wireless communication, the EH-CIoT network is susceptible to jamming attacks, which leads to a serious decrease in throughput. Therefore, this paper investigates an anti-jamming resource-allocation method, aiming to maximize the Long-Term Throughput (LTT) of the EH-CIoT network. Specifically, the resource-allocation problem is modeled as a Markov Decision Process (MDP) without prior knowledge. On this basis, this paper carefully designs a two-dimensional reward function that includes throughput and energy rewards. On the one hand, the Agent Base Station (ABS) intuitively evaluates the effectiveness of its actions through throughput rewards to maximize the LTT. On the other hand, considering the EH characteristics and battery capacity limitations, this paper proposes energy rewards to guide the ABS to reasonably allocate channels for Secondary Users (SUs) with insufficient power to harvest more energy for transmission, which can indirectly improve the LTT. In the case where the activity states of Primary Users (PUs), channel information and the jamming strategies of the jammer are not available in advance, this paper proposes a Linearly Weighted Deep Deterministic Policy Gradient (LWDDPG) algorithm to maximize the LTT. The LWDDPG is extended from DDPG to adapt to the design of the two-dimensional reward function, which enables the ABS to reasonably allocate transmission channels, continuous power and work modes to the SUs, and to let the SUs not only transmit on unjammed channels, but also harvest more RF energy to supplement the battery power. Finally, the simulation results demonstrate the validity and superiority of the proposed method compared with traditional methods under multiple jamming attacks.

VN-MADDPG: A Variable-Noise-Based Multi-Agent Reinforcement Learning Algorithm for Autonomous Vehicles at Unsignalized Intersections

The decision-making performance of autonomous vehicles tends to be unstable at unsignalized intersections, making it difficult for them to make optimal decisions. We propose a decision-making model based on the Variable-Noise Multi-Agent Deep Deterministic Policy Gradient (VN-MADDPG) algorithm to address these issues. The variable-noise mechanism reduces noise dynamically, enabling the agent to utilize the learned policy more effectively to complete tasks. This significantly improves the stability of the decision-making model in making optimal decisions. The importance sampling module addresses the inconsistency between outdated experience in the replay buffer and current environmental features. This enhances the model’s learning efficiency and improves the robustness of the decision-making model. Experimental results on the CARLA simulation platform show that the success rate of decision making at unsignalized intersections by autonomous vehicles has significantly increased, and the pass time has been reduced. The decision-making model based on the VN-MADDPG algorithm demonstrates stable and excellent decision-making performance.

Bio-inspired distributed load frequency control in Islanded Microgrids: A multi-agent deep reinforcement learning approach

To prevent errors in load frequency control (LFC) decision-making in an islanded microgrid and reduce the waste of frequency regulation resources, a fully distributed “starfish” load frequency control method is proposed. For this method, a novel deep reinforcement learning algorithm called the distributed decomposed multirole multiagent deep deterministic policy gradient (DDMR-MADDPG) algorithm is introduced. This algorithm treats each unit as an agent, enabling centralized training of all the agents. Through the coordination and control of multiple agents, a global optimal strategy can be obtained. During online application, the algorithm employs a distributed implementation strategy, enabling each unit to make decisions autonomously. This decision-making strategy is akin to the distributed neural network of a starfish, and it can fundamentally improve the efficiency and correctness of decision-making. In addition, it employs multiple techniques, including value function decomposition, multirole learning, imitation learning, and curriculum learning approaches. The performance of the proposed method compared with 24 existing methods is demonstrated on a simulated LFC model of the Zhuzhou Island microgrid in China.

Geological adaptive intelligent control of earth pressure balance shield machine based on deep reinforcement learning

Scientific and precise control of tunnelling parameters is of utmost importance during the construction of shield machines. Given the complexity of the working environment, manual operation is highly prone to causing safety accidents. Therefore, achieving intelligent control of the shield machine is crucial. Based on this, this paper proposes a geological adaptive intelligent control method of earth pressure balance shield machine using the Deep Deterministic Policy Gradient (DDPG) algorithm as the framework, with Actor-Critic as the basis. Firstly, DDPG agent is constructed to replace the screw conveyor control system as the main body of strategy implementation. Secondly, an environmental model is established by utilizing the mechanism model between the sealed cabin pressure and the screw conveyor speed. The real-time sealed cabin pressure, target pressure, and pressure error serve as the state space, while the screw conveyor speed is used as the action space. A combined reward function is set based on safety and accuracy. Finally, the Actor network interacts with the environment under the supervision of the reward function and Critic network. Successful training is achieved when the cumulative reward value is maximized, resulting in the output of optimal control strategy. In this paper, the method dynamically regulates the screw conveyor speed by interacting with the geological environment, to realize the precise control of the sealed cabin pressure and ensure the dynamic balance between sealed cabin pressure and excavation face pressure. The test results show that this method has a good control effect on the sealed cabin pressure under various geological conditions, and can complete 72 kinds of soil transition tasks. It has strong soil adaptability and can respond well to the dynamic changes of soil conditions. This approach enhances the intelligence of the shield machine, mitigating inaccuracies attributed to human operation, which provides a guarantee of safe shield machine operation, whilst exhibiting valuable engineering applications.