Deep Deterministic Policy Gradient Algorithm Research Articles

Deep Reinforcement Learning for Ecological and Distributed Urban Traffic Signal Control with Multi-Agent Equilibrium Decision Making

Multi-Agent Reinforcement Learning (MARL) has shown strong advantages in urban multi-intersection traffic signal control, but it also suffers from the problems of non-smooth environment and inter-agent coordination. However, most of the existing research on MARL traffic signal control has focused on designing efficient communication to solve the environment non-smoothness problem, while neglecting the coordination between agents. In order to coordinate among agents, this paper combines MARL and the regional mixed-strategy Nash equilibrium to construct a Deep Convolutional Nash Policy Gradient Traffic Signal Control (DCNPG-TSC) model, which enables agents to perceive the traffic environment in a wider range and achieves effective agent communication and collaboration. Additionally, a Multi-Agent Distributional Nash Policy Gradient (MADNPG) algorithm is proposed in this model, which is the first time the mixed-strategy Nash equilibrium is used for the improvement in the Multi-Agent Deep Deterministic Policy Gradient algorithm traffic signal control strategy to provide the optimal signal phase for each intersection. In addition, the eco-mobility concept is integrated into MARL traffic signal control to reduce pollutant emissions at intersections. Finally, simulation results in synthetic and real-world traffic road networks show that DCNPG-TSC outperforms other state-of-the-art MARL traffic signal control methods in almost all performance metrics, because it can aggregate the information of neighboring agents and optimize the agent’s decisions through gaming to find an optimal joint equilibrium strategy for the traffic road network.

Joint Phase Shift Design and Resource Management for a Non-Orthogonal Multiple Access-Enhanced Internet of Vehicle Assisted by an Intelligent Reflecting Surface-Equipped Unmanned Aerial Vehicle

This paper integrates intelligent reflecting surfaces (IRS) with unmanned aerial vehicles (UAV) to enhance the transmission performance of the Internet of Vehicles (IoV) through non-orthogonal multiple access (NOMA). It focuses on strengthening the signals from cell edge vehicles (CEVs) to the base station by optimizing the wireless propagation environment via an IRS-equipped UAV. The primary goal is to maximize the sum data rate of CEVs while satisfying the constraint of the successive interference cancellation (SIC) decoding threshold. The challenge lies in the non-convex nature of jointly considering the power control, subcarrier allocation, and phase shift design, making the problem difficult to optimally solve. To address this, the problem is decomposed into two independent subproblems, which are then solved iteratively. Specifically, the optimal phase shift design is achieved using the deep deterministic policy gradient (DDPG) algorithm. Furthermore, the graph theory is applied to determine the subcarrier allocation policy and derive a closed-form solution for optimal power control. Finally, the simulation results show that the proposed joint phase shift and resource management scheme significantly enhances the sum data rate compared to the state-of-the-art schemes, thereby demonstrating the benefits of integrating the IRS-equipped UAV into NOMA-enhanced IoV.

Distributed auto disturbances rejection resilient control of permanent magnetic maglev trains based on the optimized deep deterministic policy gradient algorithm

Distributed auto disturbances rejection resilient control of permanent magnetic maglev trains based on the optimized deep deterministic policy gradient algorithm

A Pilot Study of Observation Poisoning on Selective Reincarnation in Multi-Agent Reinforcement Learning

This research explores the vulnerability of selective reincarnation, a concept in Multi-Agent Reinforcement Learning (MARL), in response to observation poisoning attacks. Observation poisoning is an adversarial strategy that subtly manipulates an agent’s observation space, potentially leading to a misdirection in its learning process. The primary aim of this paper is to systematically evaluate the robustness of selective reincarnation in MARL systems against the subtle yet potentially debilitating effects of observation poisoning attacks. Through assessing how manipulated observation data influences MARL agents, we seek to highlight potential vulnerabilities and inform the development of more resilient MARL systems. Our experimental testbed was the widely used HalfCheetah environment, utilizing the Independent Deep Deterministic Policy Gradient algorithm within a cooperative MARL setting. We introduced a series of triggers, namely Gaussian noise addition, observation reversal, random shuffling, and scaling, into the teacher dataset of the MARL system provided to the reincarnating agents of HalfCheetah. Here, the “teacher dataset” refers to the stored experiences from previous training sessions used to accelerate the learning of reincarnating agents in MARL. This approach enabled the observation of these triggers’ significant impact on reincarnation decisions. Specifically, the reversal technique showed the most pronounced negative effect for maximum returns, with an average decrease of 38.08% in Kendall’s tau values across all the agent combinations. With random shuffling, Kendall’s tau values decreased by 17.66%. On the other hand, noise addition and scaling aligned with the original ranking by only 21.42% and 32.66%, respectively. The results, quantified by Kendall’s tau metric, indicate the fragility of the selective reincarnation process under adversarial observation poisoning. Our findings also reveal that vulnerability to observation poisoning varies significantly among different agent combinations, with some exhibiting markedly higher susceptibility than others. This investigation elucidates our understanding of selective reincarnation’s robustness against observation poisoning attacks, which is crucial for developing more secure MARL systems and also for making informed decisions about agent reincarnation.

Reinforcement learning-based order-dispatching optimization in the ride-sourcing service

Since the emergence of ride-sourcing, the dispatching problem has been a hot research issue in the travel market. Previous studies designed various refined algorithms to improve the dispatching efficiency. However, the heterogeneity of passenger cancellation behavior and driver's work patterns is ignored in the modeling of these studies, which may result in the algorithms failing to entirely capture the randomness in the real world, thereby affecting dispatching efficiency. To fill this research gap, this paper employs a discrete choice model and Gaussian mixture model to model passenger cancellation behavior and extract driver working patterns, respectively. Furthermore, a cooperative deep deterministic policy gradient (DDPG) algorithm is proposed for dispatching. The algorithm constructs decision networks separately for drivers with different working patterns to learn the optimal dispatching decision and designs a centralized neural network to coordinate the dispatching of all vehicles to maximize the global benefit. A price incentive strategy is proposed to improve passenger loyalty by providing incentive prices. Moreover, based on the constructed discrete choice model, a simulation environment is designed to describe the cancellation behavior in the real world and support the training of the algorithm. Experimental results based on real data in Beijing show that the cooperative DDPG algorithm can improve the order response rate by 2% to 6% and platform revenue by 3% to 9% in multiple travel periods while providing a better travel experience to passengers. Additionally, the price incentive strategy also effectively reduces passenger cancellation.

Multi-agent reinforcement learning based computation offloading and resource allocation for LEO Satellite edge computing networks

Due to the limitations caused by geographical conditions and economic requirements, it is difficult to provide computing services by terrestrial networks for mobile terminals in remote areas. To address this issue, mobile edge computing (MEC) servers can be deployed in the low earth orbit (LEO) satellites to act as a complement and accommodate the unserved terminals. However, offloading computing tasks to servers in satellites may increase the energy consumption of ground terminals. Considering the limited battery capacity of ground terminals, how to perform the computation offloading and resource allocation are key challenges in the LEO satellite edge computing networks. Therefore, in this paper, we investigate the energy minimization problem for LEO satellite edge computing networks, where a multi-agent deep reinforcement learning algorithm with global rewards is proposed to optimize the transmit power, CPU frequency, bit allocation, offloading decision and bandwidth allocation via a decentralized method. Simulation results show that our proposed algorithm can converge faster. Most importantly, compared with the random algorithm, the proximal policy optimization (PPO) algorithm, and the deep deterministic policy gradient (DDPG) algorithm, the ground terminals’ energy consumption can be effectively reduced by our proposed algorithm.

VizNav: A Modular Off-Policy Deep Reinforcement Learning Framework for Vision-Based Autonomous UAV Navigation in 3D Dynamic Environments

Unmanned aerial vehicles (UAVs) provide benefits through eco-friendliness, cost-effectiveness, and reduction of human risk. Deep reinforcement learning (DRL) is widely used for autonomous UAV navigation; however, current techniques often oversimplify the environment or impose movement restrictions. Additionally, most vision-based systems lack precise depth perception, while range finders provide a limited environmental overview, and LiDAR is energy-intensive. To address these challenges, this paper proposes VizNav, a modular DRL-based framework for autonomous UAV navigation in dynamic 3D environments without imposing conventional mobility constraints. VizNav incorporates the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with Prioritized Experience Replay and Importance Sampling (PER) to improve performance in continuous action spaces and mitigate overestimations. Additionally, VizNav employs depth map images (DMIs) to enhance visual navigation by accurately estimating objects’ depth information, thereby improving obstacle avoidance. Empirical results show that VizNav, by leveraging TD3, improves navigation, and the inclusion of PER and DMI further boosts performance. Furthermore, the deployment of VizNav across various experimental settings confirms its flexibility and adaptability. The framework’s architecture separates the agent’s learning from the training process, facilitating integration with various DRL algorithms, simulation environments, and reward functions. This modularity creates a potential to influence RL simulation in various autonomous navigation systems, including robotics control and autonomous vehicles.

Trajectory Tracking Control of Variable Sweep Aircraft Based on Reinforcement Learning

An incremental deep deterministic policy gradient (IDDPG) algorithm is devised for the trajectory tracking control of a four-wing variable sweep (FWVS) aircraft with uncertainty. The IDDPG algorithm employs the line-of-sight (LOS) method for path tracking, formulates a reward function based on position and attitude errors, and integrates long short-term memory (LSTM) units into IDDPG algorithm to enhance its adaptability to environmental changes during flight. Finally, environmental disturbance factors are introduced in simulation to validate the designed controller’s ability to track climbing trajectories of morphing aircraft in the presence of uncertainty.

Deep Reinforcement Learning Lane-Changing Decision Algorithm for Intelligent Vehicles Combining LSTM Trajectory Prediction

Intelligent decisions for autonomous lane-changing in vehicles have consistently been a focal point of research in the industry. Traditional lane-changing algorithms, which rely on predefined rules, are ill-suited for the complexities and variabilities of real-world road conditions. In this study, we propose an algorithm that leverages the deep deterministic policy gradient (DDPG) reinforcement learning, integrated with a long short-term memory (LSTM) trajectory prediction model, termed as LSTM-DDPG. In the proposed LSTM-DDPG model, the LSTM state module transforms the observed values from the observation module into a state representation, which then serves as a direct input to the DDPG actor network. Meanwhile, the LSTM prediction module translates the historical trajectory coordinates of nearby vehicles into a word-embedding vector via a fully connected layer, thus providing predicted trajectory information for surrounding vehicles. This integrated LSTM approach considers the potential influence of nearby vehicles on the lane-changing decisions of the subject vehicle. Furthermore, our study emphasizes the safety, efficiency, and comfort of the lane-changing process. Accordingly, we designed a reward and penalty function for the LSTM-DDPG algorithm and determined the optimal network structure parameters. The algorithm was then tested on a simulation platform built with MATLAB/Simulink. Our findings indicate that the LSTM-DDPG model offers a more realistic representation of traffic scenarios involving vehicle interactions. When compared to the traditional DDPG algorithm, the LSTM-DDPG achieved a 7.4% increase in average single-step rewards after normalization, underscoring its superior performance in enhancing lane-changing safety and efficiency. This research provides new ideas for advanced lane-changing decisions in autonomous vehicles.

Deep Reinforcement Learning-Based Task Offloading and Load Balancing for Vehicular Edge Computing

Vehicular edge computing (VEC) effectively reduces the computational burden on vehicles by offloading tasks from resource-constrained vehicles to edge nodes. However, non-uniformly distributed vehicles offloading a large number of tasks cause load imbalance problems among edge nodes, resulting in performance degradation. In this paper, we propose a deep reinforcement learning-based decision scheme for task offloading and load balancing with the optimization objective of minimizing the system cost considering the split offloading of tasks and the load dynamics of edge nodes. First, we model the mutual interaction between mobile vehicles and Mobile Edge Computing (MEC) servers using a Markov decision process. Second, the optimal task-offloading and resource allocation decision is obtained by utilizing the twin delayed deep deterministic policy gradient algorithm (TD3), and server load balancing is achieved through edge collaboration using a server selection algorithm based on the technique for order preference by similarity to the ideal solution (TOPSIS). Finally, we have conducted extensive simulation experiments and compared the results with several other baseline schemes. The proposed scheme can more effectively reduce the system cost and increase the system resource utilization.

Energy-Delay Joint Optimization for Task Offloading in Digital Twin-Assisted Internet of Vehicles

Vehicle edge computing (VEC) provides efficient services for vehicles by offloading tasks to edge servers. Notably, extant research mainly employs methods such as deep learning and reinforcement learning to make resource allocation decisions, without adequately accounting for the ramifications of high-speed mobility of vehicles and the dynamic nature of the Internet of Vehicles (IoV) on the decision-making process. This paper endeavours to tackle the aforementioned issue through the introduction of a novel concept, namely, a digital twin-assisted IoV. Among them, the digital twin of IoV offers training data for computational offloading and content caching decisions, which allows edge servers to directly interact with the dynamic environment while capturing its dynamic changes in real-time. Through this collaborative endeavour, edge intelligent servers can promptly respond to vehicular requests and return results. We transform the dynamic edge computing problem into a Markov decision process (MDP), and then solve it with the twin delayed deep deterministic policy gradient (TD3) algorithm. Simulation experiments demonstrate the adaptability of our proposed approach in the dynamic environment while successfully enhancing the Quality of Service, that is, decreasing total delay and energy consumption.

A Deep Reinforcement Learning Approach for Optimizing Inventory Management in the Agri-Food Supply Chain

This research aims to improve inventory management throughout the agri-food supply chain, through the use of Deep Reinforcement Learning algorithms. Three Deep Reinforcement Learning algorithms, including Deep Q-Networks, Deep Deterministic Policy Gradient, and Proximal Policy Optimization algorithms were implemented and tested in order to evaluate their ability to actively manage inventories and improve the performance of supply chains. The results of the experimental phase offered important information regarding the performance of each Deep Reinforcement Learning algorithm. The Deep Deterministic Policy Gradient algorithm was identified as a viable choice, offering the best results in terms of accuracy to set the optimal inventories for the supply chain and improving the efficiency of the supply chain. The percentage of cost efficiency improved from 92.5% to 95.8% in the case of the DDPG model. The inventory turnover was also improved, surpassing the level of 8.1 units from the original level of 7.3 units which means that the system converts the inventor into sales in less time. The metric for on-time delivery also beneficiated from several improvements, reaching 96.5% form the level of 93.2% return. The quality metrics also registered a significant improvement and reported level of 96.2% after the implementation of the DDPG algorithm and compared to the level of 94.6% prior to the implementation of the algorithm. These results suggest that using such a system will bring beneficial changes to the supply chain and will offer the possibility of implementing a data-driven inventory system based on Deep Reinforcement Learning.

A Multi-Objective Optimal Control Method for Navigating Connected and Automated Vehicles at Signalized Intersections Based on Reinforcement Learning

The emergence and application of connected and automated vehicles (CAVs) have played a positive role in improving the efficiency of urban transportation and achieving sustainable development. To improve the traffic efficiency at signalized intersections in a connected environment while simultaneously reducing energy consumption and ensuring a more comfortable driving experience, this study investigates a flexible and real-time control method to navigate the CAVs at signalized intersections utilizing reinforcement learning (RL). Initially, control of CAVs at intersections is formulated as a Markov Decision Process (MDP) based on the vehicles’ motion state and the intersection environment. Subsequently, a comprehensive reward function is formulated considering energy consumption, efficiency, comfort, and safety. Then, based on the established environment and the twin delayed deep deterministic policy gradient (TD3) algorithm, a control algorithm for CAVs is designed. Finally, a simulation study is conducted using SUMO, with Lankershim Boulevard as the research scenario. Results indicate that the proposed methods yield a 13.77% reduction in energy consumption and a notable 18.26% decrease in travel time. Vehicles controlled by the proposed method also exhibit smoother driving trajectories.

AoI minimization of ambient backscatter-assisted EH-CRN with cooperative spectrum sensing

To evaluate the timeliness and freshness of information from energy-constrained devices, we minimize the weighted average age of information (AoI) of the ambient backscatter-assisted energy harvesting cognitive radio network (AB-EH-CRN), where secondary transmitters (STs) follow the non-orthogonal multiple access (NOMA) strategy to transmit status updates. The secondary receiver (SR) schedules the selected actions for STs to perform spectrum access, and receives status updates from them. We formulate the AoI minimization problem where the action selection and transmission power of STs are considered as variables. To exploit spectrum resources in the AB-EH-CRN with unknown signal-to-noise ratios (SNRs) at the STs, we propose the deep neural network (DNN)-based cooperative spectrum sensing (CSS), and adopt an energy threshold approach for STs to determine the energy allocation between spectrum sensing (SS) and packet transmission. Moreover, we address the AoI minimization problem by the proposed hybrid action and energy penalty deep deterministic policy gradient (HAEP-DDPG) algorithm, which adapts to the hybrid continuous and discrete action spaces. Simulation results validate the advantage of the proposed HAEP-DDPG algorithm compared with comparison schemes in terms of AoI.

Deep reinforcement learning algorithm based ramp merging decision model

On-ramp merge is a complex traffic scenario in autonomous driving. Because of the uncertainty of the driving environment, most rule-based models cannot solve such a problem. This paper designs a ramp merging decision model based on deep deterministic policy gradient algorithm (DDPG) to solve the vehicle merging problem. To address the problems of slow algorithm merging and poor robustness of previous deep reinforcement learning algorithms in the field of intelligent vehicle ramp merging leading to the low success rate of intelligent vehicle merging, first, we introduce a simple recurrent unit (SRU) for extracting intelligent vehicle states and environment features and use the DDPG algorithm for intelligent vehicle decision making. Second, the experience playback pool of DDPG algorithm is improved by using priority sampling instead of uniform sampling. Finally, a multi-objective reward function is set up during training, considering factors such as safety and efficiency. The simulation experiments show that the improved algorithm improves the merging speed of the model, reduces the collision rate, and enables the vehicle to make more reasonable decisions. In addition, the superiority of the method is demonstrated by comparing with the advanced method.

Fuzzy-based collective pitch control for wind turbine via deep reinforcement learning

Wind turbines (WTs) have highly nonlinear and uncertain dynamics due to aerodynamic complexity, mechanical factors, and fluctuations in wind conditions. Turbulence and wind shear add complexity to modelling, especially in constant power region (region 3). Thus, an effective control design demands a deep understanding of the nonlinearities and uncertainties. This paper suggests a novel model-free reinforcement learning (RL) collective pitch angle controller to operate efficiently in region 3. The proposed controller stabilizes generator speed, maximizes power output, and minimizes fluctuations while accommodating system uncertainties, nonlinearity, and pitch limits. The disparity between WT dynamics due to wind speed perturbations and uncertainties is measured using a gap-metric criterion. The controller design adopts a deep deterministic policy gradient (DDPG) algorithm to train six agents in a medium-fidelity WT environment at different mean wind speeds to ensure the controller's robustness. Initially, imitation learning is used for efficient sample collection to fasten training convergence. Afterwards, the agent learns by interacting with the environment. After the training, the pitch control outputs from multi-trained agents are processed by a fuzzy system to have smooth transitions under different operating conditions. The resulting fuzzy DDPG (F-DDPG) controller is deployed to obtain the optimal pitch control action. The performance of the proposed F-DDPG controller is compared to the gain-scheduled PI (GSPI), Linear-Quadratic-Regulator (LQR), and single-DDPG-agent controllers. The controllers are simulated in high-fidelity onshore and offshore 5-MW WT environments using the OpenFAST/MATLAB simulation tools. The results reveal the superiority of the proposed controller in generalizing its optimal performance in different operating conditions.

Investigating the Impact of Curriculum Learning on Reinforcement Learning for Improved Navigational Capabilities in Mobile Robots

This paper proposes a method for finding the shortest path of a mobile robot using deep reinforcement learning with utilizing Proximal policy optimization algorithm (PPO) enhanced with curriculum learning. By modelling the environment in 3D space using the Webots simulator, we extend the PPO algorithm's capabilities to handle continuous states from 8 IR sensors and control the velocities of two motors of E-puck robot. Our study uniquely integrates curriculum learning into the PPO framework, aiming to improve adaptability and training efficiency in complex environments. A comparative analysis is conducted between the modified PPO, the original PPO, and the deep deterministic policy gradient algorithm, highlighting the strengths of our approach The results demonstrate that our curriculum-augmented PPO algorithm not only accelerates the training process but also shows superior adaptability and generalization in new environments. This work underscores the significant potential of curriculum learning in enhancing the performance of deep reinforcement learning algorithms for robust and efficient robotic navigation.

Energy Management for Hybrid Energy Storage System in Electric Based on Deep Deterministic Policy Gradient

In this paper, an intelligent control system design scheme based on deep deterministic policy gradient (DDPG) algorithm is proposed for the complex continuous action space problem in the hybrid energy storage system of electric vehicles. Firstly, the basic principle and internal logic of DDPG algorithm are introduced, including key elements such as Actor-Critic architecture, experience playback, target network, reward signal, policy gradient and value function update. Then, how to apply the DDPG algorithm to the industrial control system is described in detail. The Actor network learns the optimal strategy, the Critic network evaluates the value of the state-action pair, and uses the experience playback and the target network to improve the system stability and performance. Finally, the effect of the intelligent control system based on DDPG algorithm in complex environment is verified by simulation experiments. The results show that the system can effectively optimize the control strategy, improve the response speed and stability of the system, and has a good engineering application prospect.

Optimization of news dissemination push mode by intelligent edge computing technology for deep learning

The Internet era is an era of information explosion. By 2022, the global Internet users have reached more than 4 billion, and the social media users have exceeded 3 billion. People face a lot of news content every day, and it is almost impossible to get interesting information by browsing all the news content. Under this background, personalized news recommendation technology has been widely used, but it still needs to be further optimized and improved. In order to better push the news content of interest to different readers, users' satisfaction with major news websites should be further improved. This study proposes a new recommendation algorithm based on deep learning and reinforcement learning. Firstly, the RL algorithm is introduced based on deep learning. Deep learning is excellent in processing large-scale data and complex pattern recognition, but it often faces the challenge of low sample efficiency when it comes to complex decision-making and sequential tasks. While reinforcement learning (RL) emphasizes learning optimization strategies through continuous trial and error through interactive learning with the environment. Compared with deep learning, RL is more suitable for scenes that need long-term decision-making and trial-and-error learning. By feeding back the reward signal of the action, the system can better adapt to the unknown environment and complex tasks, which makes up for the relative shortcomings of deep learning in these aspects. A scenario is applied to an action to solve the sequential decision problem in the news dissemination process. In order to enable the news recommendation system to consider the dynamic changes in users' interest in news content, the Deep Deterministic Policy Gradient algorithm is applied to the news recommendation scenario. Opposing learning complements and combines Deep Q-network with the strategic network. On the basis of fully summarizing and thinking, this paper puts forward the mode of intelligent news dissemination and push. The push process of news communication information based on edge computing technology is proposed. Finally, based on Area Under Curve a Q-Leaning Area Under Curve for RL models is proposed. This indicator can measure the strengths and weaknesses of RL models efficiently and facilitates comparing models and evaluating offline experiments. The results show that the DDPG algorithm improves the click-through rate by 2.586% compared with the conventional recommendation algorithm. It shows that the algorithm designed in this paper has more obvious advantages in accurate recommendation by users. This paper effectively improves the efficiency of news dissemination by optimizing the push mode of intelligent news dissemination. In addition, the paper also deeply studies the innovative application of intelligent edge technology in news communication, which brings new ideas and practices to promote the development of news communication methods. Optimizing the push mode of intelligent news dissemination not only improves the user experience, but also provides strong support for the application of intelligent edge technology in this field, which has important practical application prospects.

Deep Reinforcement Learning Car-Following Control Based on Multivehicle Motion Prediction

Reinforcement learning (RL)–based car-following (CF) control strategies have attracted significant attention in academia, emerging as a prominent research topic in recent years. Most of these control strategies focus solely on the motion status of the immediately preceding vehicle. However, with the development of vehicle-to-vehicle (V2V) communication technologies, intelligent vehicles such as connected autonomous vehicles (CAVs) can gather information about surrounding vehicles. Therefore, this study proposes an RL-based CF control strategy that takes multivehicle scenarios into account. First, the trajectories of two preceding vehicles and one following vehicle relative to the subject vehicle (SV) are extracted from a highD dataset to construct the environment. Then the twin-delayed deep deterministic policy gradient (TD3) algorithm is implemented as the control strategy for the agent. Furthermore, a sequence-to-sequence (seq2seq) module is developed to predict the uncertain motion statuses of surrounding vehicles. Once integrated into the RL framework, this module enables the agent to account for dynamic changes in the traffic environment, enhancing its robustness. Finally, the performance of the CF control strategy is validated both in the highD dataset and in two traffic perturbation scenarios. In the highD dataset, the TD3-based prediction CF control strategy outperforms standard RL algorithms in terms of convergence speed and rewards. Its performance also surpasses that of human drivers in safety, efficiency, comfort, and fuel consumption. In traffic perturbation scenarios, the performance of the proposed CF control strategy is compared with the model predictive controller (MPC). The results show that the TD3-based prediction CF control strategy effectively mitigates undesired traffic waves caused by the perturbations from the head vehicle. Simultaneously, it maintains the desired traffic state and consistently ensures a stable and efficient traffic flow.