Deterministic Policy Gradient Research Articles

Continuous trading strategy based on deep reinforcement learning

Automatic trading policy has been researched with reinforcement learning (RL). Designing a profitable and applicable policy is of great significance for research in quantitative finance. Incoprating with deep learning, typical deep reinforcement learning (DRL) algorithms such as Proximal Policy Optimization (PPO) have shown their effectiveness. To improve the practical applicability, the manner in which we train our model should better simulate the dynamic of the stock market. A sliding window training strategy is a better solution, which employes training, validation and trading procedures on dataset with a sliding window. However, this empirical strategy can still be further investigated in algorithm evaluation and the experiment designation such as choice of sliding window. In this paper, we further investigated the continuous trading strategy (CTS). We evaluated the performance of a wider range of algorithms, including PPO, Deep Deterministic Policy Gradient (DDPG), Advantage Actor-Critic (A2C), Soft Actor-Critic (SAC). Moreover, we trained our models on a longer term period. We also provide detailed observations and suggestions on experiment settings. These discussions will facilitate researchers in their future work.

Multi-agent reinforcement learning method for cutting parameters optimization based on simulation and experiment dual drive environment

Improving production efficiency while ensuring product surface quality is a constant focus of manufacturers. Cutting parameter optimization is an important technique for ensuring high-efficiency and high-quality production. In this paper, a novel method for cutting parameter optimization that integrates multi-agent reinforcement learning with a dual-drive virtual machining environment is proposed. First, a feature extraction, fusion and generation model for cutting simulation and experimental data is proposed to solve the problem of incomplete data acquisition in the production process. Second, a Markov decision model for optimizing cutting parameters is defined, and a virtual machining environment driven by both simulation and experimental data is constructed. Third, a novel multi-agent reinforcement learning method called Q-MIX-MATD3, in which a twin delay deep deterministic policy gradient, value function decomposition and a teacher model are combined, is proposed to explore the cutting parameter optimization policy by interacting with the virtual machining environment. Finally, the proposed method is verified on a commutator production line. Moreover, the results show that the accuracy of the virtual machining environment driven by both simulation and experiment increases by more than 5 %, response efficiency increases by 31 %, and Q-MIX-MATD3-based cutting parameter optimization method reduces time cost by 98 % and achieves the optimization effect of the classical optimization method.

Continuous control of structural vibrations using hybrid deep reinforcement learning policy

An efficient active structural control scheme by utilizing the collective concepts of data-driven and physics-inspired reinforcement learning (RL) approaches is being introduced here. The controller is designed to deliver optimal feedback forces based on the full-state information, wherein the training process involves the use of deep neural networks (NNs) and a specially designed gradient descent-based sequence within the RL framework. This integration of algorithms results in a unique active control policy that accelerates the learning process, thereby demands considerably less computational resources to develop an optimal and stable controller as compared to existing data-driven approaches. Most importantly, the mentioned data-driven and physics-inspired approaches encompass deep deterministic policy gradient (DDPG) and an iterative gradient-based state feedback control (SFSC) algorithms, respectively, to establish the architecture of the hybrid control policy, referred to as hybrid RL-controller. Additional advantage of the hybrid RL-controller is its ability to operate in continuous state–action spaces, which allows the designed controller to address diverse structural control problems. The staggered performance of the hybrid RL-controller, both in continuous and discrete time, is evaluated in three case studies that involve structures in linear and nonlinear regimes. The outcomes include a detailed comparison of the hybrid RL-controller with the individually designed DDPG and SFSC RL control strategies, as well as the uncontrolled scenario. Furthermore, this study thoroughly investigates the real-life implementation concerns of control strategies, such as perturbations in model parameters and input forces, and time delays in the feedback loop. Finally, the results from this study corroborate that the designed RL-controller showcases superior performance and faster execution time in the feedback loop, making it suitable for the vibration control of multidimensional complex structures.

Manipulating Camera Gimbal Positioning by Deep Deterministic Policy Gradient Reinforcement Learning for Drone Object Detection

The object recognition technology of unmanned aerial vehicles (UAVs) equipped with “You Only Look Once” (YOLO) has been validated in actual flights. However, here, the challenge lies in efficiently utilizing camera gimbal control technology to swiftly capture images of YOLO-identified target objects in aerial search missions. Enhancing the UAV’s energy efficiency and search effectiveness is imperative. This study aims to establish a simulation environment by employing the Unity simulation software for target tracking by controlling the gimbal. This approach involves the development of deep deterministic policy-gradient (DDPG) reinforcement-learning techniques to train the gimbal in executing effective tracking actions. The outcomes of the simulations indicate that when actions are appropriately rewarded or penalized in the form of scores, the reward value can be consistently converged within the range of 19–35. This convergence implies that a successful strategy leads to consistently high rewards. Consequently, a refined set of training procedures is devised, enabling the gimbal to accurately track the target. Moreover, this strategy minimizes unnecessary tracking actions, thus enhancing tracking efficiency. Numerous benefits arise from training in a simulated environment. For instance, the training in this simulated environment is facilitated through a dataset composed of actual flight photographs. Furthermore, offline operations can be conducted at any given time without any constraint of time and space. Thus, this approach effectively enables the training and enhancement of the gimbal’s action strategies. The findings of this study demonstrate that a coherent set of action strategies can be proficiently cultivated by employing DDPG reinforcement learning. Furthermore, these strategies empower the UAV’s gimbal to rapidly and precisely track designated targets. Therefore, this approach provides both convenience and opportunities to gather more flight-scenario training data in the future. This gathering of data will lead to immediate training opportunities and help improve the system’s energy consumption.

VizNav: A Modular Off-Policy Deep Reinforcement Learning Framework for Vision-Based Autonomous UAV Navigation in 3D Dynamic Environments

Unmanned aerial vehicles (UAVs) provide benefits through eco-friendliness, cost-effectiveness, and reduction of human risk. Deep reinforcement learning (DRL) is widely used for autonomous UAV navigation; however, current techniques often oversimplify the environment or impose movement restrictions. Additionally, most vision-based systems lack precise depth perception, while range finders provide a limited environmental overview, and LiDAR is energy-intensive. To address these challenges, this paper proposes VizNav, a modular DRL-based framework for autonomous UAV navigation in dynamic 3D environments without imposing conventional mobility constraints. VizNav incorporates the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with Prioritized Experience Replay and Importance Sampling (PER) to improve performance in continuous action spaces and mitigate overestimations. Additionally, VizNav employs depth map images (DMIs) to enhance visual navigation by accurately estimating objects’ depth information, thereby improving obstacle avoidance. Empirical results show that VizNav, by leveraging TD3, improves navigation, and the inclusion of PER and DMI further boosts performance. Furthermore, the deployment of VizNav across various experimental settings confirms its flexibility and adaptability. The framework’s architecture separates the agent’s learning from the training process, facilitating integration with various DRL algorithms, simulation environments, and reward functions. This modularity creates a potential to influence RL simulation in various autonomous navigation systems, including robotics control and autonomous vehicles.

Trajectory Tracking Control of Variable Sweep Aircraft Based on Reinforcement Learning.

An incremental deep deterministic policy gradient (IDDPG) algorithm is devised for the trajectory tracking control of a four-wing variable sweep (FWVS) aircraft with uncertainty. The IDDPG algorithm employs the line-of-sight (LOS) method for path tracking, formulates a reward function based on position and attitude errors, and integrates long short-term memory (LSTM) units into IDDPG algorithm to enhance its adaptability to environmental changes during flight. Finally, environmental disturbance factors are introduced in simulation to validate the designed controller's ability to track climbing trajectories of morphing aircraft in the presence of uncertainty.

An Efficient Spoofing Attack Detection Using Deep Learning Based Physical Layer Security Technique

Spoofing attack detection plays a crucial role in the defence field, involving critical and highly secured data processing. The accurate attack detection mechanism prevents unauthorised access to sensitive information, thereby protecting National security. Physical Layer Security (PLS) is a promising emerging technique that uses the wireless channel’s randomness to secure the communication network. The spoofing attack is one of the severe threats to the wireless network, where the attacker imitates the legitimate user to launch an attack against the network. This paper investigates the channel characteristics-based physical layer technique to detect spoofing attacks. For static radio environments, the two-sample independent hypothesis testing is used to identify the spoofing attack, showing an improvement in detection accuracy of 97 %. The attack detection problem is considered a Reinforcement Learning (RL) based classification problem for a challenging dynamic radio environment. It is simulated using the actor-critic-based Deep Reinforcement Learning (DRL) technique with the help of the Reformed Deep Deterministic Policy Gradient (Re-DDPG) algorithm. The simulated results show that the proposed method performs better than the existing strategies and achieves a Receiver Operating Characteristics (ROC) value of 0.96. The detection accuracy of the proposed method can reach up to 98 %, with precision and recall of about 98 % and 99 %, respectively.

Deep Reinforcement Learning Lane-Changing Decision Algorithm for Intelligent Vehicles Combining LSTM Trajectory Prediction

Intelligent decisions for autonomous lane-changing in vehicles have consistently been a focal point of research in the industry. Traditional lane-changing algorithms, which rely on predefined rules, are ill-suited for the complexities and variabilities of real-world road conditions. In this study, we propose an algorithm that leverages the deep deterministic policy gradient (DDPG) reinforcement learning, integrated with a long short-term memory (LSTM) trajectory prediction model, termed as LSTM-DDPG. In the proposed LSTM-DDPG model, the LSTM state module transforms the observed values from the observation module into a state representation, which then serves as a direct input to the DDPG actor network. Meanwhile, the LSTM prediction module translates the historical trajectory coordinates of nearby vehicles into a word-embedding vector via a fully connected layer, thus providing predicted trajectory information for surrounding vehicles. This integrated LSTM approach considers the potential influence of nearby vehicles on the lane-changing decisions of the subject vehicle. Furthermore, our study emphasizes the safety, efficiency, and comfort of the lane-changing process. Accordingly, we designed a reward and penalty function for the LSTM-DDPG algorithm and determined the optimal network structure parameters. The algorithm was then tested on a simulation platform built with MATLAB/Simulink. Our findings indicate that the LSTM-DDPG model offers a more realistic representation of traffic scenarios involving vehicle interactions. When compared to the traditional DDPG algorithm, the LSTM-DDPG achieved a 7.4% increase in average single-step rewards after normalization, underscoring its superior performance in enhancing lane-changing safety and efficiency. This research provides new ideas for advanced lane-changing decisions in autonomous vehicles.

Off-policy actor-critic deep reinforcement learning methods for alert prioritization in intrusion detection systems

Alert prioritization plays a very important role in network security as it helps security teams manage and respond to the overwhelming volume of alerts generated by intrusion detection systems (IDSs). Although current deep reinforcement learning (DRL) based deep deterministic policy gradient (DDPG) has achieved good performance for alert prioritization, there are still limitations in dealing with DDPG's instability and limited exploration capabilities. In this paper, we propose TD3-AP and SAC-AP, two DRL empowered off-policy actor-critic alert prioritization methods based on twin delayed deep deterministic policy gradient (TD3) and soft actor-critic (SAC), respectively. We model the interaction between an adversary and a defender as a zero-sum game and use a double oracle framework to obtain mixed strategy Nash equilibrium (MSNE). Our objective is to minimize the defender's loss which represents the inability of the defender to investigate alerts generated due to the attacks. To demonstrate the benefits and scalability of the proposed approaches, we conduct extensive experiments on three datasets: MQTT-IoT-IDS2020, DARPA 2000 LLDOS 1.0 and CSE-CIC-IDS2018. The results depict that TD3-AP and SAC-AP exhibit a decrease in the defender's loss by 50% and 14.28%, respectively, in comparison to the alert prioritization method based on DDPG. The improvements are significantly higher when the proposed approaches are compared with traditional alert prioritization methods including Uniform, Snort, Fuzzy and RAP. Additionally, we also provide an analysis of the interpretability of our results through the utilization of SHapley Additive exPlanations (SHAP). We assess the sensitivity of our proposed approaches to different hyperparameters and evaluate the training time and computational resources necessary for their implementation.

Energy Management Strategy Based on Reinforcement Learning and Frequency Decoupling for Fuel Cell Hybrid Powertrain

This study presents a Two-Layer Deep Deterministic Policy Gradient (TL-DDPG) energy management strategy for Hydrogen fuel cell hybrid train, that aims to solve the problem that traditional reinforcement learning strategies require high initial values and are difficult to optimize global variables. Augmenting the optimization capabilities of the inner layer, a frequency decoupling algorithm integrates into the outer layer, furnishing a fitting initial value for strategy optimization. This addition aims to bolster the stability of fuel cell output, thereby enhancing the overall efficiency of the hybrid power system. In comparison with the traditional reinforcement learning algorithm, the proposed approach demonstrates notable improvements: a reduction in hydrogen consumption per 100 km by 16.3 kg, a 9.7% increase in the output power stability of the fuel cell, and a 1.8% enhancement in its efficiency.

A deep learning optimized LQR method for enhanced formation control with embedded systems

To achieve higher accuracy throughout the formation control processes and enhance precision in dynamic environments, particularly for the formation control of follower vehicles with embedded systems, this paper proposes a method and framework for vehicle formation control. An Ackermann-model based Linear Quadratic Regulator (LQR) controller is developed for lateral distance control and a Proportional-Derivative (PD) controller for longitudinal distance control. To enhance the efficacy of the LQR controller, the Deep Deterministic Policy Gradient Derivative (DDPG) method is introduced into the control system. The DDPG networks are trained in a simulation environment and can subsequently predict LQR parameters in real-time experiments. The practical application of the method is showcased, and the concluding remarks emphasize the potential and superior performance of our proposed formation control approach by experimental comparison with other controllers. This method can be implemented in small vehicles that possess limited computational resources and is also suitable for scenarios requiring dynamic motion control with higher tracking accuracy and stability.

Optimizing UAV-UGV coalition operations: A hybrid clustering and multi-agent reinforcement learning approach for path planning in obstructed environment

One of the most critical applications undertaken by Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs) is reaching predefined targets by following the most time-efficient routes while avoiding collisions. Unfortunately, UAVs are hampered by limited battery life, and UGVs face challenges in reachability due to obstacles and elevation variations, which is why a coalition of UAVs and UGVs can be highly effective. Existing literature primarily focuses on one-to-one coalitions, which constrains the efficiency of reaching targets. In this work, we introduce a novel approach for a UAV-UGV coalition with a variable number of vehicles, employing a modified mean-shift clustering algorithm (MEANCRFT) to segment targets into multiple zones. This approach of assigning targets to various circular zones based on density and range significantly reduces the time required to reach these targets. Moreover, introducing variability in the number of UAVs and UGVs in a coalition enhances task efficiency by enabling simultaneous multi-target engagement. In our approach, each vehicle of the coalitions is trained using two advanced deep reinforcement learning algorithms in two separate experiments, namely Multi-agent Deep Deterministic Policy Gradient (MADDPG) and Multi-agent Proximal Policy Optimization (MAPPO). The results of our experimental evaluation demonstrate that the proposed MEANCRFT-MADDPG method substantially surpasses current state-of-the-art techniques, nearly doubling efficiency in terms of target navigation time and task completion rate.

Digital twin-assisted service function chaining in multi-domain computing power networks with multi-agent reinforcement learning

The emerging computing power network (CPN) is believed to undergo the paradigm reformation of network function virtualization (NFV) and service function chaining (SFC). It is prerequisite to explore the performance upper bound of NFV-assisted CPN before truly deploying the NFV and SFC technologies onto physical networks. Inspired by the application of digital twin (DT) in the industry and due to its advantage in synchronizing physical objects with their virtual replicas, we propose to use the DT to assist the SFC deployment in the multi-domain CPN, with the aid of multi-agent deep deterministic policy gradient (MADDPG) framework. First, we build a dynamic SFC mapping problem in the virtual twin network layer, by modeling the computing power, link bandwidth, delay performance and the VNF ordering as DT objects and constraints, to jointly optimize the energy consumption, end-to-end delay and the VNF re-deploying cost. Then, the prioritized experience replay and re-parameterization trick-empowered centralized training and decentralized execution MADDPG framework is utilized to learn the SFC deployment, by taking each domain controller as one agent. Finally, numerical experiments are carried out to validate the effectiveness of MADDPG in the cross-domain SFC deployment. For performance verification, the deployment success rate, number of crossed domains, energy consumption, end-to-end latency and load balancing degree are all taken as metrics, to show the performance of MADDPG compared to other learning frameworks.

Spectrum-efficient user grouping and resource allocation based on deep reinforcement learning for mmWave massive MIMO-NOMA systems

Millimeter-wave (mmWave) massive multiple-input multiple-output non-orthogonal multiple access (MIMO-NOMA) is proven to be a primary technique for sixth-generation (6G) wireless communication networks. However, the great increase in users and antennas brings challenges for interference suppression and resource allocation for mmWave massive MIMO-NOMA systems. This study proposes a spectrum-efficient and fast convergence deep reinforcement learning (DRL)-based resource allocation framework to optimize user grouping and allocation of subchannel and power. First, an enhanced K-means grouping algorithm is proposed to reduce the multi-user interference and accelerate the convergence. Then, a dueling deep Q-network (DQN) structure is proposed to perform subchannel allocation, which further improves the convergence speed. Moreover, a deep deterministic policy gradient (DDPG)-based power resource allocation algorithm is designed to avoid the performance loss caused by power quantization and improve the system’s achievable sum-rate. The simulation results demonstrate that our proposed scheme outperforms other neural network-based algorithms in terms of convergence performance, and can achieve higher system capacity compared with the greedy algorithm, the random algorithm, the RNN algorithm, and the DoubleDQN algorithm.

Hierarchical path planner combining probabilistic roadmap and deep deterministic policy gradient for unmanned ground vehicles with non-holonomic constraints

Path planning plays a vital role in autonomous driving as it needs to guide the vehicle to achieve the target position without collision while satisfying the vehicle's non-holonomic constraints. A good path planner should be able to cope with sophisticated environments and try to make the planned path as short as possible. Traditional path planning methods when being applied on four-wheel vehicles may suffer from one of the following disadvantages including conflicting with non-holonomic constraints, far from close-optimal length solution, or easy to get stuck in specific environments. In this paper, a hierarchical path planning method for a four-wheel vehicle combining probabilistic roadmap (PRM) and deep deterministic policy gradient (DDPG) is proposed to address existing problems. In the upper level, PRM is used to generate a guidance path quickly, which doesn't consider the vehicle's non-holonomic constraints but gives the sketch of the path that helps the vehicle to jump out of some dilemmas in specific environments while maintaining a relatively short path distance. In the lower level, reinforcement learning, specifically DDPG, is used to optimize the guidance path generated by PRM considering multiple factors including non-holonomic constraints and obstacle avoidance. The proposed method is validated in four different environments including a simple map, an easy-to-stuck specific map, a maze-like complex map and an office-like complex map. Results show that the proposed method could generate a smooth path satisfying non-holonomic constraints while demonstrating desirable obstacle avoidance performance. The planned path of the proposed method is superior to that of both traditional and state-of-the-art methods in terms of path length and smoothness, which could be 1.3–12.9 % shorter and 2.6–52.1 % smoother in different environments.

An improved energy management strategy for hybrid electric powered aircraft based on deep reinforcement learning

Electric airplanes have attracted a great deal of attentions because of their strategic commercial values and low emissions. One of the essentials for efficient control and management of hybrid electric propulsion aircraft is a well-designed energy management strategy (EMS). A new EMS approach based on deep learning is introduced in this study to realize thrust distribution and power distribution, consequently lowering aircraft fuel consumption. A model of hybrid electric propulsion power system including turbine engines and lithium-ion battery pack is constructed to act as an application environment. An intelligent energy management strategy based on deep reinforcement learning (DRL) is then proposed in this paper. Two DRL agents based on twin delayed deep deterministic policy gradient (TD3) are combined in the hierarchical framework. The multi-agents are trained to handle the energy management problem in this paper. In a finite period of time, the training process is accomplished. After training, the simulation result of the test shows that this DRL-based EMS reduces the equivalent fuel consumption by 2.2% compared with that using the existing method, and preserves 99.9% of the battery state of health (SOH). The battery pack's state of charge (SOC) regresses precisely to the reference value at the end. Particularly, the proposed EMS is able to work properly for different flight profiles and various initial SOCs. The results of this paper provide preliminary supports for the feasibility and superiority of deep reinforcement learning applied to the energy management strategy of series hybrid electric aircraft.

Deep Reinforcement Learning-Based Task Offloading and Load Balancing for Vehicular Edge Computing

Vehicular edge computing (VEC) effectively reduces the computational burden on vehicles by offloading tasks from resource-constrained vehicles to edge nodes. However, non-uniformly distributed vehicles offloading a large number of tasks cause load imbalance problems among edge nodes, resulting in performance degradation. In this paper, we propose a deep reinforcement learning-based decision scheme for task offloading and load balancing with the optimization objective of minimizing the system cost considering the split offloading of tasks and the load dynamics of edge nodes. First, we model the mutual interaction between mobile vehicles and Mobile Edge Computing (MEC) servers using a Markov decision process. Second, the optimal task-offloading and resource allocation decision is obtained by utilizing the twin delayed deep deterministic policy gradient algorithm (TD3), and server load balancing is achieved through edge collaboration using a server selection algorithm based on the technique for order preference by similarity to the ideal solution (TOPSIS). Finally, we have conducted extensive simulation experiments and compared the results with several other baseline schemes. The proposed scheme can more effectively reduce the system cost and increase the system resource utilization.

A memory and attention-based reinforcement learning for musculoskeletal robots with prior knowledge of muscle synergies

PurposeLimited by the types of sensors, the state information available for musculoskeletal robots with highly redundant, nonlinear muscles is often incomplete, which makes the control face a bottleneck problem. The aim of this paper is to design a method to improve the motion performance of musculoskeletal robots in partially observable scenarios, and to leverage the ontology knowledge to enhance the algorithm’s adaptability to musculoskeletal robots that have undergone changes.Design/methodology/approachA memory and attention-based reinforcement learning method is proposed for musculoskeletal robots with prior knowledge of muscle synergies. First, to deal with partially observed states available to musculoskeletal robots, a memory and attention-based network architecture is proposed for inferring more sufficient and intrinsic states. Second, inspired by muscle synergy hypothesis in neuroscience, prior knowledge of a musculoskeletal robot’s muscle synergies is embedded in network structure and reward shaping.FindingsBased on systematic validation, it is found that the proposed method demonstrates superiority over the traditional twin delayed deep deterministic policy gradients (TD3) algorithm. A musculoskeletal robot with highly redundant, nonlinear muscles is adopted to implement goal-directed tasks. In the case of 21-dimensional states, the learning efficiency and accuracy are significantly improved compared with the traditional TD3 algorithm; in the case of 13-dimensional states without velocities and information from the end effector, the traditional TD3 is unable to complete the reaching tasks, while the proposed method breaks through this bottleneck problem.Originality/valueIn this paper, a novel memory and attention-based reinforcement learning method with prior knowledge of muscle synergies is proposed for musculoskeletal robots to deal with partially observable scenarios. Compared with the existing methods, the proposed method effectively improves the performance. Furthermore, this paper promotes the fusion of neuroscience and robotics.

Inspection Robot Navigation Based on Improved TD3 Algorithm.

The swift advancements in robotics have rendered navigation an essential task for mobile robots. While map-based navigation methods depend on global environmental maps for decision-making, their efficacy in unfamiliar or dynamic settings falls short. Current deep reinforcement learning navigation strategies can navigate successfully without pre-existing map data, yet they grapple with issues like inefficient training, slow convergence, and infrequent rewards. To tackle these challenges, this study introduces an improved two-delay depth deterministic policy gradient algorithm (LP-TD3) for local planning navigation. Initially, the integration of the long-short-term memory (LSTM) module with the Prioritized Experience Re-play (PER) mechanism into the existing TD3 framework was performed to optimize training and improve the efficiency of experience data utilization. Furthermore, the incorporation of an Intrinsic Curiosity Module (ICM) merges intrinsic with extrinsic rewards to tackle sparse reward problems and enhance exploratory behavior. Experimental evaluations using ROS and Gazebo simulators demonstrate that the proposed method outperforms the original on various performance metrics.

Energy-Delay Joint Optimization for Task Offloading in Digital Twin-Assisted Internet of Vehicles

Vehicle edge computing (VEC) provides efficient services for vehicles by offloading tasks to edge servers. Notably, extant research mainly employs methods such as deep learning and reinforcement learning to make resource allocation decisions, without adequately accounting for the ramifications of high-speed mobility of vehicles and the dynamic nature of the Internet of Vehicles (IoV) on the decision-making process. This paper endeavours to tackle the aforementioned issue through the introduction of a novel concept, namely, a digital twin-assisted IoV. Among them, the digital twin of IoV offers training data for computational offloading and content caching decisions, which allows edge servers to directly interact with the dynamic environment while capturing its dynamic changes in real-time. Through this collaborative endeavour, edge intelligent servers can promptly respond to vehicular requests and return results. We transform the dynamic edge computing problem into a Markov decision process (MDP), and then solve it with the twin delayed deep deterministic policy gradient (TD3) algorithm. Simulation experiments demonstrate the adaptability of our proposed approach in the dynamic environment while successfully enhancing the Quality of Service, that is, decreasing total delay and energy consumption.