Deep Deterministic Policy Gradient Algorithm Research Articles

A Wide-Range TCSC Based ADN in Mountainous Areas Considering Hydropower-Photovoltaic-ESS Complementarity.

Due to the radial network structures, small cross-sectional lines, and light loads characteristic of existing AC distribution networks in mountainous areas, the development of active distribution networks (ADNs) in these regions has revealed significant issues with integrating distributed generation (DGs) and consuming renewable energy. Focusing on this issue, this paper proposes a wide-range thyristor-controlled series compensation (TCSC)-based ADN and presents a deep reinforcement learning (DRL)-based optimal operation strategy. This strategy takes into account the complementarity of hydropower, photovoltaic (PV) systems, and energy storage systems (ESSs) to enhance the capacity for consuming renewable energy. In the proposed ADN, a wide-range TCSC connects the sub-networks where PV and hydropower systems are located, with ESSs configured for each renewable energy generation. The designed wide-range TCSC allows for power reversal and improves power delivery efficiency, providing conditions for the optimization operation. The optimal operation issue is formulated as a Markov decision process (MDP) with continuous action space and solved using the twin delayed deep deterministic policy gradient (TD3) algorithm. The optimal objective is to maximize the consumption of renewable energy sources (RESs) and minimize line losses by coordinating the charging/discharging of ESSs with the operation mode of the TCSC. The simulation results demonstrate the effectiveness of the proposed method.

Research on the Motion Control Strategy of a Lower-Limb Exoskeleton Rehabilitation Robot Using the Twin Delayed Deep Deterministic Policy Gradient Algorithm.

The motion control system of a lower-limb exoskeleton rehabilitation robot (LLERR) is designed to assist patients in lower-limb rehabilitation exercises. This research designed a motion controller for an LLERR-based on the Twin Delayed Deep Deterministic policy gradient (TD3) algorithm to control the lower-limb exoskeleton for gait training in a staircase environment. Commencing with the establishment of a mathematical model of the LLERR, the dynamics during its movement are systematically described. The TD3 algorithm is employed to plan the motion trajectory of the LLERR's right-foot sole, and the target motion curve of the hip (knee) joint is deduced inversely to ensure adherence to human physiological principles during motion execution. The control strategy of the TD3 algorithm ensures that the movement of each joint of the LLERR is consistent with the target motion trajectory. The experimental results indicate that the trajectory tracking errors of the hip (knee) joints are all within 5°, confirming that the LLERR successfully assists patient in completing lower-limb rehabilitation training in a staircase environment. The primary contribution of this study is to propose a non-linear control strategy tailored for the staircase environment, enabling the planning and control of the lower-limb joint motions facilitated by the LLERR.

Research on Obstacle Avoidance Strategy of Automated Heavy Vehicle Platoon in High-Speed Scenarios

Amidst the rapid progression of the road transport industry, the safety and efficiency of heavy-vehicle platoons have garnered significant attention. The study tackles the challenge of obstacle avoidance presented by vehicles owing to their considerable mass, delayed response times, and line-of-sight impediments, by introducing a cooperative obstacle avoidance system for heavy-vehicle platoons based on deep reinforcement learning. The system comprises three primary modules: perception, decision-making, and control. Initially, the perception module acquires real-time environmental data. Subsequently, the decision-making module formulates obstacle avoidance decisions based on the acquired data. Specifically, it implements a two-stage braking obstacle avoidance strategy under low collision risk scenarios, while employing a fifth-degree polynomial for planning and tracking obstacle avoidance paths under high collision risk conditions suitable for steering maneuvers. The control module utilizes the local multi-agent deep deterministic policy gradient (LADDPG) algorithm to train the heavy-vehicle platoon agents, ensuring the formation’s maintenance while mitigating collisions with other vehicles and obstacles. The effectiveness of the proposed system is substantiated through simulation experiments, demonstrating its adaptability to various traffic conditions, selection of suitable obstacle avoidance strategies, and significant enhancement of obstacle avoidance performance and heavy-vehicle platoon stability.

Data-driven two-stage scheduling of multi-energy systems for operational flexibility enhancement

The multi-energy system, encompassing electricity networks, district heating networks (DHNs), and hydrogen-enriched compressed natural gas (HCNG) networks, provides an alternative for enhancing operational flexibility. This paper investigates a data-driven two-stage scheduling strategy of multi-energy system to promote uncertain renewable energy integration and improve economic benefits. Dynamic models for DHNs and HCNG networks are established, and the flexibility of multi-energy system is quantified through distribution-level power aggregation. To compromise flexibility enhancement and operational cost reduction, the multiobjective day-ahead scheduling optimization is conducted based on adaptive batch-ParEGO method. Taking day-ahead scheduling as a baseline, a data-driven real-time dispatch method is proposed based on the deep deterministic policy gradient algorithm, which adaptively modifies the energy management scheme in smaller temporal dimension to address uncertainties on both the load and source sides. The performance of the proposed two-stage scheduling method is demonstrated through numerical tests.

Learning-based hierarchical cooperative eco-driving with traffic flow prediction for hybrid electric vehicles

The integration of autonomous driving and hybrid electric vehicle technologies presents a promising solution for achieving environmental sustainability. This paper introduces an innovative energy-efficient driving strategy for hybrid electric vehicles that incorporates real-time traffic flow prediction. The study delves into the impact of both lateral and longitudinal vehicle maneuvers on energy consumption within dynamic traffic environments, offering novel insights into optimizing energy utilization. Firstly, a multi-lane traffic flow state rolling predictor is constructed based on the Hankel dynamic mode decomposition algorithm. Subsequently, a vehicle longitudinal and lateral coordinated control strategy is established by integrating the prioritized experience replay double deep Q-network algorithm. Finally, a novel energy management strategy is proposed that leverages Simulink dynamic model and the deep deterministic policy gradient algorithm to address the vehicle dynamic decision-making planning results. Within a hierarchical cooperative optimization framework, this research comprehensively considers safety, comfort, traffic efficiency, and fuel economy. By introducing a novel hierarchical collaborative ecological driving framework, we have achieved a substantial improvement in environmental sustainability, with traffic efficiency increasing by 10.27%-14.41% and fuel economy rising by 9.44%-10.47%. Hardware-in-the-loop validation has confirmed the proposed approach’s real-time capabilities and promising practical applications.

Research on priority scheduling strategy for smoothing power fluctuations of microgrid tie‐lines based on PER‐DDPG algorithm

AbstractThe variability of renewable energy within microgrids (MGs) necessitates the smoothing of power fluctuations through the effective scheduling of internal power equipment. Otherwise, significant power variations on the tie‐line connecting the MG to the main power grid could occur. This study introduces an innovative scheduling strategy that utilizes a data‐driven approach, employing a deep reinforcement learning algorithm to achieve this smoothing effect. The strategy prioritizes the scheduling of MG's internal power devices, taking into account the stochastic charging patterns of electric vehicles. The scheduling optimization model is initially described as a Markov decision process with the goal of minimizing power fluctuations on the interconnection lines and operational costs of the MG. Subsequently, after preprocessing the historical operational data of the MG, an enhanced scheduling strategy is developed through a neural network learning process. Finally, the results from four scheduling scenarios demonstrate the significant impact of the proposed strategy. Comparisons of reward curves before and after data preprocessing underscore its importance. In contrast to optimization results from deep deterministic policy gradient, soft actor‐critic, and particle swarm optimization algorithms, the superiority of the deep deterministic policy gradient algorithm with the addition of a priority experience replay mechanism is highlighted.

Adaptive Control of Ships’ Oil-Fired Boilers Using Flame Image-Based IMC-PID and Deep Reinforcement Learning

The control system of oil-fired boiler units on ships plays a crucial role in reducing the emissions of atmospheric pollutants such as nitrogen oxides (NOx), sulfur dioxides (SO2), and carbon dioxide (CO2). Traditional control methods using conventional measurement sensors face limitations in real-time control due to response delays, which has led to the growing interest in combustion control methods using flame images. To ensure the precision of such combustion control systems, the system model must be thoroughly considered during controller design. However, finding the optimal tuning point is challenging due to the changes in the system model and nonlinearity caused by environmental variations. This study proposes a controller that integrates an internal model control (IMC)-based PID controller with the deep deterministic policy gradient (DDPG) algorithm of deep reinforcement learning to enhance the adaptability of image-based combustion control systems to environmental changes. The proposed controller adjusts the PID parameter values in real-time through the learning of the determination constant lambda (λ) of the IMC internal model. This approach reduces computational resources by shrinking the learning dimensions of the DDPG agent and limits transient responses through constrained learning of control parameters. Experimental results show that the proposed controller exhibited rapid adaptive performance in the learning process for the target oxygen concentration, achieving a reward value of −0.05 within just 105 episodes. Furthermore, when compared to traditional PID tuning methods, the proposed controller demonstrated superior performance, achieving a target value error of 0.0032 and a low overshoot range of 0.0498 to 0.0631, providing the fastest response speed and minimal oscillation. Additionally, experiments conducted on an actual operating ship verified the practical feasibility of this system, highlighting its potential for real-time control and pollutant reduction in marine applications.

Optimizing Energy Harvesting in Wireless Body Area Networks: A Deep Reinforcement Learning Approach to Dynamic Sampling

In energy-harvesting wireless body area networks (EH-WBAN), the self-sustainability of body sensor nodes without compromising the service quality criteria is very important. In remote vital signs monitoring applications, the sampling rate of body nodes in each period should be determined to achieve this goal. There are two fundamental challenges in determining the sampling rate: 1) In EH-WBAN, the rate of the harvestable energy is time-dependent and unpredictable. 2) The vital signs of the patient have different change rates relevant to the time. Therefore, a technique is required that can learn the changes of the harvestable energy and the change rates of vital signs simultaneously and specify the proper sampling rate for BNs. The reinforcement learning algorithms are among the efficient solutions for this problem because they are capable of learning environmental uncertainties. Previous RL-based methods for determining the BN's sampling rate have three fundamental problems: 1) focus on the optimization of the transmission energy and ignore the sensing energy, 2) only affect one of the two aspects of energy or sensed data in determining the sampling rate and 3) discretization of the problem space does not ensure the determination of the optimal sampling rate. Therefore, this paper proposes a method named “deep reinforcement learning-based dynamic sampling” (DRDS) which first formulates the sampling rate determination problem as a Markov decision process (MDP) and proposes a deep deterministic policy gradient algorithm (DDPG) to solve it. The proposed method considers both energy and data variability aspects in determining the sampling rate. Two parameters, the super-capacitor’s voltage level, and ambient light intensity, are considered for the energy aspect; for the data variability aspect, the long short-term memory (LSTM) algorithm is developed to predict the change rate of data in the next round. Simulations indicate that by preserving the data integrity, the proposed method can decrease the sampling rate and unnecessary data transmission by about 49.95% and 89.7%, respectively, compared with state-of-the-art methods, and allow the sensor to achieve self-sustainability.

Experimental Implementation of a TD3 Agent Based Speed Controller for Direct Torque Control of PMSM Drives

This manuscript presents a novel control approach for Permanent Magnet Synchronous Motors (PMSMs) by integrating the widely used Direct Torque Control (DTC) method with Deep Reinforcement Learning (DRL), specifically the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. The TD3 algorithm is well-suited to address the challenges posed by high-dimensional state and action spaces in PMSM drives. The conventional Proportional–Integral (PI) controller in the outer loop of DTC is replaced with the DRL based TD3 agent to overcome the limitations of traditional PI controllers, such as model dependency and complex parameter tuning. The DRL technique allows the agent to learn optimal control policies from experience, making it suitable for complex and nonlinear control problems. Extensive simulation studies demonstrate the effectiveness of the TD3-based control in achieving accurate speed tracking under varying operating conditions. Real-time experimental validation using a TMS320F28379D digital signal processor confirms the feasibility of the proposed DRL based approach. The research offers new insights into improving the performance of PMSM drives and paves the way for future advancements in electric drive control.

Method for remaining useful life prediction of rolling bearings based on deep reinforcement learning.

In contemporary industrial systems, the prediction of remaining useful life (RUL) is recognized as a valuable maintenance strategy for health management due to its ability to monitor equipment operational status in real time and ensure the safety of industrial production. Current studies have largely concentrated on deep learning (DL) techniques, leading to a shortage of RUL prediction methods that utilize deep reinforcement learning (DRL). To further enhance application and research, this paper introduces a novel approach to RUL prediction based on DRL, specifically using a combination of Convolutional Neural Network-Bidirectional Long Short-Term Memory Network (CNN-BiLSTM) and the Deep Deterministic Policy Gradient (DDPG) algorithm. The proposed method reframes the conventional task of estimating RUL as a Markov decision process (MDP), effectively integrating the feature extraction capabilities of DL with the decision-making abilities of DRL. Initially, a hybrid CNN-BiLSTM is employed to establish an agent that can extract degradation features from raw signals. Subsequently, the DDPG algorithm within DRL is leveraged to develop the RUL prediction mechanism, completing the MDP by defining appropriate action spaces and reward functions. The agent, through repeated trials and optimization, learns to map the current operational state of the rolling bearing to its remaining service life. Validation analysis was performed on the intelligent maintenance systems (IMS) bearing dataset. The findings suggest that the DRL-based approach outperforms the current methodologies, demonstrating a superior performance in root mean square error (MSE) and MSE metrics. The predicted outcomes align more closely with the actual lifespan values.

Mapless Path Planning for Mobile Robot Based on Improved Deep Deterministic Policy Gradient Algorithm.

In the traditional Deep Deterministic Policy Gradient (DDPG) algorithm, path planning for mobile robots in mapless environments still encounters challenges regarding learning efficiency and navigation performance, particularly adaptability and robustness to static and dynamic obstacles. To address these issues, in this study, an improved algorithm frame was proposed that designs the state and action spaces, and introduces a multi-step update strategy and a dual-noise mechanism to improve the reward function. These improvements significantly enhance the algorithm's learning efficiency and navigation performance, rendering it more adaptable and robust in complex mapless environments. Compared to the traditional DDPG algorithm, the improved algorithm shows a 20% increase in the stability of the navigation success rate with static obstacles along with a 25% reduction in pathfinding steps for smoother paths. In environments with dynamic obstacles, there is a remarkable 45% improvement in success rate. Real-world mobile robot tests further validated the feasibility and effectiveness of the algorithm in true mapless environments.

An Alternative Reinforcement Learning (ARL) control strategy for data center air-cooled HVAC systems

Energy efficiency of data center is of great concern globally due to their large amount of energy consumption and the foreseeable growth in the demand of digital services in the future. Advanced control strategies are needed to reduce energy consumption. However, the optimization of data center HVAC control is a challenge task due to the complexity of the thermal dynamic models of buildings and uncertainties associated with both server loads and outdoor temperature. In this paper, we propose a new control strategy called Alternately-Reinforcement Learning-control(ARL), which realizes the alternating control of RL and proportional, integral and derivative (PID) for optimizing the control of air-cooled HVAC systems in data centers. The control object of the ARL is the speed set of the compressor and the condensing fan, and the control goal is to minimize energy consumption while maintaining temperature stability. The applied RL algorithm is Deep deterministic policy gradient (DDPG) and Proximal policy optimization (PPO). We pre-train the ARL strategy in offline environment firstly and deploy it in real data center environment for online testing. The test results show that the ARL has significant advantages in maintaining temperature stability while reducing energy consumption, and the PPO algorithm performs better than the DDPG algorithm. Compared with the PID algorithm, the PPO algorithm can save energy by 5.27%, and the temperature control effect can be improved by 3.27%,which indicates the feasibility of the ARL for implementation in real data centers.

Bio-inspired distributed load frequency control in Islanded Microgrids: A multi-agent deep reinforcement learning approach

To prevent errors in load frequency control (LFC) decision-making in an islanded microgrid and reduce the waste of frequency regulation resources, a fully distributed “starfish” load frequency control method is proposed. For this method, a novel deep reinforcement learning algorithm called the distributed decomposed multirole multiagent deep deterministic policy gradient (DDMR-MADDPG) algorithm is introduced. This algorithm treats each unit as an agent, enabling centralized training of all the agents. Through the coordination and control of multiple agents, a global optimal strategy can be obtained. During online application, the algorithm employs a distributed implementation strategy, enabling each unit to make decisions autonomously. This decision-making strategy is akin to the distributed neural network of a starfish, and it can fundamentally improve the efficiency and correctness of decision-making. In addition, it employs multiple techniques, including value function decomposition, multirole learning, imitation learning, and curriculum learning approaches. The performance of the proposed method compared with 24 existing methods is demonstrated on a simulated LFC model of the Zhuzhou Island microgrid in China.

Container Scheduling Algorithms for Distributed Cloud Environments

Due to the difficulty of existing container scheduling algorithms to adapt to large-scale complex scenarios and meet the diverse application and load requirements, this study delves into a groundbreaking hybrid scheduling approach that melds Deep Deterministic Policy Gradient (DDPG) with a Genetic Algorithm (GA). The proposed method initially employs a container grouping policy to reduce the overhead associated with frequent inter-container calls. Subsequently, to address the computational inefficiency of large-scale scheduling, a genetic algorithm is utilized for rapid global optimization. To overcome the problem of genetic algorithms being highly susceptible to falling into local optimality, the DDPG algorithm is applied for local optimization, with a cross-mutation operation introduced to escape local optima. The experimental outcomes demonstrate that the proposed algorithm enhances cluster load balancing by 81.13%, improves the fitness function by 3.26%, reduces completion time by 19.06%, and decreases container dependency overhead by 2.75%. Furthermore, under the experimental conditions, the system performs best when the group size is 10. This research offers a novel paradigm for the development of container scheduling algorithms in distributed intelligent data clouds, advancing the field of resource management in cloud computing environments.

Optimizing passengers’ experience: A goal-oriented reinforcement learning speed control approach for urban railway trains

Prolonged vibration can be uncomfortable for passengers utilizing urban rail transit systems. This study proposes an automatic speed control framework for urban railway trains to reduce vertical vibrations experienced by passengers. We suggest the concept of the “segmented comfort speed limit” to represent the vertical passing comfort of oncoming sections. This speed limit is calculated from 1/3 octave band acceleration and smoothed through lag-type speed control mode. The deep deterministic policy gradient algorithm with hindsight experience replay mechanism (HER-DDPG) is designed, to balance safety, comfort, and energy efficiency driving. Verify the speed control framework based on HER-DDPG through the rail data collected from Dalian Metro Line 12. Compared with the DDPG-based model, the vertical comfort is improved by 2.34%, and the longitudinal acceleration and total energy consumption are reduced by 45% and 8.1%. Compared with the real-world train control trajectory, HER-DDPG improves vertical comfort by 9.76% and reduces energy consumption by 12.4%. The results show that the proposed framework can effectively improve the ride experience of passengers.

A Multiproject and Multilevel Plan Management Model Based on a Hybrid Program Evaluation and Review Technique and Reinforcement Learning Mechanism

It is very difficult for manufacturing enterprises to achieve automatic coordination of multiproject and multilevel planning when they are unable to make large-scale resource adjustments. In addition, planning and coordination work mostly relies on human experience, and inaccurate planning often occurs. This article innovatively proposes the PERT-RP-DDPGAO algorithm, which effectively combines the program evaluation and review technique (PERT) and deep deterministic policy gradient (DDPG) technology. Innovatively using matrix computing, the resource plan (RP) itself is used for the first time as an intelligent agent for reinforcement learning, achieving automatic coordination of multilevel plans. Through experiments, this algorithm can achieve automatic planning and has interpretability in management theory. To solve the problem of continuous control, the second half of the new algorithm adopts the DDPG algorithm, which has advantages in convergence and response speed compared to traditional reinforcement learning algorithms and heuristic algorithms. The response time of this algorithm is 3.0% lower than the traditional deep Q-network (DQN) algorithm and more than 8.4% shorter than the heuristic algorithm.

Collaborative task decision-making of multi-UUV in dynamic environments based on deep reinforcement learning

ABSTRACT The problem of collaborative task decision-making for unmanned underwater vehicles (UUVs) in an unknown dynamic ocean environment is investigated. In view of the unknown marine environment, a partially observable Markov decision process is designed to achieve partially observable path planning for multi-UUV. Aiming at the problems of long planning time, large amount of data, and insufficient multi-task decision-making capability in the process of multi-UUV underwater collaborative operation, a multi-UUV collaborative task decision-making model based on the multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm for the dynamic environment is constructed. The implementation of the centralised training with distributed execution (CT-DE) training framework enables each UUV to possess autonomous decision-making capability and ensures task safety under weak communication conditions. The pre-training phase shows that the MATD3 algorithm outperforms the multi-agent deep deterministic policy gradient (MADDPG) algorithm for training multi-UUV autonomous decision-making scenarios in dynamic environments. The simulation experiment results verify that the autonomous decision-making method based on the MATD3 algorithm can effectively solve the multi-UUV collaborative task decision-making problem in unknown dynamic environments, and the task process satisfies the requirements of real-time, safety and economy.

Integrated energy cluster hierarchical regulation technology considering demand response

Integrated Energy Cluster (IEC), the regional aggregation of integrated energy systems (IES), has accumulated plenty of dispatchable resources with the development of energy market. This, while significantly providing system Demand Response (DR) potential, also complicates the interaction of the IEC with the main grid and increases the difficulty of system scheduling. To address this issue, this paper proposes a Reinforcement Learning-driven multi-agent hierarchical regulation framework that makes full use of DR to maximize the benefits of both IEC and main grid. Firstly, in the context of the DR market, a mechanism for IECs to bid in the real-time DR market is proposed. Furthermore, an "IEC-main network" hierarchical regulation model taking account of DR is established to minimize the IEC operation cost and maximize the societal benefit. Moreover, an optimization algorithm utilizing Deep Deterministic Policy Gradient (DDPG) with Multi-process (MP) and Priority Experience Replay (PER) mechanism is proposed to allow adaptability to high-latitude and large-scale applications. In case study, the proposed model and algorithm is tested on an 8-node system and a 24-node system. The result indicates that the hierarchical regulation model considering DR can improve the system economy by 2.59 % than that without DR and the improved DDPG algorithm can enhance training effectiveness in comparison with DDPG and PER-DDPG.

A soft actor-critic reinforcement learning framework for optimal energy management in electric vehicles with hybrid storage

The efficient energy management of electric vehicles (EVs) equipped with hybrid energy storage systems (HESS) poses a significant challenge due to its vast search space, numerous control variables, and intricate driving conditions. In response to this challenge, this paper introduces a novel deep reinforcement learning (DRL) algorithm, specifically the soft actor-critic (SAC), tailored for optimizing energy distribution within EVs equipped with battery-supercapacitor (SC) HESS. The proposed SAC-based energy management system (EMS) is designed to address inherent limitations present in most existing DRL algorithms, such as slower convergence rates, discretization errors, unstable training dynamics, and suboptimal optimization effects. The SAC-based EMS undergoes training in a continuous action space through self-play, utilizing a complex driving cycle and a novel reward function. The algorithm demonstrates its efficiency by rapidly maximizing cumulative rewards and refining its decision-making policies. Afterward, extensive experiments showcase the superiority of the proposed SAC-based EMS over rule-based (RB) techniques, deep deterministic policy gradient (DDPG), and battery-only configurations across various driving cycles. Notably, the approach effectively allocates high-power surges and braking energy to the SC, reducing the frequency of charge/discharge cycles and thereby extending the battery's lifespan. Additionally, the learned SAC-based EMS policy achieves substantial reductions in electricity consumption by 39.6 %, 47.87 %, and 45 % compared to battery-only configuration, and 42.56 %, 47.87 %, and 45.83 % compared to RB technique under the NYCC, LA92, and FTP cycles, respectively. In contrast, the DDPG algorithm tends to rely heavily on the SC to deliver the total power required, even under normal driving conditions.

Automated architectural spatial composition via multi-agent deep reinforcement learning for building renovation

This paper focuses on utilizing deep reinforcement learning technology to explore how to achieve automated architectural space composition under established built environment conditions to address the extensive demands of old building renovation. A reinforcement learning platform has been develop0065d, centered around the multi-agent deep deterministic policy gradient (MADDPG) algorithm. The paper leverages functional blocks as agents within Grasshopper to simulate existing architectural structures and their interactions, establishing a task-based reward system to guide the design process. Furthermore, data exchange between algorithmic and modeling software is facilitated through UDP communication. Training results using typical frame structure spaces indicate a significant increase in cumulative rewards around the 650,000th training step, effectively achieving the given design tasks. The paper demonstrates the potential of automated architectural space composition methods, based on deep reinforcement learning, to enhance design efficiency, optimize spatial arrangements, and promote human-machine collaborative design in building renovations.