Deterministic Policy Gradient Research Articles

Unlocking optimal performance and flow level control of three-phase separator based on reinforcement learning: A case study in Basra refinery

This research explores the application of a new reinforcement learning (RL-based) controller for a three-phase separator connected to a gas turbine. The control of flow levels within the separator directly impacts fluid flow turbulence, especially when the equipment is linked to waste heat gas from the turbine to improve gas quality. The study introduces the novel RL-based controller and validates its effectiveness in real-world conditions using three-phase separators in Basra, Iraq, and through a review of relevant literature. The controller can adapt to inlet conditions such as pressure, temperature, mass flow rate, and incoming heat from the gas turbine. Waste heat recovery from the gas can enhance gas purity but also increase turbulence in water and oil. Maintaining a calm flow while ensuring high-speed flow over the baffle in the middle of the separator is crucial for optimal performance. The study considers two geometrical configurations of the vessel for redesigning the separator at the Basra refinery. The controller was implemented using the groovyBC utility within the OpenFOAM software. This model was then utilized to simulate real-world scenarios at the Basra refinery, displaying faster convergence, more rapid response, and more accurate tracking of the target fluid level. This study marks the initial effort to apply the deep deterministic policy gradient (DDPG) controller in computational fluid dynamic (CFD) work. The findings demonstrated a significant enhancement in separation efficiency by more than 36%, as well as smoother streamlines through the control and maintenance of pressure and velocity over the baffle.

Design of a Path-Following Controller for Autonomous Vehicles Using an Optimized Deep Deterministic Policy Gradient Method

Design of a Path-Following Controller for Autonomous Vehicles Using an Optimized Deep Deterministic Policy Gradient Method

The Optimal Strategies of Maneuver Decision in Air Combat of UCAV Based on the Improved TD3 Algorithm

Nowadays, unmanned aerial vehicles (UAVs) pose a significant challenge to air defense systems. Unmanned combat aerial vehicles (UCAVs) have been proven to be an effective method to counter the threat of UAVs in application. Therefore, maneuver decision-making has become the crucial technology to achieve autonomous air combat for UCAVs. In order to solve the problem of maneuver decision-making, an autonomous model of UCAVs based on the deep reinforcement learning method was proposed in this paper. Firstly, the six-degree-of-freedom (DoF) dynamic model was built in three-dimensional space, and the continuous actions of tangential overload, normal overload, and roll angle were selected as the maneuver inputs. Secondly, to improve the convergence speed for the deep reinforcement learning method, the idea of “scenario-transfer training” was introduced into the twin delayed deep deterministic (TD3) policy gradient algorithm, the results showing that the improved algorithm could cut off about 60% of the training time. Thirdly, for the “nose-to-nose turns”, which is one of the classical maneuvers for experienced pilots, the optimal maneuver generated by the proposed method was analyzed. The results showed that the maneuver strategy obtained by the proposed method was highly consistent with that made by experienced fighter pilots. This is also the first time in a public article that compared the maneuver decisions made by the deep reinforcement learning method with experienced fighter pilots. This research can provide some meaningful references to generate autonomous decision-making strategies for UCAVs.

A Wide-Range TCSC Based ADN in Mountainous Areas Considering Hydropower-Photovoltaic-ESS Complementarity

Due to the radial network structures, small cross-sectional lines, and light loads characteristic of existing AC distribution networks in mountainous areas, the development of active distribution networks (ADNs) in these regions has revealed significant issues with integrating distributed generation (DGs) and consuming renewable energy. Focusing on this issue, this paper proposes a wide-range thyristor-controlled series compensation (TCSC)-based ADN and presents a deep reinforcement learning (DRL)-based optimal operation strategy. This strategy takes into account the complementarity of hydropower, photovoltaic (PV) systems, and energy storage systems (ESSs) to enhance the capacity for consuming renewable energy. In the proposed ADN, a wide-range TCSC connects the sub-networks where PV and hydropower systems are located, with ESSs configured for each renewable energy generation. The designed wide-range TCSC allows for power reversal and improves power delivery efficiency, providing conditions for the optimization operation. The optimal operation issue is formulated as a Markov decision process (MDP) with continuous action space and solved using the twin delayed deep deterministic policy gradient (TD3) algorithm. The optimal objective is to maximize the consumption of renewable energy sources (RESs) and minimize line losses by coordinating the charging/discharging of ESSs with the operation mode of the TCSC. The simulation results demonstrate the effectiveness of the proposed method.

Sea-Based UAV Network Resource Allocation Method Based on an Attention Mechanism

As humans continue to exploit the ocean, the number of UAV nodes at sea and the demand for their services are increasing. Given the dynamic nature of marine environments, traditional resource allocation methods lead to inefficient service transmission and ping-pong effects. This study enhances the alignment between network resources and node services by introducing an attention mechanism and double deep Q-learning (DDQN) algorithm that optimizes the service-access strategy, curbs action outputs, and improves service-node compatibility, thereby constituting a novel method for UAV network resource allocation in marine environments. A selective suppression module minimizes the variability in action outputs, effectively mitigating the ping-pong effect, and an attention-aware module is designed to strengthen node-service compatibility, thereby significantly enhancing service transmission efficiency. Simulation results indicate that the proposed method boosts the number of completed services compared with the DDQN, soft actor–critic (SAC), and deep deterministic policy gradient (DDPG) algorithms and increases the total value of completed services.

Research on the Motion Control Strategy of a Lower-Limb Exoskeleton Rehabilitation Robot Using the Twin Delayed Deep Deterministic Policy Gradient Algorithm

The motion control system of a lower-limb exoskeleton rehabilitation robot (LLERR) is designed to assist patients in lower-limb rehabilitation exercises. This research designed a motion controller for an LLERR-based on the Twin Delayed Deep Deterministic policy gradient (TD3) algorithm to control the lower-limb exoskeleton for gait training in a staircase environment. Commencing with the establishment of a mathematical model of the LLERR, the dynamics during its movement are systematically described. The TD3 algorithm is employed to plan the motion trajectory of the LLERR’s right-foot sole, and the target motion curve of the hip (knee) joint is deduced inversely to ensure adherence to human physiological principles during motion execution. The control strategy of the TD3 algorithm ensures that the movement of each joint of the LLERR is consistent with the target motion trajectory. The experimental results indicate that the trajectory tracking errors of the hip (knee) joints are all within 5°, confirming that the LLERR successfully assists patient in completing lower-limb rehabilitation training in a staircase environment. The primary contribution of this study is to propose a non-linear control strategy tailored for the staircase environment, enabling the planning and control of the lower-limb joint motions facilitated by the LLERR.

Health-awareness energy management strategy for battery electric vehicles based on self-attention deep reinforcement learning

The economical and safe energy management strategy (EMS) is essential for a battery electric vehicle (BEV). With an improved deep deterministic policy gradient (DDPG)-based EMS proposed for a dual-motor BEV, energy consumption and electric component aging are minimized. Firstly, this study incorporates health awareness of electric components into EMS. Then, this study employs a gated recurrent unit (GRU) as well as a self-attention (SA) mechanism for improving the original DDPG's performance. The GRU enhances the performance of DDPG agents, post-convergence, by furnishing them with additional state information. The SA mechanism empowers agents to distinguish the importance of state information, thus improving its adaptability in new environments. The proposed EMS decreases battery energy consumption to 7.95 kW-hour and enhances battery state of health (SOH) to 96.83 % of the dynamic programming (DP)-based EMS. The two motors of the proposed EMS possess the highest end-value SOH, i.e., 99.99929 % and 99.99978 %, respectively. The proposed EMS demonstrates superior adaptability to other EMSs, regardless of the initial state of charge (SOC). It shows the best adaptability at an initial SOC of 0.6, and achieves 95.87 % and 94.67 % of the energy consumption of DP-based EMS in the mixed and actual test cycles, respectively.

Research on Obstacle Avoidance Strategy of Automated Heavy Vehicle Platoon in High-Speed Scenarios

Amidst the rapid progression of the road transport industry, the safety and efficiency of heavy-vehicle platoons have garnered significant attention. The study tackles the challenge of obstacle avoidance presented by vehicles owing to their considerable mass, delayed response times, and line-of-sight impediments, by introducing a cooperative obstacle avoidance system for heavy-vehicle platoons based on deep reinforcement learning. The system comprises three primary modules: perception, decision-making, and control. Initially, the perception module acquires real-time environmental data. Subsequently, the decision-making module formulates obstacle avoidance decisions based on the acquired data. Specifically, it implements a two-stage braking obstacle avoidance strategy under low collision risk scenarios, while employing a fifth-degree polynomial for planning and tracking obstacle avoidance paths under high collision risk conditions suitable for steering maneuvers. The control module utilizes the local multi-agent deep deterministic policy gradient (LADDPG) algorithm to train the heavy-vehicle platoon agents, ensuring the formation’s maintenance while mitigating collisions with other vehicles and obstacles. The effectiveness of the proposed system is substantiated through simulation experiments, demonstrating its adaptability to various traffic conditions, selection of suitable obstacle avoidance strategies, and significant enhancement of obstacle avoidance performance and heavy-vehicle platoon stability.

Adaptive Control for Underwater Simultaneous Lightwave Information and Power Transfer: A Hierarchical Deep-Reinforcement Approach

In this work, we consider a point-to-point underwater optical wireless communication scenario where an underwater sensor (US) transmits its sensing data to a remotely operated vehicle (ROV). Before the US transmits its data to the ROV, the ROV performs simultaneous lightwave information and power transfer (SLIPT), delivering both control data and lightwave power to the US. Under the considered scenario, our objective is to maximize energy harvesting at the US while supporting predetermined communication performance between the two nodes. To achieve this objective, we develop a hierarchical deep Q-network (DQN)–deep deterministic policy gradient (DDPG)-based online algorithm. This algorithm involves two reinforcement learning agents: the ROV and US. The role of the ROV agent is to determine an optimal beam-divergence angle that maximizes the received optical signal power at the US while ensuring a seamless optical link. Meanwhile, the US agent, which is influenced by the decision of the ROV agent, is responsible for determining the time-switching and power-splitting ratios to maximize energy harvesting without compromising the required communication performance. Unlike existing studies that do not account for adaptive parameter control in underwater SLIPT, the proposed algorithm’s adaptive nature allows for the dynamic fine-tuning of optimization parameters in response to varying underwater environmental conditions and diverse user requirements.

Few-Human-Interaction Reinforcement Learning for Autonomous Transbronchial Intervention.

The transbronchial interventional surgery presents challenges with winding and convoluted pathways, prone to compression and friction. Current autonomous planning struggles to reach deeper bronchial positions, and hard to consider multiple conflicting goals simultaneously. This article introduces an innovative planning scheme with preference weights to achieve smooth, frictionless, and collision-free autonomous transbronchial intervention with continuum robot (CR). A few-human-interaction twin-delayed deep deterministic policy gradient (FHITD3) generated from surgeon preference guidance is proposed, which determines the optimal strategy for the motion of CR. Preference knowledge is generated through interaction between human and few diversity samples. An abstract actuator space description is proposed for the posture and position representation of CR during movement within bronchus. A contact motion analysis strategy is proposed to calculate real-time attitude of CR in contact with bronchus. In addition, an oscillation suppression approach to address CR's unsmooth distal end trajectory is proposed. Simulated experiments show that the CR autonomously completes intervention tasks with a smooth and stable trajectory, reducing distal end oscillation by over 45%. It achieves a target endpoint within the fourth level bronchus (approximately 5 mm diameter) with over 90% probability.

Cost-effective task offloading and trajectory optimization in UAV assisted edge networks with DDPG

Purpose Unmanned aerial vehicles (UAVs), known for their exceptional flexibility and maneuverability, have become an integral part of mobile edge computing systems in edge networks. This paper aims to minimize system costs within a communication cycle. To this end, this paper has developed a model for task offloading in UAV-assisted edge networks under dynamic channel conditions. This study seeks to efficiently execute task offloading while satisfying UAV energy constraints, and validates the effectiveness of the proposed method through performance comparisons with other similar algorithms. Design/methodology/approach To address this issue, this paper proposes a task offloading and trajectory optimization algorithm using deep deterministic policy gradient, which jointly optimizes Internet of Things (IoT) device scheduling, power distribution, task offloading and UAV flight trajectory to minimize system costs. Findings The analysis of simulation results indicates that this algorithm achieves lower redundancy compared to others, along with reductions in task size by 22.8%, flight time by 34.5%, number of IoT devices by 11.8%, UAV computing power by 25.35% and the required cycle for per-bit tasks by 33.6%. Originality/value A multi-objective optimization problem is established under dynamic channel conditions, and the effectiveness of this approach is validated.

UAV-IRS-assisted energy harvesting for edge computing based on deep reinforcement learning

In the internet of everything (IoE) era, the proliferation of internet of things (IoT) devices is accelerating rapidly. Particularly, smaller devices are increasingly constrained by hardware limitations that impact their computational capacity, communication bandwidth, and battery longevity. Our research explores a multi-device, multi-access edge computing (MEC) environment within small cells to address the challenges posed by the hardware limitations of IoT devices in this environment. We employ wireless power transfer (WPT) to ensure these IoT devices have sufficient energy for task processing. We propose a system architecture in which an intelligent reflective surface (IRS) is carried by an unmanned aerial vehicle (UAV) to enhance communication conditions. For sustainable energy harvesting (EH), we integrate a normal distribution into the objective function. We utilize a softmax deep double deterministic policy gradients (SD3) algorithm, based on deep reinforcement learning (DRL), to optimize the computational and communication capabilities of IoT devices. Simulation experiments demonstrate that our SD3-based EH edge computing (EHEC-SD3) algorithm surpasses existing DRL algorithms in our explored environments, achieving more than 90% in overall optimization and EH performance.

Data-driven two-stage scheduling of multi-energy systems for operational flexibility enhancement

The multi-energy system, encompassing electricity networks, district heating networks (DHNs), and hydrogen-enriched compressed natural gas (HCNG) networks, provides an alternative for enhancing operational flexibility. This paper investigates a data-driven two-stage scheduling strategy of multi-energy system to promote uncertain renewable energy integration and improve economic benefits. Dynamic models for DHNs and HCNG networks are established, and the flexibility of multi-energy system is quantified through distribution-level power aggregation. To compromise flexibility enhancement and operational cost reduction, the multiobjective day-ahead scheduling optimization is conducted based on adaptive batch-ParEGO method. Taking day-ahead scheduling as a baseline, a data-driven real-time dispatch method is proposed based on the deep deterministic policy gradient algorithm, which adaptively modifies the energy management scheme in smaller temporal dimension to address uncertainties on both the load and source sides. The performance of the proposed two-stage scheduling method is demonstrated through numerical tests.

Learning-based hierarchical cooperative eco-driving with traffic flow prediction for hybrid electric vehicles

The integration of autonomous driving and hybrid electric vehicle technologies presents a promising solution for achieving environmental sustainability. This paper introduces an innovative energy-efficient driving strategy for hybrid electric vehicles that incorporates real-time traffic flow prediction. The study delves into the impact of both lateral and longitudinal vehicle maneuvers on energy consumption within dynamic traffic environments, offering novel insights into optimizing energy utilization. Firstly, a multi-lane traffic flow state rolling predictor is constructed based on the Hankel dynamic mode decomposition algorithm. Subsequently, a vehicle longitudinal and lateral coordinated control strategy is established by integrating the prioritized experience replay double deep Q-network algorithm. Finally, a novel energy management strategy is proposed that leverages Simulink dynamic model and the deep deterministic policy gradient algorithm to address the vehicle dynamic decision-making planning results. Within a hierarchical cooperative optimization framework, this research comprehensively considers safety, comfort, traffic efficiency, and fuel economy. By introducing a novel hierarchical collaborative ecological driving framework, we have achieved a substantial improvement in environmental sustainability, with traffic efficiency increasing by 10.27%-14.41% and fuel economy rising by 9.44%-10.47%. Hardware-in-the-loop validation has confirmed the proposed approach’s real-time capabilities and promising practical applications.

Adaptive Control of Ships’ Oil-Fired Boilers Using Flame Image-Based IMC-PID and Deep Reinforcement Learning

The control system of oil-fired boiler units on ships plays a crucial role in reducing the emissions of atmospheric pollutants such as nitrogen oxides (NOx), sulfur dioxides (SO2), and carbon dioxide (CO2). Traditional control methods using conventional measurement sensors face limitations in real-time control due to response delays, which has led to the growing interest in combustion control methods using flame images. To ensure the precision of such combustion control systems, the system model must be thoroughly considered during controller design. However, finding the optimal tuning point is challenging due to the changes in the system model and nonlinearity caused by environmental variations. This study proposes a controller that integrates an internal model control (IMC)-based PID controller with the deep deterministic policy gradient (DDPG) algorithm of deep reinforcement learning to enhance the adaptability of image-based combustion control systems to environmental changes. The proposed controller adjusts the PID parameter values in real-time through the learning of the determination constant lambda (λ) of the IMC internal model. This approach reduces computational resources by shrinking the learning dimensions of the DDPG agent and limits transient responses through constrained learning of control parameters. Experimental results show that the proposed controller exhibited rapid adaptive performance in the learning process for the target oxygen concentration, achieving a reward value of −0.05 within just 105 episodes. Furthermore, when compared to traditional PID tuning methods, the proposed controller demonstrated superior performance, achieving a target value error of 0.0032 and a low overshoot range of 0.0498 to 0.0631, providing the fastest response speed and minimal oscillation. Additionally, experiments conducted on an actual operating ship verified the practical feasibility of this system, highlighting its potential for real-time control and pollutant reduction in marine applications.

Research on priority scheduling strategy for smoothing power fluctuations of microgrid tie‐lines based on PER‐DDPG algorithm

AbstractThe variability of renewable energy within microgrids (MGs) necessitates the smoothing of power fluctuations through the effective scheduling of internal power equipment. Otherwise, significant power variations on the tie‐line connecting the MG to the main power grid could occur. This study introduces an innovative scheduling strategy that utilizes a data‐driven approach, employing a deep reinforcement learning algorithm to achieve this smoothing effect. The strategy prioritizes the scheduling of MG's internal power devices, taking into account the stochastic charging patterns of electric vehicles. The scheduling optimization model is initially described as a Markov decision process with the goal of minimizing power fluctuations on the interconnection lines and operational costs of the MG. Subsequently, after preprocessing the historical operational data of the MG, an enhanced scheduling strategy is developed through a neural network learning process. Finally, the results from four scheduling scenarios demonstrate the significant impact of the proposed strategy. Comparisons of reward curves before and after data preprocessing underscore its importance. In contrast to optimization results from deep deterministic policy gradient, soft actor‐critic, and particle swarm optimization algorithms, the superiority of the deep deterministic policy gradient algorithm with the addition of a priority experience replay mechanism is highlighted.

Optimizing Energy Harvesting in Wireless Body Area Networks: A Deep Reinforcement Learning Approach to Dynamic Sampling

In energy-harvesting wireless body area networks (EH-WBAN), the self-sustainability of body sensor nodes without compromising the service quality criteria is very important. In remote vital signs monitoring applications, the sampling rate of body nodes in each period should be determined to achieve this goal. There are two fundamental challenges in determining the sampling rate: 1) In EH-WBAN, the rate of the harvestable energy is time-dependent and unpredictable. 2) The vital signs of the patient have different change rates relevant to the time. Therefore, a technique is required that can learn the changes of the harvestable energy and the change rates of vital signs simultaneously and specify the proper sampling rate for BNs. The reinforcement learning algorithms are among the efficient solutions for this problem because they are capable of learning environmental uncertainties. Previous RL-based methods for determining the BN's sampling rate have three fundamental problems: 1) focus on the optimization of the transmission energy and ignore the sensing energy, 2) only affect one of the two aspects of energy or sensed data in determining the sampling rate and 3) discretization of the problem space does not ensure the determination of the optimal sampling rate. Therefore, this paper proposes a method named “deep reinforcement learning-based dynamic sampling” (DRDS) which first formulates the sampling rate determination problem as a Markov decision process (MDP) and proposes a deep deterministic policy gradient algorithm (DDPG) to solve it. The proposed method considers both energy and data variability aspects in determining the sampling rate. Two parameters, the super-capacitor’s voltage level, and ambient light intensity, are considered for the energy aspect; for the data variability aspect, the long short-term memory (LSTM) algorithm is developed to predict the change rate of data in the next round. Simulations indicate that by preserving the data integrity, the proposed method can decrease the sampling rate and unnecessary data transmission by about 49.95% and 89.7%, respectively, compared with state-of-the-art methods, and allow the sensor to achieve self-sustainability.

Experimental Implementation of a TD3 Agent Based Speed Controller for Direct Torque Control of PMSM Drives

This manuscript presents a novel control approach for Permanent Magnet Synchronous Motors (PMSMs) by integrating the widely used Direct Torque Control (DTC) method with Deep Reinforcement Learning (DRL), specifically the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. The TD3 algorithm is well-suited to address the challenges posed by high-dimensional state and action spaces in PMSM drives. The conventional Proportional–Integral (PI) controller in the outer loop of DTC is replaced with the DRL based TD3 agent to overcome the limitations of traditional PI controllers, such as model dependency and complex parameter tuning. The DRL technique allows the agent to learn optimal control policies from experience, making it suitable for complex and nonlinear control problems. Extensive simulation studies demonstrate the effectiveness of the TD3-based control in achieving accurate speed tracking under varying operating conditions. Real-time experimental validation using a TMS320F28379D digital signal processor confirms the feasibility of the proposed DRL based approach. The research offers new insights into improving the performance of PMSM drives and paves the way for future advancements in electric drive control.

Multi‐agent multi‐dimensional joint optimisation of jamming decision‐making against multi‐functional radar

AbstractThe authors propose a multi‐agent, multi‐dimensional joint optimisation decision‐making strategy to tackle the challenges jammers encounter when interfering with multi‐functional radars, which possess advanced anti‐interference capabilities. The strategy is divided into optimising the interference style and three parameters: timing, power, and duration, to enhance interference effectiveness. Furthermore, state value reuse is combined with the duelling double deep Q‐learning and multi‐agent deep deterministic policy gradient algorithms to effectively manage the multi‐dimensional optimisation of the interference strategy. Simulation results demonstrate that the proposed algorithm outperforms Ant‐QL, heuristic accelerated Q‐learning, and asynchronous advantage actor‐critic algorithms in terms of convergence speed and stability. It identifies a more efficient strategy for transitioning the radar from its initial state to the state with the lowest threat level to the jammer, achieving a 24% improvement in convergence speed. Additionally, the performance curve exhibits better stability.

A NoisyNet deep reinforcement learning method for frequency regulation in power systems

AbstractThis study thoroughly investigates the NoisyNet Deep Deterministic Policy Gradient (DDPG) for frequency regulation. Compared with the conventional DDPG method, the suggested method can provide several benefits. First, the parameter noise will explore different strategies more thoroughly and can potentially discover better policies that it might miss if only action noise were used, which helps the actor achieve an optimal control strategy, resulting in enhanced dynamic response. Second, by employing the delayed policy update policy work with the proposed framework, the training process exhibits faster convergence, enabling rapid adaptation to changing disturbances. To substantiate its efficacy, the scheme is subjected to simulation tests on both an IEEE three‐area power system, an IEEE 39 bus power system, and an IEEE 68 bus system. A comprehensive performance comparison was performed against other DDPG‐based methods to validate and evaluate the performance of the proposed LFC scheme.