Deep Deterministic Policy Gradient Algorithm Research Articles

A collaborative-learning multi-agent reinforcement learning method for distributed hybrid flow shop scheduling problem

As the increasing level of implementation of artificial intelligence technology in solving complex engineering optimization problems, various learning mechanisms, including deep learning (DL) and reinforcement learning (RL), have been developed for manufacturing scheduling. In this paper, a collaborative-learning multi-agent RL method (CL-MARL) is proposed for solving distributed hybrid flow-shop scheduling problem (DHFSP), minimizing both makespan and total energy consumption. First, the DHFSP is formulated as the Markov decision process, the features of machines and jobs are represented as state and observation matrixes according to their characteristics, the candidate operation set is used as action space, and a reward mechanism is designed based on the machine utilization. Next, a set of critic networks and actor networks, consist of recurrent neural networks and fully connected networks, are employed to map the states and observations into the output values. Then, a novel distance matching strategy is designed for each agent to select the most appropriate action at each scheduling step. Finally, the proposed CL-MARL model is trained through multi-agent deep deterministic policy gradient algorithm in collaborative-learning manner. The numerical results prove the effectiveness of the proposed multi-agent system, and the comparisons with existing algorithms demonstrate the high-potential of CL-MARL in solving DHFSP.

A novel method for solving dynamic flexible job-shop scheduling problem via DIFFormer and deep reinforcement learning

Due to the dynamic changes of manufacturing environments, heuristic scheduling rules are unstable in dynamic scheduling. Although meta-heuristic methods provide the best scheduling quality, their solution efficiency is limited by the scale of the problem. Therefore, a novel method for solving the dynamic flexible job-shop scheduling problem (DFJSP) via diffusion-based transformer (DIFFormer) and deep reinforcement learning (D-DRL) is proposed. Firstly, the DFJSP is modeled as a Markov decision process, where the state space is constructed in the form of the heterogeneous graph and the reward function is designed to minimize the makespan and maximize the machine utilization rate. Secondly, DIFFormer is used to encode the operation and machine nodes to better capture the complex dependencies between nodes, which can effectively improve the representation ability of the model. Thirdly, a selective rescheduling strategy is designed for dynamic events to enhance the solution quality of DFJSP. Fourthly, the twin delayed deep deterministic policy gradient (TD3) algorithm is adopted for training an efficient scheduling model. Finally, the effectiveness of the proposed D-DRL is validated through a series of experiments. The results indicate that D-DRL achieves better solution quality and higher solution efficiency when solving DFJSP instances.

Improved DDPG algorithm-based path planning for unmanned surface vehicles

As a promising mode of water transportation, unmanned surface vehicles (USVs) are used in various fields owing to their small size, high flexibility, favorable price, and other advantages. Traditional navigation algorithms are affected by various path planning issues. To address the limitations of the traditional deep deterministic policy gradient (DDPG) algorithm, namely slow convergence speed and sparse reward and punishment functions, we proposed an improved DDPG algorithm for USV path planning. First, the principle and workflow of the DDPG deep reinforcement learning (DRL) algorithm are described. Second, the improved method (based on the USVs kinematic model) is proposed, and a continuous state and action space is designed. The reward and punishment function are improved, and the principle of collision avoidance at sea is introduced. Dynamic region restriction is added, distant obstacles in the state space are ignored, and the nearby obstacles are observed to reduce the number of algorithm iterations and save computational resources. The introduction of a multi-intelligence approach combined with a prioritized experience replay mechanism accelerates algorithm convergence, thereby increasing the efficiency and robustness of training. Finally, through a combination of theory and simulation, the DDPG DRL is explored for USV obstacle avoidance and optimal path planning.

Decision-Making Policy for Autonomous Vehicles on Highways Using Deep Reinforcement Learning (DRL) Method

Automated driving (AD) is a new technology that aims to mitigate traffic accidents and enhance driving efficiency. This study presents a deep reinforcement learning (DRL) method for autonomous vehicles that can safely and efficiently handle highway overtaking scenarios. The first step is to create a highway traffic environment where the agent can be guided safely through surrounding vehicles. A hierarchical control framework is then provided to manage high-level driving decisions and low-level control commands, such as speed and acceleration. Next, a special DRL-based method called deep deterministic policy gradient (DDPG) is used to derive decision strategies for use on the highway. The performance of the DDPG algorithm is compared with that of the DQN and PPO algorithms, and the results are evaluated. The simulation results show that the DDPG algorithm can effectively and safely handle highway traffic tasks.

TD3-based trajectory optimization for energy consumption minimization in UAV-assisted MEC system

Unmanned Aerial Vehicle (UAV) assisted Mobile Edge Computing (MEC) systems provide substantial benefits for task offloading and communication services, especially in situations where traditional communication infrastructure is unavailable. Current research emphasizes maintaining communication quality while minimizing total energy consumption and optimizing UAV flight trajectories. However, several issues remain: First, the energy consumption objective function lacks comprehensiveness, neglecting the impact of UAV flight energy consumption; second, an effective Deep Reinforcement Learning (DRL) algorithm has not been employed to address the non-convexity of the objective function; third, there is insufficient discussion regarding the practical significance of the proposed approach. To address these issues, this paper formulates an objective function aimed at minimizing MEC energy consumption by considering task offloading decisions, communication delays, computational energy consumption, and UAV flight energy consumption. We propose a Population Diversity-based Particle Swarm Optimization-Double Delay Deep Deterministic Policy Gradient (PDPSO-TD3) algorithm to find the optimal solution, enhance UAV flight trajectories through optimized offloading decisions, ensure efficient communication, and minimize the total energy consumption of the MEC system. Furthermore, we discuss the practical applicability of PDPSO-TD3 in detail and present the proposed scheme. Experimental results demonstrate that compared to the Deep Deterministic Policy Gradient (DDPG) algorithm, for transmission delay, MEC energy consumption, UAV flight energy consumption, and User Equipments (UEs) access rate metrics. The proposed PDPSO-TD3 algorithm can improvement the performance by about 14.3%, 10.1%, 6.1%, and 3.3%, respectively.

Human-in-the-Loop Reinforcement Learning in Continuous-Action Space.

Human-in-the-loop for reinforcement learning (RL) is usually employed to overcome the challenge of sample inefficiency, in which the human expert provides advice for the agent when necessary. The current human-in-the-loop RL (HRL) results mainly focus on discrete action space. In this article, we propose a Q value-dependent policy (QDP)-based HRL (QDP-HRL) algorithm for continuous action space. Considering the cognitive costs of human monitoring, the human expert only selectively gives advice in the early stage of agent learning, where the agent implements human-advised action instead. The QDP framework is adapted to the twin delayed deep deterministic policy gradient algorithm (TD3) in this article for the convenience of comparison with the state-of-the-art TD3. Specifically, the human expert in the QDP-HRL considers giving advice in the case that the difference between the twin Q -networks' output exceeds the maximum difference in the current queue. Moreover, to guide the update of the critic network, the advantage loss function is developed using expert experience and agent policy, which provides the learning direction for the QDP-HRL algorithm to some extent. To verify the effectiveness of QDP-HRL, the experiments are conducted on several continuous action space tasks in the OpenAI gym environment, and the results demonstrate that QDP-HRL greatly improves learning speed and performance.

Peer-to-Peer Energy Transactions for Prosumers Based on Improved Deep Deterministic Policy Gradient Algorithm

Peer-to-Peer Energy Transactions for Prosumers Based on Improved Deep Deterministic Policy Gradient Algorithm

The docking control system of an autonomous underwater vehicle combining intelligent object recognition and deep reinforcement learning

This study develops a visual-based docking system (VDS) for an autonomous underwater vehicle (AUV), significantly enhancing docking performance by integrating intelligent object recognition and deep reinforcement learning (DRL). The system overcomes traditional navigation limitations in complex and unpredictable environments by using a variable information dock (VID) for precise multi-sensor docking recognition in the AUV. Employing image-based visual servoing (IBVS) technology, the VDS efficiently converts 2D visual data into accurate 3D motion control commands. It integrates the YOLO (short for You Only Look Once) algorithm for object recognition and the deep deterministic policy gradient (DDPG) algorithm, improving continuous motion control, docking accuracy, and adaptability. Experimental validation at the National Cheng Kung University towing tank demonstrates that the VDS enhances control stability and operational reliability, reducing the mean absolute error (MAE) in depth control by 42.03% and pitch control by 98.02% compared to the previous method. These results confirm the VDS's reliability and its potential for transforming AUV docking.

Quadrotor Trajectory Tracking using Combined Stochastic Model-Free Position and DDPG-based Attitude Control

This article presents a cascade controller for the quadrotor to track the desired trajectory effectively. Unlike previous approaches, this method avoids simplification and linearization assumptions, making it applicable in a wider range of scenarios. A novel linear quadratic tracking method is utilized, which takes into account both process noise and measurement noise while maintaining a model-free nature. Furthermore, the stability analysis of this stochastic method is thoroughly investigated. In terms of attitude control, a model-free approach is adopted. The Deep Deterministic Policy Gradient (DDPG) algorithm is implemented, leveraging an actor-critic network to handle the nonlinearities associated with attitude control. This model-free approach eliminates the need for an accurate model of the quadrotor's dynamics. Simulations are conducted to evaluate the performance of the proposed controller, and the results demonstrate its ability to effectively control the quadrotor, ensuring accurate trajectory tracking and stability.

Deep reinforcement learning-based multi-objective optimization for electricity–gas–heat integrated energy systems

With the increasing global attention on energy efficiency and carbon emissions, the optimization of integrated energy systems (IES) has become the key to improve energy efficiency and reduce pollution emissions. However, most of the existing optimization methods cannot effectively deal with the complexity of high dimensional continuous action space. Therefore, this paper focuses on a novel multi-objective optimization strategy for the electricity–gas–heat integrated energy systems (EGH-IES). Firstly, considering the absorption capacity of wind power and the emission of pollutant gases, a multi-objective optimization model is constructed based on the mechanism model and operation constraints of each device in EGH-IES, in which the integrated operation cost and the environmental factors are taken as optimization objectives. Then, the multi-objective optimization problem is designed as the optimal strategy of interaction learning between agent and environment in reinforcement learning, and the output power of the devices constitutes the action of reinforcement learning. Additionally, the Ornstein–Uhlenbeck process is introduced to enhance the training efficiency and exploration performance of the agent, and the deep deterministic policy gradients (DDPG) algorithm is employed to optimize the action, thus the output power of the appliances could be obtained. Finally, the simulation results show that compared with deep Q network (DQN) method and proximal policy optimization (PPO) method, the reward function value of the proposed method increases by 2.43% and 6.09%, respectively, which represents a reduction in economic cost and pollutant emissions. These verify the effectiveness and superiority of the proposed multi-objective optimization scheme in cost reduction and benefit improvement for the EGH-IES.

Design of sliding mode controller for servo feed system based on generalized extended state observer with reinforcement learning

Nonlinear friction, system uncertainty, and external disturbances have a significant impact on the performance of high-precision servo feed systems. In order to achieve higher tracking accuracy, a sliding mode controller based on a generalized extended state observer with the double critic deep deterministic policy gradient algorithm is designed. Firstly, a flexible two mass drive model (FTMDM) is established for the two-axis differential micro-feed system (TDMS). Next, a generalized extended state observer (GESO) is designed to estimate matched interference and mismatched interference. And it is proved that the observation error of GESO is bounded. A sliding mode controller based on GESO is further proposed. The stability of the controller has been proven through Lyapunov theory, and the error is bounded and converges to zero in finite time. The tuning process of controller parameters is simplified by using quadratic optimal control principle. Furthermore, a double critic deep deterministic policy gradient algorithm (DCDDPG) is proposed to achieve dynamic optimization of parameters about GESO. The simulation results show that GESO with DCDDPG can reduce the observation error of step signal and sinusoidal signal, and improve the observation accuracy of nonlinear friction significantly. Finally, experimental results show that the proposed control method achieves more accurate position tracking performance on TDMS.

Adaptive MPC path-tracking controller based on reinforcement learning and preview-based PID controller

Path-tracking control is a crucial process for autonomous vehicles, ensuring that the vehicle drives safely along the reference path, and the suitable controller parameters ensure the accuracy and stability of this process. To enhance the adaptability of traditional path-tracking controller parameters, this paper proposes an adaptive model predictive control (MPC) controller based on a preview-based PID controller and deep deterministic policy gradient (DDPG) algorithm to achieve adaptive tuning of the controller parameters. Starting with the design of the dynamics tracking error model of the vehicle and the MPC controller. Based on the actor-critic reinforcement learning architecture, the DDPG agent is designed to tune the prediction horizon and weight matrix of the MPC controller. A preview-based PID controller is proposed to improve the efficiency and stability of reinforcement learning and compensate for the error in vehicle modeling. The improved algorithm performance is verified through the simulation scenarios of high-speed lane changing and accelerated overtaking scenarios constructed by MATLAB/Simulink. The results show that the improved algorithm significantly improves the adaptive ability of the traditional MPC controller to time-varying conditions and achieves higher tracking accuracy and stability.

Adaptive Deep Ant Colony Optimization–Asymmetric Strategy Network Twin Delayed Deep Deterministic Policy Gradient Algorithm: Path Planning for Mobile Robots in Dynamic Environments

Path planning is one of the main focal points and challenges in mobile robotics research. Traditional ant colony optimization (ACO) algorithms encounter issues such as low efficiency, slow convergence, and a tendency to become stuck in local optima and search stagnation when applied to complex dynamic environments. Addressing these challenges, this study introduces an adaptive deep ant colony optimization (ADACO) algorithm, which significantly improves efficiency and convergence speed through enhanced pheromone diffusion mechanisms and updating strategies, applied to global path planning. To adapt to dynamically changing environments and achieve more precise local path planning, an asymmetric strategy network TD3 algorithm (ATD3) is further proposed, which utilizes global path planning information within the strategy network only, creating a new hierarchical path planning algorithm—ADACO-ATD3. Simulation experiments demonstrate that the proposed algorithm significantly outperforms in terms of path length and number of iterations, effectively enhancing the mobile robot’s path planning performance in complex dynamic environments.

Integrating model-driven and data-driven methods for under-frequency load shedding control

With the access of a high percentage of new energy sources, power system frequency stability is challenged. Under-frequency load shedding (UFLS) is one of the primary measures to maintain frequency stability. Due to the mismatch between the amount of load shed by the traditional UFLS methods and the actual active power deficit, a new UFLS method needs to be designed. An approach utilizing a deep deterministic policy gradient (DDPG) algorithm for the problem is proposed. First, the DDPG algorithm is modified to adapt to the UFLS problem. Then, the idea of model-driven is introduced to improve the validity of the model. Thus, a novel UFLS method is proposed, which integrates data-driven and model-driven ideas. Furthermore, a structure called the dual experience pool is designed to accelerate training speed and improve stability. Based on the proposed method, a UFLS control framework is designed. Finally, the suggested methodology is validated using the IEEE-39 bus system and the Fujian power grid as test cases.

Reinforcement Learning-Based Optimal Hull Form Design with Variations in Fore and Aft Parts

Abstract With recent advancements in artificial intelligence technology, various studies are being conducted in the shipbuilding industry. Traditionally, hull form variation methods have relied on the intuition and expertise of designers, leading to inconsistent results and unintended changes in the ship's main dimensions depending on the designer's competence. Moreover, the iterative process of design variation and analysis to derive the optimal hull form is both costly and time-consuming. To address these issues, this study proposes an optimal hull design technique utilizing reinforcement learning, a type of unsupervised learning in machine learning. Reinforcement learning allows the model to learn from past policies by recording and accumulating the rewards associated with various actions taken by an agent in a specific environment. In this study, after calculating the main parameters of the ship, the agent defines a state representing hull information and performs local transformations of the bow and stern. The reward of reinforcement learning is defined as the change in total resistance due to the hull deformation, constrained by limiting the tolerance of the ship's prismatic coefficient (CP) and longitudinal center of buoyancy (LCB). In this study, the problem is solved by comparing the proximal policy optimization (PPO) algorithm and the deep deterministic policy gradient (DDPG) algorithm to find the best deep reinforcement learning (DRL) model for the hull optimization problem. The results were compared with the genetic algorithm and speed-constrained multi-objective particle swarm optimization (SMPSO), and the optimal hull resistance values were less different, but the time of the reinforcement learning model was five times shorter.

A multi-objective reinforcement learning framework for real-time drilling optimization based on symbolic regression and perception

In the context of the global energy transition, optimizing deep-water oil and gas drilling parameters is crucial for ensuring safety while improving efficiency. Traditional methods face limitations in highly dynamic and nonlinear drilling environments, struggling to balance speed and cost-effectiveness. Furthermore, these methods rely on real-time logging-while-drilling (LWD) data for decision-making, but delays in data collection and processing hinder timely adjustments of drilling parameters, affecting decision accuracy and responsiveness. This paper proposes a multi-objective drilling parameter optimization framework, incorporating symbolic regression, time-series networks, and Markov decision processes to precisely predict ROP, formation conditions, and optimize drilling parameters in real time. Key innovations include a multi-population evolutionary symbolic regression algorithm for constructing empirical equations, the integration of variational mode decomposition (VMD) and sample entropy for data preprocessing, and multi-head self-attention time-series networks to enhance prediction accuracy. Quantile regression further estimates the range of drilling parameter adjustments. Additionally, a drilling parameter optimization deep deterministic policy gradient (DPODDPG) algorithm was developed to automate real-time parameter adjustments. Empirical analysis on the Ledong 10-1 block in the South China Sea demonstrated significant improvements: ROP increased from 54.18 m/hr to 122.17 m/hr, mechanical specific energy (MSE) decreased from 100.82 MPa to 97.78 MPa, and cost per foot reduced from 121.16 × 102 CNY/m to 51.31 × 102 CNY/m. Compared to traditional methods, the proposed framework showed clear advantages in enhancing ROP, reducing MSE, and controlling costs, further validating its superiority in complex drilling environments. This method not only significantly improves drilling efficiency and economic benefits but also adapts to complex and changing drilling conditions, showing broad application potential, particularly in challenging deep-water oil and gas drilling operations, where it can provide more efficient and reliable optimization solutions.

Multi-Agent DRL for Air-to-Ground Communication Planning in UAV-Enabled IoT Networks.

In this paper, we present a novel method to enhance the sum-rate effectiveness in full-duplex unmanned aerial vehicle (UAV)-assisted communication networks. Existing approaches often couple uplink and downlink associations, resulting in suboptimal performance, particularly in dynamic environments where user demands and network conditions are unpredictable. To overcome these limitations, we propose a decoupling of uplink and downlink associations for ground-based users (GBUs), significantly improving network efficiency. We formulate a comprehensive optimization problem that integrates UAV trajectory design and user association, aiming to maximize the overall sum-rate efficiency of the network. Due to the problem's non-convexity, we reformulate it as a Partially Observable Markov Decision Process (POMDP), enabling UAVs to make real-time decisions based on local observations without requiring complete global information. Our framework employs multi-agent deep reinforcement learning (MADRL), specifically the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, which balances centralized training with distributed execution. This allows UAVs to efficiently learn optimal user associations and trajectory controls while dynamically adapting to local conditions. The proposed solution is particularly suited for critical applications such as disaster response and search and rescue missions, highlighting the practical significance of utilizing UAVs for rapid network deployment in emergencies. By addressing the limitations of existing centralized and distributed solutions, our hybrid model combines the benefits of centralized training with the adaptability of distributed inference, ensuring optimal UAV operations in real-time scenarios.

Energy management with adaptive moving average filter and deep deterministic policy gradient reinforcement learning for fuel cell hybrid electric vehicles

Fuel cell hybrid electric vehicles (FCHEV) with battery (BAT) and supercapacitor (SC) advance in flexible configuration and high energy efficiency. However, the complex coupling relationship among various power sources poses a severe challenge to the design of the energy management system (EMS), including multi-degrees of freedom power allocation, fuel economy, and power sources lifespan of the FCHEV. This paper proposes an EMS based on a dual-layer power distribution structure. In the upper layer, adaptive moving average filter (AMAF) is designed to separate different frequency power, where the energy supply of the SC is managed to attenuate fluctuating power and simplifies the optimization problem and reduces computational costs. The lower layer is constructed by the deep deterministic policy gradient (DDPG) algorithm, where fuel cell system (FCS) hydrogen consumption and degradation rewards are designed to simultaneously enhance fuel efficiency and degradation performance by regulating the FCS real-time power variation. The proposed strategy has been evaluated regarding FCHEV fuel economy and FCS durability under combined driving cycle simulation, which shows AMAF + DDPG strategy reduces fuel consumption by 7.24 % and 1.3 %, also the degradation reduces by 0.04 % and 0.02 % compared with different EMS. Simulation results demonstrate that AMAF + DDPG optimizes the output characteristics of power sources.

Joint Computation Offloading and Trajectory Optimization for Edge Computing UAV: A KNN-DDPG Algorithm

Unmanned aerial vehicles (UAVs) are widely used to improve the coverage and communication quality of wireless networks and assist mobile edge computing (MEC) due to their flexible deployments. However, the UAV-assisted MEC systems also face challenges in terms of computation offloading and trajectory planning in the dynamic environment. This paper employs deep reinforcement learning to jointly optimize the computation offloading and trajectory planning for UAV-assisted MEC system. Specifically, this paper investigates a general scenario where multiple pieces of user equipment (UE) offload tasks to a UAV equipped with a MEC server to collaborate on a complex job. By fully considering UAV and UE movement, computation offloading ratio, and blocked relations, a joint computation offloading and trajectory optimization problem is formulated to minimize the maximum computational delay. Due to the non-convex nature of the problem, it is converted into a Markov decision process, and solved by the deep deterministic policy gradient (DDPG) algorithm. To enhance the exploration capability and stability of DDPG, the K-nearest neighbor (KNN) algorithm is employed, namely KNN-DDPG. Moreover, the prioritized experience replay algorithm, where the constant learning rate is replaced by the decaying learning rate, is utilized to enhance the converge. To validate the effectiveness and superiority of the proposed algorithm, KNN-DDPG is compared with the benchmark DDPG algorithm. Simulation results demonstrate that KNN-DDPG can converge and achieve 3.23% delay reduction compared to DDPG.

Advanced security measures in coupled phase-shift STAR-RIS networks: A DRL approach

The rapid development of next-generation wireless networks has intensified the need for robust security measures, particularly in environments susceptible to eavesdropping. Simultaneous transmitting and reflecting reconfigurable intelligent surfaces (STAR-RIS) have emerged as a transformative technology, offering full-space coverage by manipulating electromagnetic wave propagation. However, the inherent flexibility of STAR-RIS introduces new vulnerabilities, making secure communication a significant challenge. To overcome these challenges, we propose a deep reinforcement learning (DRL) based secure and efficient beamforming optimization strategy, leveraging the deep deterministic policy gradient (DDPG) algorithm. By framing the problem as a Markov decision process (MDP), our approach enables the DDPG algorithm to learn optimal strategies for beamforming and transmission and reflection coefficients (TARCs) configurations. This method is specifically designed to optimize phase-shift coefficients within the STAR-RIS environment, effectively managing the coupled phase shifts and complex interactions that are critical for enhancing physical layer security (PLS). Through extensive simulations, we demonstrate that our DRL-based strategy not only outperforms traditional optimization techniques but also achieves real-time adaptive optimization, significantly improving both confidentiality and network efficiency. This research addresses key gaps in secure wireless communication and sets a new standard for future advancements in intelligent, adaptable network technologies.