Deep Deterministic Policy Gradient Algorithm Research Articles

Deep Reinforcement Learning for Nash Equilibrium of Differential Games.

Nash equilibrium is a significant solution concept representing the optimal strategy in an uncooperative multiagent system. This study presents two deep reinforcement learning (DRL) algorithms for solving the Nash equilibrium of differential games. Both algorithms are built upon the distributed distributional deep deterministic policy gradient (D4PG) algorithm, which is a one-sided learning method. We modified it to a two-sided adversarial learning method. The first is D4PG for games (D4P2G), which directly applies an adversarial play framework based on the D4PG. A simultaneous policy gradient descent (SPGD) method is employed to optimize the policies of the players with conflicting objectives. The second is the distributional deep deterministic symplectic policy gradient (D4SPG) algorithm, which is our main contribution. More specifically, it newly designs a minimax learning framework that combines the critics of the two players and proposes a symplectic policy gradient adjustment method to find a better policy gradient. Simulations show that both algorithms converge to the Nash equilibrium in most cases, but D4SPG can learn the Nash equilibrium more accurately and efficiently, especially in Hamiltonian games. Moreover, it can handle games with complex dynamics, which is challenging for traditional methods.

Health-conscious deep reinforcement learning energy management for fuel cell buses integrating environmental and look-ahead road information

The escalating level of vehicle electrification and intelligence makes higher requirements for the energy management strategy (EMS) of fuel cell vehicles. Environmental and road conditions can significantly influence the power demand of the load, thereby affecting the lifespan and efficiency of vehicular energy systems. To ensure that the vehicle is always in optimal working condition, this study innovatively proposes a health-conscious EMS framework based on twin delayed deep deterministic policy gradient (TD3) algorithm for fuel cell hybrid electric bus (FCHEB). First, the environment and look-ahead road information obtained through vehicle sensors, GPS and Geographic Information System is used to establish the energy management problem formulation. Secondly, a TD3-based data-driven EMS is developed with the objective of optimizing hydrogen fuel economy, fuel cell durability and battery thermal health status. Finally, the strategy validation is performed in a developed validation environment that contains terrain information, ambient temperature, and real-world collected driving conditions. The validation results indicate that compared to the state-of-the-art TD3-based EMS, the proposed EMS can improve battery life by 28.02 % and overall vehicle economy by 8.92 %.

Application of Deep Reinforcement Learning for Proportional–Integral–Derivative Controller Tuning on Air Handling Unit System in Existing Commercial Building

An effective control of air handling unit (AHU) systems is crucial not only for managing the energy consumption of buildings but ensuring indoor thermal comfort for occupants. Although the initial control schema of AHU is appropriate at installation and testing, it is frequently necessary to adjust the control variables due to the changing thermal response of the building envelope and space usage. This paper presents a novel optimization process for the control parameters of old AHU systems in existing commercial buildings without system downtime and massive operational data. First, calibrating the building and system simulator with limited system operation data and unknown building parameters can provide identical responses to the system operation with the Hooke–Jeeves algorithm during the cooling season. The deep deterministic policy gradient algorithm is employed to determine the optimal control parameters for the valve opening position of the cooling coil within less than three hours of training based on the calibrated simulator. By using actual implementations with the developed optimal control variables for an old AHU in a real building, the proposed auto-tuned PID control in the simulator and with machine learning improves thermal environments with a steady room temperature (23.5 ± 0.5 °C) by 97% in occupied periods. It is also proved that this can reduce cooling energy consumption by up to 13.71% on a daily average. The successful AHU controller can improve not only the stability of AHU systems but the efficiency of a building’s energy use and indoor thermal comfort.

Design of Intelligent Controller for Aero-engine Based on TD3 Algorithm

Recently, higher structure complicacy and performances requirements of the aero-engine have brought higher demands on its control system. For the control of aerodynamic thermodynamic system, the intelligent control method with self-learning ability will be a promising choice. In the paper, we propose an aero-engine intelligent controller design method based on twin delayed deep deterministic policy gradient (TD3) algorithm. The method enables the intelligent controller to learn continuously according to the feedback of the environment and control the aero-engine. The paper takes the intelligent controller design of the JT9D turbofan engine as an example. First, the aero-engine control problem is described as a Markov decision process for deep reinforcement learning algorithms. Second, a complete intelligent controller design process is constructed by reasonably designing the network structure and reward function. Finally, the comparison simulations are conducted to verify the effectiveness of the proposed methods. The simulation results show that the TD3 controller outperforms deep deterministic policy gradient (DDPG) and the proportional-integral-derivative (PID) in the aero-engine control task. And the TD3 controller can realize the tracking control of low-pressure turbine speed with quick response and small overshoot.

Realizing asynchronous finite-time robust tracking control of switched flight vehicles by using nonfragile deep reinforcement learning.

In this study, a novel nonfragile deep reinforcement learning (DRL) method was proposed to realize the finite-time control of switched unmanned flight vehicles. Control accuracy, robustness, and intelligence were enhanced in the proposed control scheme by combining conventional robust control and DRL characteristics. In the proposed control strategy, the tracking controller consists of a dynamics-based controller and a learning-based controller. The conventional robust control approach for the nominal system was used for realizing a dynamics-based baseline tracking controller. The learning-based controller based on DRL was developed to compensate model uncertainties and enhance transient control accuracy. The multiple Lyapunov function approach and mode-dependent average dwell time approach were combined to analyze the finite-time stability of flight vehicles with asynchronous switching. The linear matrix inequalities technique was used to determine the solutions of dynamics-based controllers. Online optimization was formulated as a Markov decision process. The adaptive deep deterministic policy gradient algorithm was adopted to improve efficiency and convergence. In this algorithm, the actor-critic structure was used and adaptive hyperparameters were introduced. Unlike the conventional DRL algorithm, nonfragile control theory and adaptive reward function were used in the proposed algorithm to achieve excellent stability and training efficiency. We demonstrated the effectiveness of the presented algorithm through comparative simulations.

Double RISs assisted task offloading for NOMA-MEC with action-constrained deep reinforcement learning

Reconfigurable intelligent surface (RIS) is expected to enhance task offloading performance in non-line-of-sight mobile edge computing (MEC) scenarios. This paper aims at reducing the long-term energy consumption for task offloading in a MEC system assisted by double RISs while considering non-orthogonal multiple access (NOMA). Due to the non-convexity of the formulated problem, we propose an action-constrained deep reinforcement learning (DRL) framework based on twin delayed deep deterministic policy gradient (TD3) algorithm that includes inner and outer optimization processes, which greatly reduces the action space size of the DRL agent, making it easy to implement and achieve fast convergence. During the inner optimization phase, by utilizing theoretical derivations, we propose a low-complexity method to optimally derive the transmit power, the local computing frequency, and the computation resource allocation in the base station. In the outer optimization phase, building on the solution derived from the inner optimization, we further use the TD3 algorithm to jointly determine the phase shifting of RIS elements, the offloading ratio, and the transmission time for each time slot. Experiment results demonstrate that the proposed algorithm achieves rapid convergence performance. In comparison to single RIS assisted offloading, the double RISs assisted offloading scheme proposed in this paper can reduce energy consumption by 42.8% on average.

Joint Task Offloading and Resource Allocation for Intelligent Reflecting Surface-Aided Integrated Sensing and Communication Systems Using Deep Reinforcement Learning Algorithm.

This paper investigates an intelligent reflecting surface (IRS)-aided integrated sensing and communication (ISAC) framework to cope with the problem of spectrum scarcity and poor wireless environment. The main goal of the proposed framework in this work is to optimize the overall performance of the system, including sensing, communication, and computational offloading. We aim to achieve the trade-off between system performance and overhead by optimizing spectrum and computing resource allocation. On the one hand, the joint design of transmit beamforming and phase shift matrices can enhance the radar sensing quality and increase the communication data rate. On the other hand, task offloading and computation resource allocation optimize energy consumption and delay. Due to the coupled and high dimension optimization variables, the optimization problem is non-convex and NP-hard. Meanwhile, given the dynamic wireless channel condition, we formulate the optimization design as a Markov decision process. To tackle this complex optimization problem, we proposed two innovative deep reinforcement learning (DRL)-based schemes. Specifically, a deep deterministic policy gradient (DDPG) method is proposed to address the continuous high-dimensional action space, and the prioritized experience replay is adopted to speed up the convergence process. Then, a twin delayed DDPG algorithm is designed based on this DRL framework. Numerical results confirm the effectiveness of proposed schemes compared with the benchmark methods.

Multimedia Computation Offloading for 5G Networks Using the Volcano Eruption Optimization Algorithm

The communication services of multimedia, such as live online and short videos, gained significant attention and have been prominent in daily social interactions due to the fast development of 5G technology. However, the increase in multimedia communication requirements presents further difficulties for both the communication capacity of the wireless network and its processing capability. Mobile Edge Computing (MEC) is considered an effective technology to overcome the mentioned challenges. The subject of long-term task allocation including resource coordination is the primary focus of this study, with a special emphasis on multimedia services that must be processed, uploaded, and shared in network. The optimization issue is formulated as a stochastic optimization issue to reduce the system’s time-average energy consumption. An online energy-efficient task allocation and computing offloading method is proposed to determine task allocation, coordinate and optimize wireless, computing resource allocation, considering dynamic wireless state and service delay constraints. The proposed method is implemented in MATLAB and the efficacy of the MCO-VEA framework is estimated and performance metrics, such as minimum user rate, sum rate, energy, and spectral performance are analyzed. Then, the performance of MEC-MCO-VEA provides 22.35%, 25.84% and 36.58% higher throughput and 25.45%, 41.28%, and 28.56% higher energy efficiency compared to the existing models, such as an energy-effective multimedia task assignment with computing offloading for mobile edge computing (BA-LO-EE), energy efficiency using a communication deep neural network optimized with power allocation optimization (EE-CDNN-PAO), and energy efficiency using deep transfer deterministic policy gradient with deep deterministic policy gradient algorithm (DT-DPG-EE), respectively.

Reinforcement learning for Hybrid Disassembly Line Balancing Problems

With the rapid development of the economy and technology, the rate of product replacement has accelerated, resulting in a large number of products being discarded. Disassembly is an important way to recycle waste products, which is also helpful to reduce manufacturing costs and environmental pollution. The combination of a single-row linear disassembly line and a U-shaped disassembly line presents distinctive advantages within various application scenarios. The Hybrid Disassembly Line Balancing Problem (HDLBP) that considers the requirement of multi-skilled workers is addressed in this paper. A mathematical model is established to maximize the recovery profit according to the characteristics of the proposed problem. To facilitate the search for optimal solution, a new strategy for agents in reinforcement learning to interact with complex and changeable environments in real-time is developed, and deep reinforcement learning is used to complete the distribution of multi-products and disassembly tasks. On this basis, we propose a Soft Actor–Critic (SAC) algorithm to effectively address this problem. Compared with the Deep Deterministic Policy Gradient (DDPG) algorithm, Advantage Actor–Critic (A2C) algorithm, and Proximal Policy Optimization (PPO) algorithm, the results show that the SAC can get the approximate optimal result on small-scale cases. The performance of SAC is also better than DDPG, PPO, and A2C in solving large-scale disassembly cases.

A perspective on the use of deep deterministic policy gradient reinforcement learning for retention time modeling in reversed-phase liquid chromatography

Artificial intelligence and machine learning techniques are increasingly used for different tasks related to method development in liquid chromatography. In this study, the possibilities of a reinforcement learning algorithm, more specifically a deep deterministic policy gradient algorithm, are evaluated for the selection of scouting runs for retention time modeling. As a theoretical exercise, it is investigated whether such an algorithm can be trained to select scouting runs for any compound of interest allowing to retrieve its correct retention parameters for the three-parameter Neue-Kuss retention model. It is observed that three scouting runs are generally sufficient to retrieve the retention parameters with an accuracy (mean relative percentage error MRPE) of 1 % or less. When given the opportunity to select additional scouting runs, this does not lead to a significantly improved accuracy. It is also observed that the agent tends to give preference to isocratic scouting runs for retention time modeling, and is only motivated towards selecting gradient scouting runs when penalized (strongly) for large analysis/gradient times. This seems to reinforce the general power and usefulness of isocratic scouting runs for retention time modeling. Finally, the best results (lowest MRPE) are obtained when the agent manages to retrieve retention time data for % ACN at elution of the compound under consideration that spread the entire relevant range of ACN (5 % ACN to 95 % ACN) as well as possible, i.e., resulting in retention data at a low, intermediate and high % ACN. Based on the obtained results, we believe reinforcement learning holds great potential to automate and rationalize method development in liquid chromatography in the future.

Optimizing Automatic Voltage Regulation: A Deep Reinforcement Learning Approach

This paper presents a novel approach to enhance the performance of Automatic Voltage Regulator (AVR) systems in power systems using Deep Reinforcement Learning (DRL). The AVR plays a critical role in maintaining voltage stability and ensuring reliable power delivery. However, conventional control strategies, such as PID controllers, have limitations in handling complex and nonlinear power system dynamics. In this study, the application of DRL techniques is explored, particularly the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm, to AVR control. This algorithm offers the advantage of handling continuous action spaces and enable the controller to learn optimal control policies directly from the system's state information. The results show that the DRL approach outperforms the traditional PID and Neural Network-based control approaches, with the shortest time response and the best voltage regulation performance. The use of DRL in AVR system control shows promising potential for improving the efficiency and accuracy of power system control. This research provides insights into the advantages of DRL for process control and highlights its potential for future applications in power system control.

Hypergraph convolution mix DDPG for multi-aerial base station deployment

Aerial base stations (AeBS), as crucial components of air-ground integrated networks, can serve as the edge nodes to provide flexible services to ground users. Optimizing the deployment of multiple AeBSs to maximize system energy efficiency is currently a prominent and actively researched topic in the AeBS-assisted edge-cloud computing network. In this paper, we deploy AeBSs using multi-agent deep reinforcement learning (MADRL). We describe the multi-AeBS deployment challenge as a decentralized partially observable Markov decision process (Dec-POMDP), taking into consideration the constrained observation range of AeBSs. The hypergraph convolution mix deep deterministic policy gradient (HCMIX-DDPG) algorithm is designed to maximize the system energy efficiency. The proposed algorithm uses the value decomposition framework to solve the lazy agent problem, and hypergraph convolutional (HGCN) network is introduced to strengthen the cooperative relationship between agents. Simulation results show that the suggested HCMIX-DDPG algorithm outperforms alternative baseline algorithms in the multi-AeBS deployment scenario.

A path planning algorithm fusion of obstacle avoidance and memory functions

AbstractIn this study, to address the issues of sluggish convergence and poor learning efficiency at the initial stages of training, the authors improve and optimise the Deep Deterministic Policy Gradient (DDPG) algorithm. First, inspired by the Artificial Potential Field method, the selection strategy of DDPG has been improved to accelerate the convergence speed during the early stages of training and reduce the time it takes for the mobile robot to reach the target point. Then, optimising the neural network structure of the DDPG algorithm based on the Long Short‐Term Memory accelerates the algorithm's convergence speed in complex dynamic scenes. Static and dynamic scene simulation experiments of mobile robots are carried out in ROS. Test findings demonstrate that the Artificial Potential Field method‐Long Short Term Memory Deep Deterministic Policy Gradient (APF‐LSTM DDPG) algorithm converges significantly faster in complex dynamic scenes. The success rate is improved by 7.3% and 3.6% in contrast to the DDPG and LSTM‐DDPG algorithms. Finally, the usefulness of the method provided in this study is similarly demonstrated in real situations using real mobile robot platforms, laying the foundation for the path planning of mobile robots in complex changing conditions.

Off‐policy correction algorithm for double Q network based on deep reinforcement learning

AbstractA deep reinforcement learning (DRL) method based on the deep deterministic policy gradient (DDPG) algorithm is proposed to address the problems of a mismatch between the needed training samples and the actual training samples during the training of intelligence, the overestimation and underestimation of the existence of Q‐values, and the insufficient dynamism of the intelligence policy exploration. This method introduces the Actor‐Critic Off‐Policy Correction (AC‐Off‐POC) reinforcement learning framework and an improved double Q‐value learning method, which enables the value function network in the target task to provide a more accurate evaluation of the policy network and converge to the optimal policy more quickly and stably to obtain higher value returns. The method is applied to multiple MuJoCo tasks on the Open AI Gym simulation platform. The experimental results show that it is better than the DDPG algorithm based solely on the different policy correction framework (AC‐Off‐POC) and the conventional DRL algorithm. The value of returns and stability of the double‐Q‐network off‐policy correction algorithm for the deep deterministic policy gradient (DCAOP‐DDPG) proposed by the authors are significantly higher than those of other DRL algorithms.

A Motion Planning Method for Visual Servoing Using Deep Reinforcement Learning in Autonomous Robotic Assembly

Assembly positioning by visual servoing (VS) is a basis for autonomous robotic assembly. In practice, VS control suffers potential stability and convergence problems due to image and physical constraints, e.g., field of view constraints, image local minima, obstacle collisions, and occlusion. Therefore, this article proposes a novel deep reinforcement learning-based hybrid visual servoing (DRL-HVS) controller for motion planning of VS tasks. DRL-HVS controller takes current observed image features and camera pose as inputs, and the core parameters of hybrid VS are dynamically optimized using a deep deterministic policy gradient (DDPG) algorithm to obtain an optimal motion scheme, considering image/physical constraints and robot motion performance. In addition, an adaptive exploration strategy is proposed to further improve the training efficiency by adaptively tuning the exploration noise parameters. In this way, the offline pretrained DRL-HVS controller in the virtual environment, where the DDPG actor–critic network is continuously optimized, can be quickly deployed to a real robot system for real-time control. Experiments based on an eye-in-hand VS system are conducted with a calibrated HIKVISION RGB camera mounted on the end-effector of a GSK-RB03A1 six degree-of-freedom (6-DoF) robot. Basic VS task experiments show that the proposed controller achieves better performance than the existing methods: the servoing time is 24% smaller than that of the five-dimensional VS method, a 100% success rate with the perturbed ranges of the initial position within 25 mm for translation and 20° for rotation, and a 48% efficiency improvement. Moreover, a planetary gear component assembly process case study, where the robot aims to automatically put the gears on the gear shafts, is conducted to demonstrate the applicability of the proposed method in practice.

A deep reinforcement learning system for the allocation of epidemic prevention materials based on DDPG

COVID-19 has spread rapidly around the world since the end of 2019. Consequently, the demand for epidemic prevention materials (e.g., medical-grade masks) has increased drastically. The medical-grade masks have become a necessary item for everyone. Referring to the shortage of medical-grade masks in Taiwan, the government collected and managed them at the early stage and sold them for a fixed price. However, the government must distribute masks around the country to prevent the movement of people who can cause infection with COVID-19. Moreover, the demands for medical supplies during the pandemic have become more complex and dynamic. This study proposes a robust system for allocating epidemic prevention materials. The proposed approach adopts the reinforcement learning framework, which takes the daily supply and demand for masks as the environment, with the Deep Deterministic Policy Gradient (DDPG) algorithm for agent updates and the daily shortage as rewards and punishments. The proposed system is compared with the traditional machine learning approach used for supply chain demand forecasting through experiments. The results indicate that our proposed method is superior regarding the number of pharmacies with over-stocked masks and rewards. Moreover, our proposed system performs consistently under different total numbers of masks.

DRL-Based Service Function Chain Edge-to-Edge and Edge-to-Cloud Joint Offloading in Edge-Cloud Network

In this paper, we study service function chain (SFC) offloading in the edge-cloud network. Two offloading options are available for fully-loaded edge nodes: edge to edge (E2E) offloading and edge to cloud (E2C) offloading. Both E2E offloading and E2C offloading have been optimized in existing research, and Deep Reinforcement Learning (DRL) methods were adopted to achieve excellent performances. However, DRL-based SFC E2E and E2C joint offloading is still a research gap. In this paper, we propose a DRL-based SFC E2E and E2C joint offloading optimization algorithm for the edge-cloud network to maximize the utilization efficiency of edge resources. Twin Delayed Deep Deterministic policy gradient (TD3) algorithm is applied to the optimization problem. The simulation results indicate that the proposed algorithm has excellent convergence performance and improves the utilization efficiency of edge resources in edge-cloud network scenarios of diverse scales.

Misinformation Propagation in Online Social Networks: Game Theoretic and Reinforcement Learning Approaches

Misinformation in online social networks (OSNs) has been an ongoing problem, and it has been studied heavily over recent years. In this article, we use gamification to tackle misinformation propagation in OSNs. First, we construct a game based on the notion of cooperative games on graphs where the nodes of the social network are players. We use random regular networks and real networks in our simulations to show that the constructed game follows evolutionary dynamics and that the outcome of the game depends on the relation between the structural properties of the network and the benefit and cost variables defined in a cooperative game. Second, we create a game on the network level where the players control a set of nodes. We define agents whose goal is to maximize the total reward that we set up to be the number of nodes affected at the end of the game. We propose a deep reinforcement learning (RL) technique based on the multiagent deep deterministic policy gradient (MADDPG) algorithm. We test the proposed method along with well-known node selection algorithms and obtain promising results on different social networks.

AUV Collision Avoidance Planning Method Based on Deep Deterministic Policy Gradient

Collision avoidance planning has always been a hot and important issue in the field of unmanned aircraft research. In this article, we describe an online collision avoidance planning algorithm for autonomous underwater vehicle (AUV) autonomous navigation, which relies on its own active sonar sensor to detect obstacles. The improved particle swarm optimization (I-PSO) algorithm is used to complete the path planning of the AUV under the known environment, and we use it as a benchmark to improve the fitness function and inertia weight of the algorithm. Traditional path-planning algorithms rely on accurate environment maps, where re-adapting the generated path can be highly demanding in terms of computational cost. We propose a deep reinforcement learning (DRL) algorithm based on collision avoidance tasks. The algorithm discussed in this paper takes into account the relative position of the target point and the rate of heading change from the previous timestep. Its reward function considers the target point, running time and turning angle at the same time. Compared with the LSTM structure, the Gated Recurrent Unit (GRU) network has fewer parameters, which helps to save training time. A series of simulation results show that the proposed deep deterministic policy gradient (DDPG) algorithm can obtain excellent results in simple and complex environments.

Bearing Digital Twin Based on Response Model and Reinforcement Learning

In recent years, research on bearing fault modeling has witnessed significant advancements. However, the modeling of bearing faults using digital twins (DTs) remains an emerging area of exploration. This paper introduces a bearing digital twin developed by integrating a signal-based response model with reinforcement learning techniques. Initially, a signal-based model is constructed, comprising a unit fault impulse function and a decay oscillation function. This model illustrates the bearing’s acceleration response under fault conditions and acts as the environmental component within the bearing digital twin. Subsequently, a parameter estimation process identifies two critical parameters from the signal-based model: the load proportional factor and the decaying constant. The Deep Deterministic Policy Gradient (DDPG) algorithm is employed as the agent for online learning of these parameters. The cosine similarity metric is employed to define the state and reward by comparing the real acceleration measurements with the simulation data generated by the digital twin. To validate the effectiveness of the digital twin, experimental data sourced from the three datasets are utilized. The results underscore the digital twin’s capacity to faithfully replicate the bearing’s acceleration response under diverse conditions, demonstrating a high degree of similarity in both the time and frequency domains.