Deterministic Policy Gradient Research Articles

From Nash Q-learning to nash-MADDPG: Advancements in multiagent control for multiproduct flexible manufacturing systems

The emergence of flexible manufacturing systems (FMS) capable of processing multiple product types is a result of the growing demand for product customization and personalization. Such multiproduct systems are characterized by a higher level of uncertainty and variability when compared to traditional manufacturing systems. This paper proposes a Nash integrated multiagent deep deterministic policy gradient method (Nash-MADDPG) to control the mobile robots’ assignment in a multiproduct FMS to enable intelligent decision-making, interaction, and dynamic learning capabilities. A mathematical model of a multiproduct FMS from a previous study is described, and system dynamic property is characterized by permanent production loss (PPL). Then, by observing PPL of the system and market demand for each product type, the multi-agent control scheme is developed to assign mobile robots to load/unload various product types at various machines. First, a Nash game is developed among the mobile robots and to improve the cooperation, a collaboration cost is defined. This collaboration cost is then used in the reward function of the multiagent deep deterministic policy gradient (MADDPG) algorithm. Second, the actions are jointly defined based on the action values of MADDPG, and the mobile robots’ strategies in the Nash game, which update the environment to a new state. The performance of the proposed method is verified by comparing it with conventional Nash Q-learning, vanilla MADDPG, Q-learning based single agent reinforcement learning (SARL) and a first-come-first-serve (FCFS) based control. The results demonstrate that the multi-agent control scheme under the proposed Nash-MADDPG is effective in dealing with cooperative FMS environment that involves complicated dynamics and uncertainties.

Evolving testing scenario generation and intelligence evaluation for automated vehicles

Interaction between the background vehicles (BVs) and automated vehicles (AVs) in scenario-based testing plays a critical role in evaluating the intelligence of the AVs. Current testing scenarios typically employ predefined or scripted BVs, which inadequately reflect the complexity of human-like social behaviors in real-world driving scenarios, and also lack a systematic metric for evaluating the comprehensive intelligence of AVs. Therefore, this paper proposes an evolving scenario generation method, employing deep reinforcement learning (DRL) to construct human-like BVs that interact with AVs, and this evolving scenario is designed to test and evaluate the intelligence of AVs. Firstly, a class of BV driver models with human-like competitive, mutual, and cooperative driving motivations is designed. Then, utilizing the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and an improved level-k training procedure, the three distinct driver models acquire game-based interactive driving policies. And these driver models are combined to generate evolving scenarios in which they can interact continuously and evolve diverse contents. Next, a framework including safety, driving efficiency, and interaction utility are presented to evaluate and quantify the intelligence performance of 3 systems under test (SUTs), indicating the effectiveness of the evolving scenario for intelligence testing. Finally, the complexity and fidelity of the proposed evolving testing scenario are validated. The results demonstrate that the proposed evolving scenario exhibits the highest level of complexity compared to other baseline scenarios and has more than 85% similarity to naturalistic driving data. This highlights the potential of the proposed method to facilitate the development and evaluation of high-level AVs in a realistic and challenging environment.

Learning Framework for Joint Optimal Node Placement and Resource Management in Dynamic Fog Environment

Background: With recent improvements in fog computing, it is now feasible to offer faster response time and better service delivery quality; however, the impending challenge is to place the fog nodes within the environment optimally. A review of existing literature showcases that consideration of joint problems such as fog node placement and resource management are less reported. Irrespective of different available methodologies, it is noted that a learning scheme facilitates better capability to incorporate intelligence in the network device, which can act as an enabling technique for superior operation of fog nodes. Objective: The prime objective of the study is to introduce simplified and novel computational modelling toward the optimal placement of fog nodes with improved resource allocation mechanisms concerning bandwidth Method: Implemented in Python, the proposed scheme performs predictive operations using the Deep Deterministic Policy Gradient (DDPG) method. Markov modelling is used to frame the model. OpenAI Gym library is used for environment modelling, bridging communication between the environment and the learning agent. Result: Quantitative results indicate that the proposed scheme performs better than existing schemes by approximately 30%. Conclusion: The prime innovative approach introduced is the implementation of a reinforcement learning algorithm with a Markov chain towards enriching the predictive analytical capabilities of the controller system with faster service relaying operations a.

D2PG: deep deterministic policy gradient based for maximizing network throughput in clustered EH-WSN

D2PG: deep deterministic policy gradient based for maximizing network throughput in clustered EH-WSN

Research on a Personalized Decision Control Algorithm for Autonomous Vehicles Based on the Reinforcement Learning from Human Feedback Strategy

To address the shortcomings of previous autonomous decision models, which often overlook the personalized features of users, this paper proposes a personalized decision control algorithm for autonomous vehicles based on RLHF (reinforcement learning from human feedback). The algorithm combines two reinforcement learning approaches, DDPG (Deep Deterministic Policy Gradient) and PPO (proximal policy optimization), and divides the control scheme into three phases including pre-training, human evaluation, and parameter optimization. During the pre-training phase, an agent is trained using the DDPG algorithm. In the human evaluation phase, different trajectories generated by the DDPG-trained agent are scored by individuals with different styles, and the respective reward models are trained based on the trajectories. In the parameter optimization phase, the network parameters are updated using the PPO algorithm and the reward values given by the reward model to achieve personalized autonomous vehicle control. To validate the control algorithm designed in this paper, a simulation scenario was built using CARLA_0.9.13 software. The results demonstrate that the proposed algorithm can provide personalized decision control solutions for different styles of people, satisfying human needs while ensuring safety.

MADDPG-Based Offloading Strategy for Timing-Dependent Tasks in Edge Computing

With the increasing popularity of the Internet of Things (IoT), the proliferation of computation-intensive and timing-dependent applications has brought serious load pressure on terrestrial networks. In order to solve the problem of computing resource conflict and long response delay caused by concurrent application service applications from multiple users, this paper proposes an improved edge computing timing-dependent, task-offloading scheme based on Multi-Agent Deep Deterministic Policy Gradient (MADDPG) that aims to shorten the offloading delay and improve the resource utilization rate by means of resource prediction and collaboration among multiple agents to shorten the offloading delay and improve the resource utilization. First, to coordinate the global computing resource, the gated recurrent unit is utilized, which predicts the next computing resource requirements of the timing-dependent tasks according to historical information. Second, the predicted information, the historical offloading decisions and the current state are used as inputs, and the training process of the reinforcement learning algorithm is improved to propose a task-offloading algorithm based on MADDPG. The simulation results show that the algorithm reduces the response latency by 6.7% and improves the resource utilization by 30.6% compared with the suboptimal benchmark algorithm, and it reduces nearly 500 training rounds during the learning process, which effectively improves the timeliness of the offloading strategy.

Self‐attention convolutional neural network optimized with Remora Optimization Algorithm for energy aware resource management in Non‐orthogonal Multiple Access networks

SummaryNon‐orthogonal multiple access (NOMA) is an efficient technology for future wireless communication systems. The resource management on NOMA networks is flattering for improving energy efficiency (EE) of systems. Therefore, self‐attention convolutional neural network (SACNN) optimized with Remora Optimization Algorithm (ROA) for energy aware resource management in NOMA network (SACNN‐ROA‐EE) is proposed in this paper for solving user allocation, channel allocation, and power allocation problems. The Matching‐Coalition approach is considered to face the user allocation issue, and the SACNN is considered to face the channel allocation and power allocation problems. SACNN does not adopt any optimization methods to determine the optimal parameters in channel allocation and power allocation. Therefore, the ROA is applied to optimize the SACNN weight parameters. The proposed technique is activated in MATLAB, and its efficacy is analyzed with under performances metrics, such as minimum user rate, sum rate, energy, and spectral performance. The proposed SACNN‐ROA‐EE attains 24.35%, 27.84%, and 38.58% better sum rate and 27.45%, 43.28%, 30.56% better EE compared with existing methods, such as EE utilizing communication deep neural network optimized with power allocation optimization (PAO‐CDNN‐EE), EE under deep transfer deterministic policy gradient along deep deterministic policy gradient approach (DPPGA‐DTDPPG‐EE), and EE utilizing deep reinforcement learning with baseline algorithms (BA‐DRL‐EE), respectively.

IRS Integrated UAV Communication to Enhance Physical Layer Security: A Deep Reinforcement Learning approach

The paper investigates the security of an intelligent reflecting surface (IRS) assisted unmanned aerial vehicle (UAV) network, where a base station (BS) transmits confidential information to the ground user (GU) via IRS-assisted UAV. The study explores using UAVs with IRS to enhance the security and reliability of wireless communication systems, particularly in the presence of eavesdroppers and friendly jammers. The beamforming at IRS-assisted UAV and UAV trajectory are jointly formulated as a non-convex optimization problem, which is solved by the deep reinforcement learning (DRL) algorithm to maximize the sum secrecy rate of GU with the aid of jammer. We proposed a dual-DDPG (D3PG) algorithm that utilizes the deep deterministic policy gradient (DDPG) structure to effectively address these dual non-convex problems of the UAV-trajectory optimization and the UAV-IRS beamforming optimization. The proposed algorithm's effectiveness and robustness are demonstrated through simulation results, with the IRS significantly enhancing the sum secrecy rate. Extensive simulations show that the proposed DRL-based D3PG scheme outperforms the traditional optimization schemes and ordinary DQN schemes.

Deep Reinforcement Learning for Ecological and Distributed Urban Traffic Signal Control with Multi-Agent Equilibrium Decision Making

Multi-Agent Reinforcement Learning (MARL) has shown strong advantages in urban multi-intersection traffic signal control, but it also suffers from the problems of non-smooth environment and inter-agent coordination. However, most of the existing research on MARL traffic signal control has focused on designing efficient communication to solve the environment non-smoothness problem, while neglecting the coordination between agents. In order to coordinate among agents, this paper combines MARL and the regional mixed-strategy Nash equilibrium to construct a Deep Convolutional Nash Policy Gradient Traffic Signal Control (DCNPG-TSC) model, which enables agents to perceive the traffic environment in a wider range and achieves effective agent communication and collaboration. Additionally, a Multi-Agent Distributional Nash Policy Gradient (MADNPG) algorithm is proposed in this model, which is the first time the mixed-strategy Nash equilibrium is used for the improvement in the Multi-Agent Deep Deterministic Policy Gradient algorithm traffic signal control strategy to provide the optimal signal phase for each intersection. In addition, the eco-mobility concept is integrated into MARL traffic signal control to reduce pollutant emissions at intersections. Finally, simulation results in synthetic and real-world traffic road networks show that DCNPG-TSC outperforms other state-of-the-art MARL traffic signal control methods in almost all performance metrics, because it can aggregate the information of neighboring agents and optimize the agent’s decisions through gaming to find an optimal joint equilibrium strategy for the traffic road network.

Optimal operation of reverse osmosis desalination process with deep reinforcement learning methods

AbstractThe reverse osmosis (RO) process is a well-established desalination technology, wherein energy-efficient techniques and advanced process control methods significantly reduce production costs. This study proposes an optimal real-time management method to minimize the total daily operation cost of an RO desalination plant, integrating a storage tank system to meet varying daily freshwater demand. Utilizing the dynamic model of the RO process, a cascade structure with two reinforcement learning (RL) agents, namely the deep deterministic policy gradient (DDPG) and deep Q-Network (DQN), is developed to optimize the operation of the RO plant. The DDPG agent, manipulating the high-pressure pump, controls the permeate flow rate to track a reference setpoint value. Simultaneously, the DQN agent selects the optimal setpoint value and communicates it to the DDPG controller to minimize the plant’s operation cost. Monitoring storage tanks, permeate flow rates, and water demand enables the DQN agent to determine the required amount of permeate water, optimizing water quality and energy consumption. Additionally, the DQN agent monitors the storage tank’s water level to prevent overflow or underflow of permeate water. Simulation results demonstrate the effectiveness and practicality of the designed RL agents.

Deep Reinforcement Learning-Driven UAV Data Collection Path Planning: A Study on Minimizing AoI

As a highly efficient and flexible data collection device, Unmanned Aerial Vehicles (UAVs) have gained widespread application because of the continuous proliferation of Internet of Things (IoT). Addressing the high demands for timeliness in practical communication scenarios, this paper investigates multi-UAV collaborative path planning, focusing on the minimization of weighted average Age of Information (AoI) for IoT devices. To address this challenge, the multi-agent twin delayed deep deterministic policy gradient with dual experience pools and particle swarm optimization (DP-MATD3) algorithm is presented. The objective is to train multiple UAVs to autonomously search for optimal paths, minimizing the AoI. Firstly, considering the relatively slow learning speed and susceptibility to local minima of neural network algorithms, an improved particle swarm optimization (PSO) algorithm is utilized for parameter optimization of the multi-agent twin delayed deep deterministic policy gradient (MATD3) neural network. Secondly, with the introduction of the dual experience pools mechanism, the efficiency of network training is significantly improved. Experimental results show DP-MATD3 outperforms MATD3 in average weighted AoI. The weighted average AoI is reduced by 33.3% and 27.5% for UAV flight speeds of v = 5 m/s and v = 10 m/s, respectively.

Application of optimization algorithm for virtual reference tag assisted localization and tracking of RFID

SummaryIn an indoor setting, the radio frequency communication is utilized to locate and track mobile objects utilizing Radio Frequency Identification (RFID) technology. Measurements from the Received Signal Strength Indicator (RSSI) are typically the foundation of the localization technique. In an RFID‐based interior setting, lowering tracking mistakes and increasing the precision of tracking remain difficult tasks. In order to address these issues, we developed the VIRALTRACK (Virtual Reference Tag Localization and Tracking) framework that consists of four procedures: deep reinforced learning‐based tracking, quantum‐based localization, optimization‐based virtual reference tag allocation, and signal enhancement. In order to increase the signal's effectiveness, we initially suggested using the Extended Gradient Filter (EGF) technique to eliminate RSSI oscillations. In the second step, we suggested using the Emperor Penguin Colony (EPC) optimization technique to allocate the virtual reference tag while taking the number of tags, SNR, and temperature and humidity of the surroundings into account. In the third phase, we use a quantum neural network (QNN) for localization in order to estimate the position of the moving target. We introduced the SignRank approach to select the best virtual reference tag for localization, which lowers tracking mistakes. In conclusion, we presented the Twin Delayed Deep Deterministic Policy Gradient (TD3) method that boosts the tracking precision by tracking the moving target tag efficiently and taking into account stage, the orientation, distance, and valuable coordinates. The NS3.26 network simulator is used to run the simulation, and tracking precision, tracking error, and accumulated probabilities are used to assess effectiveness.

Joint Phase Shift Design and Resource Management for a Non-Orthogonal Multiple Access-Enhanced Internet of Vehicle Assisted by an Intelligent Reflecting Surface-Equipped Unmanned Aerial Vehicle

This paper integrates intelligent reflecting surfaces (IRS) with unmanned aerial vehicles (UAV) to enhance the transmission performance of the Internet of Vehicles (IoV) through non-orthogonal multiple access (NOMA). It focuses on strengthening the signals from cell edge vehicles (CEVs) to the base station by optimizing the wireless propagation environment via an IRS-equipped UAV. The primary goal is to maximize the sum data rate of CEVs while satisfying the constraint of the successive interference cancellation (SIC) decoding threshold. The challenge lies in the non-convex nature of jointly considering the power control, subcarrier allocation, and phase shift design, making the problem difficult to optimally solve. To address this, the problem is decomposed into two independent subproblems, which are then solved iteratively. Specifically, the optimal phase shift design is achieved using the deep deterministic policy gradient (DDPG) algorithm. Furthermore, the graph theory is applied to determine the subcarrier allocation policy and derive a closed-form solution for optimal power control. Finally, the simulation results show that the proposed joint phase shift and resource management scheme significantly enhances the sum data rate compared to the state-of-the-art schemes, thereby demonstrating the benefits of integrating the IRS-equipped UAV into NOMA-enhanced IoV.

Hybrid Centralized Training and Decentralized Execution Reinforcement Learning in Multi-Agent Path-Finding Simulations

In this paper, we propose a hybrid centralized training and decentralized execution neural network architecture with deep reinforcement learning (DRL) to complete the multi-agent path-finding simulation. In the training of physical robots, collisions and other unintended accidents are very likely to occur in multi-agent cases, so it is required to train the networks within a deep deterministic policy gradient for the virtual environment of the simulator. The simple particle multi-agent simulator designed by OpenAI (Sacramento, CA, USA) for training platforms can easily obtain the state information of the environment. The overall system of the training cycle is designed with a self-designed reward function and is completed through a progressive learning approach from a simple to a complex environment. Finally, we carried out and presented the experiments of multi-agent path-finding simulations. The proposed methodology is better than the multi-agent model-based policy optimization (MAMBPO) and model-free multi-agent soft actor–critic models.

Deep reinforcement learning-based reactive trajectory planning method for UAVs

In order to improve the ability of avoiding dynamic threats during the flight of unmanned aerial vehicles (UAVs), a deep reinforcement learning-based reactive trajectory planning method is proposed in this paper. Firstly, a constrained Rapidly-exploring Random Tree-Connect algorithm (C-RRT-Connect) is proposed as the basic algorithm of reactive trajectory planning to globally plan for avoiding static obstacles in the environment. The C-RRT-Connect algorithm introduces the idea of target attraction to constrain the optimal growth point in the RRT-Connect algorithm. Then, based on the global trajectory, the local optimization is carried out according to the dynamic threats detected by the UAV during the flight. According to the real-time relative state between the UAV and the detected dynamic threat, the reaction sampling points and directional coefficients for avoiding the corresponding dynamic threat are generated online via the action network trained with the depth deterministic policy gradient algorithm (DDPG). And then the local trajectory is adjusted to modify the flight trajectory of the UAV to achieve reactive obstacle avoidance. The simulation experiment firstly compares the global trajectory planning performance of C-RRT-Connect and RRT-Connect in static environment, and secondly compares the local trajectory planning performance of DDPG algorithm and the artificial potential field method in dynamic environment. The experimental results show that in static environment, C-RRT-Connect algorithm has faster searching speed, less invalid samples and higher searching trajectory quality than RRT-Connect algorithm; In a dynamic environment, DDPG algorithm reduces the average running time by about 26% compared with the artificial potential field method, and has a stronger ability to evade dynamic threats in real time.

Distributed auto disturbances rejection resilient control of permanent magnetic maglev trains based on the optimized deep deterministic policy gradient algorithm

Distributed auto disturbances rejection resilient control of permanent magnetic maglev trains based on the optimized deep deterministic policy gradient algorithm

Impact of memory on beamforming optimization for DDPG-assisted RIS-multi-user system

Recently, reconfigurable intelligent surfaces (RIS) have garnered considerable attention as indispensable components driving the evolution of future 6G wireless communication systems. This heightened interest is primarily attributed to notable advancements in programmable meta-material fabrication, which enable the creation of highly versatile and adaptable surfaces. These surfaces, often referred to as intelligent reflecting arrays, represent a significant departure from the conventional capabilities of massive multiple-input multiple-output (MIMO) systems, thereby catalyzing the emergence of intelligent radio environments characterized by enhanced flexibility and efficiency. Our study is dedicated to the comprehensive exploration of coordinated design strategies that encompass both the transmission beamforming matrix at the base station and the phase shift matrix at the RIS. Leveraging recent breakthroughs in deep reinforcement learning (DRL), our approach harnesses the power of a Long Short-Term Memory (LSTM) based DRL algorithm. This algorithm orchestrates the joint design process through iterative interactions with the environment, strategically guided by predefined rewards operating within a continuous framework of states and actions. While recent research has underscored the efficacy of LSTM-based architectures in bolstering the learning capacity of reinforcement learning (RL) algorithms and simplifying the search process, our investigation reveals a nuanced insight. Contrary to prevailing trends, we find that the integration of memory into the Deep Deterministic Policy Gradient (DDPG) with LSTM (DDPG-LSTM) algorithm yields unexpected consequences, negatively impacting the system's overall performance. Through preliminary simulation results, we empirically demonstrate the adverse effect of memory on DDPG performance when applied to RIS systems. This finding not only sheds light on the complexities of optimizing 6G wireless communication systems but also underscores the importance of careful algorithmic design and parameter tuning in achieving desired outcomes in emerging technologies.

A Pilot Study of Observation Poisoning on Selective Reincarnation in Multi-Agent Reinforcement Learning

This research explores the vulnerability of selective reincarnation, a concept in Multi-Agent Reinforcement Learning (MARL), in response to observation poisoning attacks. Observation poisoning is an adversarial strategy that subtly manipulates an agent’s observation space, potentially leading to a misdirection in its learning process. The primary aim of this paper is to systematically evaluate the robustness of selective reincarnation in MARL systems against the subtle yet potentially debilitating effects of observation poisoning attacks. Through assessing how manipulated observation data influences MARL agents, we seek to highlight potential vulnerabilities and inform the development of more resilient MARL systems. Our experimental testbed was the widely used HalfCheetah environment, utilizing the Independent Deep Deterministic Policy Gradient algorithm within a cooperative MARL setting. We introduced a series of triggers, namely Gaussian noise addition, observation reversal, random shuffling, and scaling, into the teacher dataset of the MARL system provided to the reincarnating agents of HalfCheetah. Here, the “teacher dataset” refers to the stored experiences from previous training sessions used to accelerate the learning of reincarnating agents in MARL. This approach enabled the observation of these triggers’ significant impact on reincarnation decisions. Specifically, the reversal technique showed the most pronounced negative effect for maximum returns, with an average decrease of 38.08% in Kendall’s tau values across all the agent combinations. With random shuffling, Kendall’s tau values decreased by 17.66%. On the other hand, noise addition and scaling aligned with the original ranking by only 21.42% and 32.66%, respectively. The results, quantified by Kendall’s tau metric, indicate the fragility of the selective reincarnation process under adversarial observation poisoning. Our findings also reveal that vulnerability to observation poisoning varies significantly among different agent combinations, with some exhibiting markedly higher susceptibility than others. This investigation elucidates our understanding of selective reincarnation’s robustness against observation poisoning attacks, which is crucial for developing more secure MARL systems and also for making informed decisions about agent reincarnation.

Local 2-D Path Planning of Unmanned Underwater Vehicles in Continuous Action Space Based on the Twin-Delayed Deep Deterministic Policy Gradient

Local 2-D Path Planning of Unmanned Underwater Vehicles in Continuous Action Space Based on the Twin-Delayed Deep Deterministic Policy Gradient

Pre-optimization-assisted deep reinforcement learning-based energy management strategy for a series-parallel hybrid electric truck

P2-P3 series-parallel hybrid electric vehicles (HEVs) feature intricate topologies with multiple power sources and multiple working modes, posing a challenge for developing effective energy management strategies (EMSs). This paper introduces a framework that combines deep reinforcement learning (DRL) with pre-optimized energy management to address this challenge. Considering the characteristics of HEVs, the framework incorporates the pre-optimized equivalent motor and pre-optimized mode selection rule into a deep deterministic policy gradient (DDPG) loop. The pre-optimization of the equivalent motor aims to reduce the energy management task’s dimensionality by equating two motors as the equivalent motor, while the pre-optimized mode selection rule systematically integrates mode selection and energy allocation into the EMS. The study assumes HEVs operate in connected urban environments and incorporates middle-horizon traffic information to enhance EMS performance. Additionally, the study integrates the frequent engine start-stop characteristics of HEVs into EMS design. Simulation results demonstrate that the fuel economy of the proposed EMSs in this study ranges from 87.2% to 90.7% of the DP-based EMS. The inclusion of an engine start-stop penalty significantly reduces the number of engine start-stops without compromising fuel economy.