Replay Mechanism Research Articles

Improved DDPG algorithm-based path planning for unmanned surface vehicles

As a promising mode of water transportation, unmanned surface vehicles (USVs) are used in various fields owing to their small size, high flexibility, favorable price, and other advantages. Traditional navigation algorithms are affected by various path planning issues. To address the limitations of the traditional deep deterministic policy gradient (DDPG) algorithm, namely slow convergence speed and sparse reward and punishment functions, we proposed an improved DDPG algorithm for USV path planning. First, the principle and workflow of the DDPG deep reinforcement learning (DRL) algorithm are described. Second, the improved method (based on the USVs kinematic model) is proposed, and a continuous state and action space is designed. The reward and punishment function are improved, and the principle of collision avoidance at sea is introduced. Dynamic region restriction is added, distant obstacles in the state space are ignored, and the nearby obstacles are observed to reduce the number of algorithm iterations and save computational resources. The introduction of a multi-intelligence approach combined with a prioritized experience replay mechanism accelerates algorithm convergence, thereby increasing the efficiency and robustness of training. Finally, through a combination of theory and simulation, the DDPG DRL is explored for USV obstacle avoidance and optimal path planning.

Path Tracking Control for Four-Wheel Independent Steering and Driving Vehicles Based on Improved Deep Reinforcement Learning

We propose a compound control framework to improve the path tracking accuracy of a four-wheel independent steering and driving (4WISD) vehicle in complex environments. The framework consists of a deep reinforcement learning (DRL)-based auxiliary controller and a dual-layer controller. Samples in the 4WISD vehicle control framework have the issues of skewness and sparsity, which makes it difficult for the DRL to converge. We propose a group intelligent experience replay (GER) mechanism that non-dominantly sorts the samples in the experience buffer, which facilitates within-group and between-group collaboration to achieve a balance between exploration and exploitation. To address the generalization problem in the complex nonlinear dynamics of 4WISD vehicles, we propose an actor-critic architecture based on the method of two-stream information bottleneck (TIB). The TIB method is used to remove redundant information and extract high-dimensional features from the samples, thereby reducing generalization errors. To alleviate the overfitting of DRL to known data caused by IB, the reverse information bottleneck (RIB) alters the optimization objective of IB, preserving the discriminative features that are highly correlated with actions and improving the generalization ability of DRL. The proposed method significantly improves the convergence and generalization capabilities of DRL, while effectively enhancing the path tracking accuracy of 4WISD vehicles in high-speed, large-curvature, and complex environments.

GWO-ANFIS-RD3PG: A reinforcement learning approach with dynamic adjustment and dual replay mechanism for building energy forecasting

Reliable prediction of building energy consumption is essential for effective and sustainable energy management. Traditional forecasting methods, however, face challenges when dealing with sudden changes in energy consumption. To address the above-mentioned issue, we introduce the GWO-ANFIS-RD3PG prediction model in this paper. This model aims to ensure global forecasting accuracy while minimizing prediction errors at sudden change points. A Grey Wolf Optimization (GWO) algorithm with the Adaptive Neuro-Fuzzy Inference System (ANFIS) is proposed to predict the energy consumption type and its fluctuation magnitude for the next time step. Within the Recurrent Dual Experience Replay Buffer DDPG (RD3PG) prediction module, we employ Long Short-Term Memory (LSTM), as the actor network in the Deep Deterministic Policy Gradient (DDPG), to account for long-term dependencies in time series data. Notably, we introduce an adaptive action modification mechanism that allows the agent to optimize actions based on the output of the aforementioned ANFIS, enabling real-time adjustments during sudden change. To further enhance the model’s performance, we adopt a dual experience replay mechanism to store data samples from sudden change points, capturing insight information during these anomalies. Experimental results on two real-world datasets demonstrate that this method reduces MAE by 5.81% and 13.47%, and RMSE by 5.69% and 15.21% compared to the state-of-art models, showing the significance of the proposed method for energy efficiency improvement in real-world building operations.

Lifelong Generative Adversarial Autoencoder.

Lifelong learning describes an ability that enables humans to continually acquire and learn new information without forgetting. This capability, common to humans and animals, has lately been identified as an essential function for an artificial intelligence system aiming to learn continuously from a stream of data during a certain period of time. However, modern neural networks suffer from degenerated performance when learning multiple domains sequentially and fail to recognize past learned tasks after being retrained. This corresponds to catastrophic forgetting and is ultimately induced by replacing the parameters associated with previously learned tasks with new values. One approach in lifelong learning is the generative replay mechanism (GRM) that trains a powerful generator as the generative replay network, implemented by a variational autoencoder (VAE) or a generative adversarial network (GAN). In this article, we study the forgetting behavior of GRM-based learning systems by developing a new theoretical framework in which the forgetting process is expressed as an increase in the model's risk during the training. Although many recent attempts have provided high-quality generative replay samples by using GANs, they are limited to mainly downstream tasks due to the lack of inference. Inspired by the theoretical analysis while aiming to address the drawbacks of existing approaches, we propose the lifelong generative adversarial autoencoder (LGAA). LGAA consists of a generative replay network and three inference models, each addressing the inference of a different type of latent variable. The experimental results show that LGAA learns novel visual concepts without forgetting and can be applied to a wide range of downstream tasks.

Sim-to-real transfer of adaptive control parameters for AUV stabilisation under current disturbance

Learning-based adaptive control methods hold the potential to empower autonomous agents in mitigating the impact of process variations with minimal human intervention. However, their application to autonomous underwater vehicles (AUVs) has been constrained by two main challenges: (1) the presence of unknown dynamics in the form of sea current disturbances, which cannot be modelled or measured due to limited sensor capability, particularly on smaller low-cost AUVs, and (2) the nonlinearity of AUV tasks, where the controller response at certain operating points must be excessively conservative to meet specifications at other points. Deep Reinforcement Learning (DRL) offers a solution to these challenges by training versatile neural network policies. Nevertheless, the application of DRL algorithms to AUVs has been predominantly limited to simulated environments due to their inherent high sample complexity and the distribution shift problem. This paper introduces a novel approach by combining the Maximum Entropy Deep Reinforcement Learning framework with a classic model-based control architecture to formulate an adaptive controller. In this framework, we propose a Sim-to-Real transfer strategy, incorporating a bio-inspired experience replay mechanism, an enhanced domain randomisation technique, and an evaluation protocol executed on a physical platform. Our experimental assessments demonstrate the effectiveness of this method in learning proficient policies from suboptimal simulated models of the AUV. When transferred to a real-world vehicle, the approach exhibits a control performance three times higher compared to its model-based nonadaptive but optimal counterpart.

Research on priority scheduling strategy for smoothing power fluctuations of microgrid tie‐lines based on PER‐DDPG algorithm

AbstractThe variability of renewable energy within microgrids (MGs) necessitates the smoothing of power fluctuations through the effective scheduling of internal power equipment. Otherwise, significant power variations on the tie‐line connecting the MG to the main power grid could occur. This study introduces an innovative scheduling strategy that utilizes a data‐driven approach, employing a deep reinforcement learning algorithm to achieve this smoothing effect. The strategy prioritizes the scheduling of MG's internal power devices, taking into account the stochastic charging patterns of electric vehicles. The scheduling optimization model is initially described as a Markov decision process with the goal of minimizing power fluctuations on the interconnection lines and operational costs of the MG. Subsequently, after preprocessing the historical operational data of the MG, an enhanced scheduling strategy is developed through a neural network learning process. Finally, the results from four scheduling scenarios demonstrate the significant impact of the proposed strategy. Comparisons of reward curves before and after data preprocessing underscore its importance. In contrast to optimization results from deep deterministic policy gradient, soft actor‐critic, and particle swarm optimization algorithms, the superiority of the deep deterministic policy gradient algorithm with the addition of a priority experience replay mechanism is highlighted.

Adaptive decision-making with deep Q-network for heterogeneous unmanned aerial vehicle swarms in dynamic environments

With the continuous advancement of Unmanned Aerial Vehicle (UAV) swarm technology, the field of adaptive decision-making for heterogeneous UAV swarms across a spectrum of tasks is rapidly emerging as a focal point of research. This domain holds significant importance for enhancing the operational efficiency and responsiveness of UAV swarms in complex and dynamic environments. This paper addresses this challenge by introducing an innovative adaptive task decision-making method, which leverages Deep Q-Network (DQN) algorithms, specifically tailored to meet the unique requirements of heterogeneous UAV swarms. We initially construct a mathematical model that transforms the intricate swarm decision-making issues into a standard Markov Decision Process (MDP), and meticulously design the state space, action space, reward function, and state transition function to cater to the decision-making needs of UAV swarms. Utilizing this model, we further integrate a hybrid experience replay mechanism, coupled with DQN algorithms for offline training, enabling the efficient transformation of training outcomes into a network model that is directly applicable to real-time online decision-making. This approach markedly enhances the decision-making adaptability and dynamic adaptability of UAV swarms when confronted with a variety of tasks. During the offline training phase, we also employ a hybrid experience replay mechanism and target network approximation techniques to bolster the network's stability and training efficiency, ensuring high reliability in online decision-making. Comparative simulations with existing DQN algorithms reveal the distinct superiority of our method in addressing adaptive task decision-making challenges. Moreover, through 1000 comprehensive simulation experiments, we further validate the stability and effectiveness of our method across diverse task environments, providing a solid theoretical foundation and technical support for the practical application and proliferation of UAV swarm technology.

Optimizing passengers’ experience: A goal-oriented reinforcement learning speed control approach for urban railway trains

Prolonged vibration can be uncomfortable for passengers utilizing urban rail transit systems. This study proposes an automatic speed control framework for urban railway trains to reduce vertical vibrations experienced by passengers. We suggest the concept of the “segmented comfort speed limit” to represent the vertical passing comfort of oncoming sections. This speed limit is calculated from 1/3 octave band acceleration and smoothed through lag-type speed control mode. The deep deterministic policy gradient algorithm with hindsight experience replay mechanism (HER-DDPG) is designed, to balance safety, comfort, and energy efficiency driving. Verify the speed control framework based on HER-DDPG through the rail data collected from Dalian Metro Line 12. Compared with the DDPG-based model, the vertical comfort is improved by 2.34%, and the longitudinal acceleration and total energy consumption are reduced by 45% and 8.1%. Compared with the real-world train control trajectory, HER-DDPG improves vertical comfort by 9.76% and reduces energy consumption by 12.4%. The results show that the proposed framework can effectively improve the ride experience of passengers.

Mars Exploration: Research on Goal-Driven Hierarchical DQN Autonomous Scene Exploration Algorithm

In the non-deterministic, large-scale navigation environment under the Mars exploration mission, there is a large space for action and many environmental states. Traditional reinforcement learning algorithms that can only obtain rewards at target points and obstacles will encounter the problems of reward sparsity and dimension explosion, making the training speed too slow or even impossible. This work proposes a deep layered learning algorithm based on the goal-driven layered deep Q-network (GDH-DQN), which is more suitable for mobile robots to explore, navigate, and avoid obstacles without a map. The algorithm model is designed in two layers. The lower layer provides behavioral strategies to achieve short-term goals, and the upper layer provides selection strategies for multiple short-term goals. Use known position nodes as short-term goals to guide the mobile robot forward and achieve long-term obstacle avoidance goals. Hierarchical execution not only simplifies tasks but also effectively solves the problems of reward sparsity and dimensionality explosion. In addition, each layer of the algorithm integrates a Hindsight Experience Replay mechanism to improve performance, make full use of the goal-driven function of the node, and effectively avoid the possibility of misleading the agent by complex processes and reward function design blind spots. The agent adjusts the number of model layers according to the number of short-term goals, further improving the efficiency and adaptability of the algorithm. Experimental results show that, compared with the hierarchical DQN method, the navigation success rate of the GDH-DQN algorithm is significantly improved, and it is more suitable for unknown scenarios such as Mars exploration.

Integrated energy cluster hierarchical regulation technology considering demand response

Integrated Energy Cluster (IEC), the regional aggregation of integrated energy systems (IES), has accumulated plenty of dispatchable resources with the development of energy market. This, while significantly providing system Demand Response (DR) potential, also complicates the interaction of the IEC with the main grid and increases the difficulty of system scheduling. To address this issue, this paper proposes a Reinforcement Learning-driven multi-agent hierarchical regulation framework that makes full use of DR to maximize the benefits of both IEC and main grid. Firstly, in the context of the DR market, a mechanism for IECs to bid in the real-time DR market is proposed. Furthermore, an "IEC-main network" hierarchical regulation model taking account of DR is established to minimize the IEC operation cost and maximize the societal benefit. Moreover, an optimization algorithm utilizing Deep Deterministic Policy Gradient (DDPG) with Multi-process (MP) and Priority Experience Replay (PER) mechanism is proposed to allow adaptability to high-latitude and large-scale applications. In case study, the proposed model and algorithm is tested on an 8-node system and a 24-node system. The result indicates that the hierarchical regulation model considering DR can improve the system economy by 2.59 % than that without DR and the improved DDPG algorithm can enhance training effectiveness in comparison with DDPG and PER-DDPG.

A multi-level action coupling reinforcement learning approach for online two-stage flexible assembly flow shop scheduling

Multi-product centralized delivery and kitting assembly present significant challenges to hierarchical co-processing in multi-stage manufacturing systems. The combinations of priority dispatching rules at each level are transiently adaptive, and the performance in online scheduling deteriorates rapidly with changing environment. This paper investigates the selection of rule combinations for sustained high-performance responsive scheduling in two-stage flexible assembly flow shop scheduling problem with asynchronous execution and complex decision correlation. A Multi-Level Action Coupling Deep Q-Network (MALC-DQN) approach is proposed for adaptive integrated scheduling in hybrid processing and assembly shops. Firstly, the problem is skillfully established as an event-triggered integrated decision markov decision process. The prioritized batch experience replay mechanism is employed to retain the complete correlation information of key decision sequences. Then, coupling and sequence feature extraction modules are developed to enhance the agent’s ability to perceive execution process and the environment. Furthermore, the multi-level wait-limit mechanism and efficient action filtering mechanism are designed to mitigate ineffective waiting waste and action space explosion during learning. Finally, a series of sophisticated experiments are conducted to validate the effectiveness of the proposed methodology. In 20 actual instances of different sizes, MLAC-DQN outperformed its closest competitor, with a 26.6% improvement in average tardiness. Moreover, extraordinary robustness is demonstrated in 16 sets of experiments involving different configurations of resources, orders, and arrival concentration levels.

Using experience classification for training non-Markovian tasks

Unlike standard Reinforcement Learning (RL) model, many real-world tasks are non-Markovian, which requires long-term memory and dependency. Hence solving a non-Markovian task is more difficult and challenging than solving a Markovian one. In this paper, we propose a novel RL approach for training non-Markovian tasks expressed in temporal logic LTLf (Linear Temporal Logic over Finite Traces). To this end, an encoding of linear complexity from LTLf into MDPs (Markov Decision Processes) is introduced in order to take advantage of advanced RL algorithms. We further propose an experience classification method based on the automaton structure (theoretically equivalent to LTLf specifications). An automatic reward shaping technique and a prioritized experience replay mechanism are developed to cooperate with the classification method to improve the performance of RL algorithms. We provide empirical evaluations on two widely used benchmark problems, Waterworld and Cartpole, both augmented with complex non-Markovian tasks. The evaluations are conducted with respect to the metrics of training speed, policy quality, convergence rates, computational efficiency and scalability etc. The experimental results show that our approach achieves superior performance over other relevant studies, specially with an average improvement of 133% in convergence rate and a reduction of 11% in training time.

Distributed Multiagent Reinforcement Learning With Action Networks for Dynamic Economic Dispatch.

A new class of distributed multiagent reinforcement learning (MARL) algorithm suitable for problems with coupling constraints is proposed in this article to address the dynamic economic dispatch problem (DEDP) in smart grids. Specifically, the assumption made commonly in most existing results on the DEDP that the cost functions are known and/or convex is removed in this article. A distributed projection optimization algorithm is designed for the generation units to find the feasible power outputs satisfying the coupling constraints. By using a quadratic function to approximate the state-action value function of each generation unit, the approximate optimal solution of the original DEDP can be obtained by solving a convex optimization problem. Then, each action network utilizes a neural network (NN) to learn the relationship between the total power demand and the optimal power output of each generation unit, such that the algorithm obtains the generalization ability to predict the optimal power output distribution on an unseen total power demand. Furthermore, an improved experience replay mechanism is introduced into the action networks to improve the stability of the training process. Finally, the effectiveness and robustness of the proposed MARL algorithm are verified by simulation.

Secure physical layer transport strategies in mobile edge computing

Abstract In the past, mobile edge computing faced problems such as information leakage and eavesdropping during offloading of computing resources. To bolster the comprehensive security of the mobile edge computing system, a physical layer secure transmission strategy in mobile edge computing is proposed. First, a drone-mobile edge computing system is studied, which consists of M legitimate drones, N user devices, and K eavesdropping drones. Second, a multi-intelligence deep deterministic policy gradient algorithm is used to optimize the flight path, transmit power, and offloading ratio of the legitimate UAVs in combination with a prioritized empirical replay mechanism. Finally, under the premise of guaranteeing offloading, it realizes that the user’s confidential information is not stolen by eavesdroppers, safeguards the overall security of the system, and maximizes the efficiency of security computing. In this paper, the proposed algorithm enhances security transmission efficiency by 50% compared to other benchmark schemes. This demonstrates that the algorithm is effective.

Smart line planning method for power transmission based on D3QN‐PER algorithm

AbstractThe planning of power transmission line projects encompasses vast and complex geographical terrains. To address the complexity of transmission line planning and achieve lower line costs, this study proposes a novel intelligent line planning method. For the first time, it combines the Dueling Double Deep Q Network (D3QN) with the prioritized experience replay (PER) mechanism. First, correlate the reward function with metrics such as line length, number of corner points, and geographical environmental data, which are pertinent to the construction costs of power transmission line. Second, the D3QN algorithm is formulated by integrating Double DQN and Dueling DQN. The network's input information is divided into two components during training, aligning with the characteristics of power transmission line planning projects. Finally, the convergence efficiency of the algorithm is improved by using the PER mechanism for the problem of cost difference due to the different number of corner points in the planning path. In order to test the feasibility of the algorithm, we conducted experiments using real maps. Compared with the traditional ant colony optimization (ACO) algorithm, the D3QN‐PER deep reinforcement learning algorithm reduces the line length by more than 4% and the number of corner points by more than 60%.

A Modular Robotic Arm Configuration Design Method Based on Double DQN with Prioritized Experience Replay

Abstract: The modular robotic arms can achieve desired performances in different scenarios through the combination of various modules, and concurrently hold the potential to exhibit geometric symmetry and uniform mass symmetry. Therefore, selecting the appropriate combination of modules is crucial for realizing the functions of the robotic arm and ensuring the elegance of the system. To this end, this paper proposes a double deep Q-network (DDQN)-based configuration design algorithm for modular robotic arms, which aims to find the optimal configuration under different tasks. First, a library of small modules of collaborative robotic arms consisting of multiple tandem robotic arms is constructed. These modules are described in a standard format that can be directly imported into the software for simulation, providing greater convenience and flexibility in the development of modular robotic arms. Subsequently, the DDQN design framework for module selection is established to obtain the optimal robotic arm configuration. The proposed method could deal with the overestimation problem in the traditional deep Q-network (DQN) method and improve the estimation accuracy of the value function for each module. In addition, the experience replay mechanism is improved based on the SumTree technique, which enables the algorithm to make effective use of historical experience and prevents the algorithm from falling into local optimal solutions. Finally, comparative experiments are carried out on the PyBullet simulation platform to verify the effectiveness and superiority of the configuration design method developed in the paper. The simulation results show that the proposed DDQN-based method with experience replay mechanism has higher search efficiency and accuracy compared to the traditional DQN scheme.

Deep reinforcement learning based latency-energy minimization in smart healthcare network

Significant breakthroughs in the Internet of Things (IoT) and 5G technologies have driven several smart healthcare activities, leading to a flood of computationally intensive applications in smart healthcare networks. Mobile Edge Computing (MEC) is considered as an efficient solution to provide powerful computing capabilities to latency or energy sensitive nodes. The low-latency and high-reliability requirements of healthcare application services can be met through optimal offloading and resource allocation for the computational tasks of the nodes. In this study, we established a system model consisting of two types of nodes by considering nondivisible and trade-off computational tasks between latency and energy consumption. To minimize processing cost of the system tasks, a Mixed-Integer Nonlinear Programming (MINLP) task offloading problem is proposed. Furthermore, this problem is decomposed into task offloading decisions and resource allocation problems. The resource allocation problem is solved using traditional optimization algorithms, and the offloading decision problem is solved using a deep reinforcement learning algorithm. We propose an Online Offloading based on the Deep Reinforcement Learning (OO-DRL) algorithm with parallel deep neural networks and a weight-sensitive experience replay mechanism. Simulation results show that, compared with several existing methods, our proposed algorithm can perform real-time task offloading in a smart healthcare network in dynamically varying environments and reduce the system task processing cost.

Collision avoidance decision-making strategy for multiple USVs based on Deep Reinforcement Learning algorithm

—To address the issue of flexible decision-making for Unmanned Surface Vehicles (USVs) in emergency collision avoidance scenarios, this paper proposes a Collision Avoidance Decision-making Strategy (CADMS) for multiple USVs based on Deep Reinforcement Learning (DRL). The novelty of this method is: (1) The Preferential Experience Replay Mechanism (PERM) and the Gate Recurrent Unit (GRU) network are employed to enhance the Deep Deterministic Policy Gradient (DDPG) algorithm. The PERM facilitates faster convergence by increasing the sampling probability, thereby aiding in data assimilation. Meanwhile, the GRU enables self-assessment of sensor input significance, leading to more accurate action prediction. (2) In the design of the reward function, the departure from COLREGs has been fully considered, including the two-ship encounter situations and the multi-ship encounter situations. This study also provides a comprehensive analysis of neural network training and generalization verification. The experimental results demonstrate the effectiveness of the proposed approach in effectively avoiding collisions with multiple USVs in complex water environments. It is noteworthy that in comparison to conventional collision avoidance methods, the proposed CADMS model exhibits a superior collision avoidance success rate and enables safer and more efficient decision-making behaviors when confronted with intricate encounter situations.

Deep Reinforcement Learning-Based 3D Trajectory Planning for Cellular Connected UAV

To address the issue of limited application scenarios associated with connectivity assurance based on two-dimensional (2D) trajectory planning, this paper proposes an improved deep reinforcement learning (DRL) -based three-dimensional (3D) trajectory planning method for cellular unmanned aerial vehicles (UAVs) communication. By considering the 3D space environment and integrating factors such as UAV mission completion time and connectivity, we develop an objective function for path optimization and utilize the advanced dueling double deep Q network (D3QN) to optimize it. Additionally, we introduce the prioritized experience replay (PER) mechanism to enhance learning efficiency and expedite convergence. In order to further aid in trajectory planning, our method incorporates a simultaneous navigation and radio mapping (SNARM) framework that generates simulated 3D radio maps and simulates flight processes by utilizing measurement signals from the UAV during flight, thereby reducing actual flight costs. The simulation results demonstrate that the proposed approach effectively enable UAVs to avoid weak coverage regions in space, thereby reducing the weighted sum of flight time and expected interruption time.

Enhanced LSTM‐DQN algorithm for a two‐player zero‐sum game in three‐dimensional space

AbstractTo tackle the challenges presented by the two‐player zero sum game (TZSG) in three‐dimensional space, this study introduces an enhanced deep Q‐learning (DQN) algorithm that utilizes long short term memory (LSTM) network. The primary objective of this algorithm is to enhance the temporal correlation of the TZSG in three‐dimensional space. Additionally, it incorporates the hindsight experience replay (HER) mechanism to improve the learning efficiency of the network and mitigate the issue of the “sparse reward” that arises from prolonged training of intelligence in solving the TZSG in the three‐dimensional. Furthermore, this method enhances the convergence and stability of the overall solution.An intelligent training environment centred around an airborne agent and its mutual pursuit interaction scenario was designed to proposed approach's effectiveness. The algorithm training and comparison results show that the LSTM‐DQN‐HER algorithm outperforms similar algorithm in solving the TZSG in three‐dimensional space. In conclusion, this paper presents an improved DQN algorithm based on LSTM and incorporates the HER mechanism to address the challenges posed by the TZSG in three‐dimensional space. The proposed algorithm enhances the solution's temporal correlation, learning efficiency, convergence, and stability. The simulation results confirm its superior performance in solving the TZSG in three‐dimensional space.