Replay Mechanism Research Articles

Task offloading and trajectory scheduling for UAV-enabled MEC networks: An MADRL algorithm with prioritized experience replay

As a new network architecture, the air-ground cooperative network is a support for future 6G network to achieve ubiquitous connectivities. To effectively relieve the computational pressure of massive data in 6G wireless networks, Unmanned Aerial Vehicles (UAVs) equipped with Mobile Edge Computing (MEC) servers have become an emerging technology that provides computing resources for Mobile Devices (MDs). Due to limited on-board energy and computational capabilities, this paper investigates a multi-UAV collaborative assisted MEC architecture. The optimization problem of minimizing the total computational cost is constructed by jointly optimizing the UAVs trajectories and MDs offloading strategies scheduling. The coupling between the optimization variables and the non-convexity of the problem can make it difficult to solve directly. To address the above concerns, the non-convex optimization problem is converted into a Markov decision process. The UAVs-assisted Offloading Strategy based on Reinforcement Learning (UOS-RL) algorithm is proposed to address the convergence difficulties caused by the high-dimensional continuous action space. Furthermore, the experience data generated by agents interacting with the environment is highly differentiated due to the highly dynamic variation of the environment. Hence, a Priority Experience Replay (PER) mechanism is proposed to improve the training efficiency of the UOS-RL algorithm based on the priority of the experience data. Simulation results show that the proposed PER-UOS-RL algorithm outperforms the existing works in terms of the computational cost.

A Novel Path Planning Approach for Mobile Robot in Radioactive Environment Based on Improved Deep Q Network Algorithm

The path planning problem of nuclear environment robots refers to finding a collision-free path under the constraints of path length and an accumulated radiation dose. To solve this problem, the Improved Dueling Deep Double Q Network algorithm (ID3QN) based on asymmetric neural network structure was proposed. To address the issues of overestimation and low sample utilization in the traditional Deep Q Network (DQN) algorithm, we optimized the neural network structure and used the double network to estimate action values. We also improved the action selection mechanism, adopted a priority experience replay mechanism, and redesigned the reward function. To evaluate the efficiency of the proposed algorithm, we designed simple and complex radioactive grid environments for comparison. We compared the ID3QN algorithm with traditional algorithms and some deep reinforcement learning algorithms. The simulation results indicate that in the simple radioactive grid environment, the ID3QN algorithm outperforms traditional algorithms such as A*, GA, and ACO in terms of path length and accumulated radiation dosage. Compared to other deep reinforcement learning algorithms, including DQN and some improved DQN algorithms, the ID3QN algorithm reduced the path length by 15.6%, decreased the accumulated radiation dose by 23.5%, and converged approximately 2300 episodes faster. In the complex radioactive grid environment, the ID3QN algorithm also outperformed the A*, GA, ACO, and other deep reinforcement learning algorithms in terms of path length and an accumulated radiation dose. Furthermore, the ID3QN algorithm can plan an obstacle-free optimal path with a low radiation dose even in complex environments. These results demonstrate that the ID3QN algorithm is an effective approach for solving robot path planning problems in nuclear environments, thereby enhancing the safety and reliability of robots in such environments.

A Multiobjective Collaborative Deep Reinforcement Learning Algorithm for Jumping Optimization of Bipedal Robot

Due to the nonlinearity and underactuation of bipedal robots, developing efficient jumping strategies remains challenging. To address this, a multiobjective collaborative deep reinforcement learning algorithm based on the actor‐critic framework is presented. Initially, two deep deterministic policy gradient (DDPG) networks are established for training the jumping motion, each focusing on different objectives and collaboratively learning the optimal jumping policy. Following this, a recovery experience replay mechanism, predicated on dynamic time warping, is integrated into the DDPG to enhance sample utilization efficiency. Concurrently, a timely adjustment unit is incorporated, which works in tandem with the training frequency to improve the convergence accuracy of the algorithm. Additionally, a Markov decision process is designed to manage the complexity and parameter uncertainty in the dynamic model of the bipedal robot. Finally, the proposed method is validated on a PyBullet platform. The results show that the method outperforms baseline methods by improving learning speed and enabling robust jumps with greater height and distance.

Data-Driven Load Frequency Control Based on Multi-Agent Reinforcement Learning With Attention Mechanism

With the massive penetration of renewable energy, traditional reinforcement learning algorithms suffer from slow convergence and area control error (ACE) in interconnected power systems. This paper proposes data-driven load frequency control (LFC) based on multi-agent reinforcement learning with attention mechanism in interconnected power systems. It can be divided into two phases; in the centralized training, the agents are trained by an experience replay mechanism; in the decentralized execution, the trained agent automatically regulates the generation power to control the load frequency by real-time access to the grid data in the area. The agent can selectively focus on specific information in the environment by introducing a criticism network with an attention mechanism. The attention mechanism can reduce the training time for reinforcement learning while improving control performance under disturbance. A novel reward function based on a cooperation mechanism is used to score the performance of agent, which can guide the reinforcement learning algorithm to reduce the ACE of each area simultaneously. The proposed method is validated by the IEEE three-area interconnected power system, and it is concluded that the method can reduce the ACE caused by load and renewable power disturbances, and greatly reduce the training time of the algorithm.

A Path-Planning Method Based on Improved Soft Actor-Critic Algorithm for Mobile Robots.

The path planning problem has gained more attention due to the gradual popularization of mobile robots. The utilization of reinforcement learning techniques facilitates the ability of mobile robots to successfully navigate through an environment containing obstacles and effectively plan their path. This is achieved by the robots' interaction with the environment, even in situations when the environment is unfamiliar. Consequently, we provide a refined deep reinforcement learning algorithm that builds upon the soft actor-critic (SAC) algorithm, incorporating the concept of maximum entropy for the purpose of path planning. The objective of this strategy is to mitigate the constraints inherent in conventional reinforcement learning, enhance the efficacy of the learning process, and accommodate intricate situations. In the context of reinforcement learning, two significant issues arise: inadequate incentives and inefficient sample use during the training phase. To address these challenges, the hindsight experience replay (HER) mechanism has been presented as a potential solution. The HER mechanism aims to enhance algorithm performance by effectively reusing past experiences. Through the utilization of simulation studies, it can be demonstrated that the enhanced algorithm exhibits superior performance in comparison with the pre-existing method.

Spiking generative networks empowered by multiple dynamic experts for lifelong learning

Spiking neural networks (SNNs) have garnered significant attention due to their ultra-high-speed and ultra-low-power operation, rendering them suitable for a range of energy-efficient applications. However, their performance in image generation is relatively mediocre, resulting in a lack of an effective generative replay mechanism to handle sequential probabilistic representations of multiple datasets. Consequently, this limitation often leads to catastrophic forgetting. This paper introduces a novel SNN lifelong learning framework namely Dynamic Lifelong learning with Spiking Generative Networks (DL-SGN), aimed at providing better generation ability while mitigating the problem of catastrophic forgetting in a continual learning scenario. DL-SGN comprises three key components: dynamic experts, a student, and an assistant. The dynamic expert is implemented as a dynamically expanding mixture model, with a proposed network expansion mechanism Dynamic Knowledge Adversarial Fusion (DKAF) facilitating the automatic handling of an increasing number of tasks. the student module adopts an SNN-based Variational Autoencoder (VAE) and classifier, leveraging accumulated knowledge from each expert. To enhance the student’s ability to generalize to images, we introduce a discriminator as an assistant module trained using adversarial training. We validate the effectiveness of the proposed by conducting experiments on image generation and classification in a lifelong learning environment, and the results substantiate the effectiveness of DL-SGN.

Ethical and moral decision-making for self-driving cars based on deep reinforcement learning

Self-driving cars are expected to replace human drivers shortly, bringing significant benefits to society. However, they have faced opposition from various organizations that argue it is challenging to respond to instances involving unavoidable personal injury. In situations involving deadly collisions, self-driving cars must make decisions that balance life and death. This paper investigates the ethical and moral decision-making challenges for self-driving cars from an algorithmic perspective. To address this issue, we introduce the accident-prioritized replay mechanism to the Deep Q-Networks (DQN) algorithm based on early humanities research. The mechanism quantifies a reward function that takes priority into account. RGB (red, green, blue) images obtained by the camera installed in front of the self-driving cars are fed into the Xception network for training. To evaluate our approach, we compare it to the conventional DQN algorithm. The simulation results indicate that the Rawlsian DQN algorithm has superior stability and interpretability in decision-making. Furthermore, the majority of respondents to our survey accept the final decision made by our algorithm. Our experiment demonstrates that it is possible to incorporate ethical considerations into self-driving car decision-making, providing a solution for rational decision-making in emergency and dilemma circumstances.

Control of a nonlinear active suspension system based on deep reinforcement learning and expert demonstrations

A well-controlled active suspension system has the potential to provide better ride comfort. Benefiting from its powerful feature extraction and nonlinear generalization capabilities, the deep reinforcement learning (DRL), such as deep deterministic policy gradient (DDPG), has shown great potential to make decisions adaptively and intelligently in the control of active suspension system. However, the DDPG is troubled by the problem of low training efficiency due to the high proportion of illegal strategies. This paper proposed a novel DDPG controller for a nonlinear uncertain active suspension system by combining DRL with expert demonstrations. Specifically, the improved training method integrated with both a pre-training mechanism based on PID expert samples and an adaptive experience replay mechanism, is put forward for the DDPG to achieve both the goals of imitating the expert and improving the training efficiency. Moreover, considering the ride comfort and the state constraints as targets, a mixed reward function is designed to guide RL agents for learning effective actions. It is shown that the proposed training methods effectively accelerate the convergence of the DDPG. Furthermore, the comparison experiments demonstrate that the proposed controller provides great vibration attenuation, and has better adaptiveness to various working conditions and parametric uncertainty.

AUV Path Planning Considering Ocean Current Disturbance Based on Cloud Desktop Technology.

In the field of ocean energy detection, Autonomous Underwater Vehicles (AUVs) offer significant advantages in terms of manpower, resource, and energy efficiency. However, the unpredictable nature of the ocean environment, particularly the real-time changes in ocean currents, poses navigational risks for AUVs. Therefore, effective path planning in dynamic environments is crucial for AUVs to perform specific tasks. This paper addresses the static path planning problem and proposes a model called the noise net double DQN network with prioritized experience replay (N-DDQNP). The N-DDQNP model combines a noise network and a prioritized experience replay mechanism to address the limited exploration and slow convergence speed issues of the DQN algorithm, which are caused by the greedy strategy and uniform sampling mechanism. The proposed approach involves constructing a double DQN network with a priority experience replay and an exploration mechanism using the noise network. Second, a compound reward function is formulated to take into account ocean current, distance, and safety factors, ensuring prompt feedback during the training process. Regarding the ocean current, the reward function is designed based on the angle between the current direction and the AUV's heading direction, considering its impact on the AUV's speed. As for the distance factor, the reward is determined by the Euclidean distance between the current position and the target point. Furthermore, the safety factor considers whether the AUV may collide with obstacles. By incorporating these three factors, the compound reward function is established. To evaluate the performance of the N-DDQNP model, experiments were conducted using real ocean data in various complex ocean environments. The results demonstrate that the path planning time of the N-DDQNP model outperforms other algorithms in different ocean current scenarios and obstacle environments. Furthermore, a user console-AUV connection has been established using spice cloud desktop technology. The cloud desktop architecture enables intuitive observation of the AUV's navigation posture and the surrounding marine environment, facilitating safer and more efficient underwater exploration and marine resource detection tasks.

A parallel heterogeneous policy deep reinforcement learning algorithm for bipedal walking motion design.

Considering the dynamics and non-linear characteristics of biped robots, gait optimization is an extremely challenging task. To tackle this issue, a parallel heterogeneous policy Deep Reinforcement Learning (DRL) algorithm for gait optimization is proposed. Firstly, the Deep Deterministic Policy Gradient (DDPG) algorithm is used as the main architecture to run multiple biped robots in parallel to interact with the environment. And the network is shared to improve the training efficiency. Furthermore, heterogeneous experience replay is employed instead of the traditional experience replay mechanism to optimize the utilization of experience. Secondly, according to the walking characteristics of biped robots, a biped robot periodic gait is designed with reference to sinusoidal curves. The periodic gait takes into account the effects of foot lift height, walking period, foot lift speed and ground contact force of the biped robot. Finally, different environments and different biped robot models pose challenges for different optimization algorithms. Thus, a unified gait optimization framework for biped robots based on the RoboCup3D platform is established. Comparative experiments were conducted using the unified gait optimization framework, and the experimental results show that the method outlined in this paper can make the biped robot walk faster and more stably.

Improved PER-DDPG based nonparametric modeling of ship dynamics with uncertainty

This study contributes to addressing the challenge of quickly obtaining an effective and accurate nonparametric model for describing ship maneuvering motion in three degrees of freedom (3-DOF). To achieve this, an intelligent ship dynamics nonparametric modeling method named improved PER-DDPG is proposed. This method leverages the deep deterministic policy gradient algorithm (DDPG) and prioritized experience replay mechanism (PER) and analyzes the characteristics between the goal of deep reinforcement learning (DRL) and the modeling process of the nonparametric model. The PER mechanism is utilized to enhance the agent’s understanding of the overall mechanism of ship motion by improving the utilization of samples. The meaning of target value is redefined due to transforming DRL aiming at maximizing cumulative rewards into maximizing the set of immediate rewards at each time step. To validate the performance of the proposed modeling method, we conduct simulation studies using a benchmark ship model i.e., a Mariner cargo ship dynamic model, and experimental studies using a real unmanned surface vehicle (USV). In the simulation test, we demonstrate the effectiveness and generalization of the proposed method through zigzag and turning circle tests. Furthermore, we verify the robustness and applicability of the proposed method by using datasets with uncertain environmental disturbances and datasets with different sampling frequencies. Additionally, the experimental tests conducted on the USV indicate the consistency of the proposed approach.

Research on energy management of hydrogen electric coupling system based on deep reinforcement learning

In this paper, a deep reinforcement learning-based energy optimization management method for hydrogen-electric coupling system is proposed for the conversion and utilization and joint optimization operation of hydrogen, wind and solar energy forms considering information uncertainty on the demand side of smart grid. Based on the wind energy, photovoltaic energy generation and load forecast information, the method uses deep Q network to simulate the energy management strategy set of the hydrogen-electric coupling system, and obtains the optimal strategy through reinforcement learning to finally realize the optimal operation of the hydrogen-electric coupling system based on the demand response. Firstly, based on the energy management model, a research framework and equipment model for integrated energy systems is established. On the basis of fundamental theories of reinforcement learning framework, Q-learning algorithm and DQN algorithm, the empirical replay mechanism and freezing parameter mechanism to improve the performance of DQN are analyzed, and the energy management and optimization of integrated energy system is completed with the objective of economy. By comparing the performance of DQN algorithms with different parameters in integrated energy system energy management, the simulation results demonstrate the improvement of algorithm performance after inheriting the set of strategies, and verify the feasibility and superiority of deep reinforcement learning compared to genetic algorithm in integrated energy system energy management applications.

Model-Free Control in Wireless Cyber-Physical System With Communication Latency: A DRL Method With Improved Experience Replay.

This article explores the model-free remote control problem in a wireless networked cyber-physical system (CPS) composed of spatially distributed sensors, controllers, and actuators. The sensors sample the states of the controlled system to generate control instructions at the remote controller, while the actuators maintain the system's stability by executing control commands. To realize the control under a model-free system, the deep deterministic policy gradient (DDPG) algorithm is adopted in the controller to enable model-free control. Unlike the traditional DDPG algorithm, which only takes the system state as input, this article incorporates historical action information as input to extract more information and achieve precise control in the case of communication latency. Additionally, in the experience replay mechanism of the DDPG algorithm, we incorporate the reward into the prioritized experience replay (PER) approach. According to the simulation results, the proposed sampling policy improves the convergence rate by determining the sampling probability of transitions based on the joint consideration of temporal difference (TD) error and reward.

Federated Zero-Shot Industrial Fault Diagnosis With Cloud-Shared Semantic Knowledge Base

Recently, a considerable literature has grown up around the few-sample fault diagnosis task, in which few samples of fault data are available for model training. The lack of fault samples usually causes significant degradation of diagnosis performance, which is referred as few-sample problem. Many federated learning-based fault diagnosis methods are proposed as a popular strategy to solve this problem. However, the vast majority of these algorithms assume that few-sample categories of the local client are well-sampled in some clients. In this sense, they are unable to cope with faults that have rare or even no sample in all clients, namely global unseen faults. To diagnosis global unseen fault categories, a federated zero-shot fault diagnosis framework is proposed in this article. First, we propose a general approach for building a semantic knowledge base, which provides auxiliary discriminant descriptions of different faults. Second, a bidirectional alignment network is designed using two coupled variational autoencoders, enabling the fusion of data and attributes and allowing attribute descriptions to assist in fault diagnosis tasks. Third, a cloud-edge collaboration model aggregation strategy is developed, which utilizes generative replay mechanism and integrates the knowledge of each client, thus enhancing the generalization and generation ability of the global model. Experiments conducted on the thermal power plant group illustrate the feasibility and effectiveness of the proposed framework to categorize local and global unseen classes.

UAV Air Game Maneuver Decision-Making Using Dueling Double Deep Q Network with Expert Experience Storage Mechanism

Deep reinforcement learning technology applied to three-dimensional Unmanned Aerial Vehicle (UAV) air game maneuver decision-making often results in low utilization efficiency of training data and algorithm convergence difficulties. To address these issues, this study proposes an expert experience storage mechanism that improves the algorithm’s performance with less experience replay time. Based on this mechanism, a maneuver decision algorithm using the Dueling Double Deep Q Network is introduced. Simulation experiments demonstrate that the proposed mechanism significantly enhances the algorithm’s performance by reducing the experience by 81.3% compared to the prioritized experience replay mechanism, enabling the UAV agent to achieve a higher maximum average reward value. The experimental results suggest that the proposed expert experience storage mechanism improves the algorithm’s performance with less experience replay time. Additionally, the proposed maneuver decision algorithm identifies the optimal policy for attacking target UAVs using different fixed strategies.

Research on adaptive obstacle avoidance algorithm of robot based on DDPG-DWA

In this paper, we propose an adaptive obstacle avoidance algorithm based on DDPG (Deep Deterministic Policy Gradient) and DWA (Dynamic-Window Approach) to study the obstacle avoidance problem of robots in complex continuous state space. First, the obstacle avoidance problem is converted into an optimal learning incentive problem, and the self-learning of the obstacle avoidance policy is realized based on DDPG; second, the DWA obstacle avoidance trajectory evaluation function is optimized using the DDPG reward incentive mechanism, and the Experience Replay mechanism; finally, the algorithm model is simulated. The experiments show that the model can significantly circumvent the deficiency of the DWA algorithm in limiting to the optimal local solution in a complex environment and solve the action output problem in the continuous velocity and turning angle value interval of the robot; through the trial and error interaction with the environment and the trajectory evaluation incentive feedback, the obstacle avoidance passing ability of the robot in a complex environment is improved.

Deep Reinforcement Learning Based Link Adaptation Technique for LTE/NR Systems

Outdated channel quality indicator (CQI) feedback causes severe performance degradation of traditional link adaptation (LA) techniques in long term evolution (LTE) and new radio (NR) systems. This paper puts forth a deep reinforcement learning (DRL) based link adaptation (LA) technique, referred to as deep reinforcement learning link adaptation (DRLLA), to select efficient modulation and coding scheme (MCS) in the presence of the outdated CQI feedback. The goal of DRLLA is to maximize the link throughput while achieving a low block error rate (BLER). We first give explicit definitions of state, action, and reward in DRL paradigms, thereby realizing DRLLA. Then, to trade off the throughput against the BLER, we further develop a new experience replay mechanism called classified experience replay (CER) as the underpinning technique in DRLLA. In CER, experiences are separated into two buckets, one for successful experiences and the other for failed experiences, and then a fixed proportion from each is sampled to replay. The essence of CER is to obtain different trade-offs via adjusting the proportion among different training experiences. Furthermore, to reduce the signaling overhead and the system reconfiguration cost caused by frequent MCS switching, we propose a new action selection strategy termed as switching-controlled <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> -greedy (SC- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> -greedy) for DRLLA. Simulation results demonstrate that compared with the state-of-the-art OLLA, LTSLA, and DRLLA with other experience replay mechanisms, DRLLA with CER can achieve higher throughput and lower BLER in various time-varying scenarios, and be more robust to different CQI feedback delays and CQI reporting periods. Furthermore, with the SC- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> -greedy policy, DRLLA can capture better trade-offs between the link transmission quality and the MCS switching overhead compared with other baselines.

Task offloading strategy and scheduling optimization for internet of vehicles based on deep reinforcement learning

Driven by the construction of smart cities, networks and communication technologies are gradually infiltrating into the Internet of Things (IoT) applications in urban infrastructure, such as automatic driving. In the Internet of Vehicles (IoV) environment, intelligent vehicles will generate a lot of data. However, the limited computing power of in-vehicle terminals cannot meet the demand. To solve this problem, we first simulate the task offloading model of vehicle terminal in Mobile Edge Computing (MEC) environment. Secondly, according to the model, we design and implement a MEC server collaboration scheme considering both delay and energy consumption. Thirdly, based on the optimization theory, the system optimization solution is formulated with the goal of minimizing system cost. Because the problem to be resolved is a mixed binary nonlinear programming problem, we model the problem as a Markov Decision Process (MDP). The original resource allocation decision is turned into a Reinforcement Learning (RL) problem. In order to achieve the optimal solution, the Deep Reinforcement Learning (DRL) method is used. Finally, we propose a Deep Deterministic Policy Gradient (DDPG) algorithm to deal with task offloading and scheduling optimization in high-dimensional continuous action space, and the experience replay mechanism is used to accelerate the convergence and enhance the stability of the network. The simulation results show that our scheme has good performance optimization in terms of convergence, system delay, average task energy consumption and system cost. For example, compared with the comparison algorithm, the system cost performance has improved by 9.12% under different task sizes, which indicates that our scheme is more suitable for highly dynamic Internet of Vehicles environment.

FLIRRAS: Fast Learning With Integrated Reward and Reduced Action Space for Online Multitask Offloading

With the rapid development of edge data intelligence, task offloading (TO) and resource allocation (RA) optimization in multiaccess edge computing networks can significantly improve the Quality of Service (QoS). However, for the online scenario, traditional methods (e.g., game theory and numerical methods) cannot adapt to dynamic environments. Deep reinforcement learning (DRL) is applied to adjust the policy to get long-term rewards. Nevertheless, since the joint problem of TO and RA is nonconvex and NP-hard, existing DRL methods cannot guarantee high efficiency because of the large action space. To solve the above problem, we propose a fast learning with integrated reward and reduced action space-based DRL framework (FLIRRAS), which adopts a low-complexity approach to jointly optimize TO and RA strategies. The FLIRRAS framework combines DRL with numerical methods to iteratively pursues the discrete TO and continuous RA. Specifically, a deep neural network (DNN) is used to learn environmental information, which can get prior knowledge of the offloading decision. Furthermore, a novel reward integrating the utility of TO and RA is designed to motivate the agent to find the optimal policy. To solve the dilemma that the action space is too large, low-complexity convex optimization methods, i.e., subgradient projection and KKT condition, are used to supplement and adjust the decision, which reduces the network parameters and the decision space. In addition, given the dynamic online environment, we introduce the experience replay mechanism, where policy is updated regularly to reflect the best mapping between states. The experiment results show that the performance of FLIRRAS is better than greedy and other DRL approaches, and it outperforms the latest DRL method by over 18.0% in terms of execution time.

A model of hippocampal replay driven by experience and environmental structure facilitates spatial learning.

Replay of neuronal sequences in the hippocampus during resting states and sleep play an important role in learning and memory consolidation. Consistent with these functions, replay sequences have been shown to obey current spatial constraints. Nevertheless, replay does not necessarily reflect previous behavior and can construct never-experienced sequences. Here, we propose a stochastic replay mechanism that prioritizes experiences based on three variables: 1. Experience strength, 2. experience similarity, and 3. inhibition of return. Using this prioritized replay mechanism to train reinforcement learning agents leads to far better performance than using random replay. Its performance is close to the state-of-the-art, but computationally intensive, algorithm by Mattar & Daw (2018). Importantly, our model reproduces diverse types of replay because of the stochasticity of the replay mechanism and experience-dependent differences between the three variables. In conclusion, a unified replay mechanism generates diverse replay statistics and is efficient in driving spatial learning.