Replay Buffer Research Articles

Real-Time On-Device Continual Learning Based on a Combined Nearest Class Mean and Replay Method for Smartphone Gesture Recognition

Sensor-based gesture recognition on mobile devices is critical to human–computer interaction, enabling intuitive user input for various applications. However, current approaches often rely on server-based retraining whenever new gestures are introduced, incurring substantial energy consumption and latency due to frequent data transmission. To address these limitations, we present the first on-device continual learning framework for gesture recognition. Leveraging the Nearest Class Mean (NCM) classifier coupled with a replay-based update strategy, our method enables continuous adaptation to new gestures under limited computing and memory resources. By employing replay buffer management, we efficiently store and revisit previously learned instances, mitigating catastrophic forgetting and ensuring stable performance as new gestures are added. Experimental results on a Samsung Galaxy S10 device demonstrate that our method achieves over 99% accuracy while operating entirely on-device, offering a compelling synergy between computational efficiency, robust continual learning, and high recognition accuracy. This work demonstrates the potential of on-device continual learning frameworks that integrate NCM classifiers with replay-based techniques, thereby advancing the field of resource-constrained, adaptive gesture recognition.

Agent Enhancement using Deep Reinforcement Learning Algorithms for Multiplayer game (Slither.io)

Developing self learning model for a game is challenging as the environment keeps changing all the time and therefore require highly intelligent models which can make decisions depending on the environment in real time. The agent has to learn the environment and takes action based on the inference. Based on the action, a positive or negative reward is given to the agent. The agent again learns from the reward and enhances / trains itself to behave better in the environment. This work aims to train an agent using deep reinforcement learning algorithms to play a multiplayer online game like SLITHER.IO. We use an OpenAI Universe environment to collect raw image inputs from sample gaming as training data. Agent learns the current state of the environment and the position of the other players (snakes). Then it takes action in the form of direction of its movement. To compare our model to other existing systems and random policy, we propose to use deep Q-learning and other actor critic approaches such as Proximal Policy Optimisation (PPO) with reward shaping and replay buffer. Out of all these algorithms the PPO agent shows significant improvement in the score over a range of episodes. PPO agent learns quickly and its reward progression is higher when compared to other techniques.

Dynamic Recognition of Speakers for Consent Management by Contrastive Embedding Replay.

Voice assistants overhear conversations, and a consent management mechanism is required. Consent management can be implemented using speaker recognition. Users that do not give consent enroll their voice, and all their further recordings are discarded. Building speaker recognition-based consent management is challenging as dynamic registration, removal, and reregistration of speakers must be efficiently handled. This work proposes a consent management system addressing the aforementioned challenges. A contrastive-based training is applied to learn the underlying speaker equivariance inductive bias. The contrastive features for buckets of speakers are trained a few steps into each iteration and act as replay buffers. These features are progressively selected using a multi-strided random sampler for classification. Moreover, new methods for dynamic registration using a portion of old utterances, removal, and reregistration of speakers are proposed. The results verify memory efficiency and dynamic capabilities of the proposed methods and outperform the existing approaches from the literature in terms of convergence rate and number of required parameters.

Optimal Adaptive Tracking Control of Partially Uncertain Nonlinear Discrete-Time Systems Using Lifelong Hybrid Learning.

This article addresses a multilayer neural network (MNN)-based optimal adaptive tracking of partially uncertain nonlinear discrete-time (DT) systems in affine form. By employing an actor-critic neural network (NN) to approximate the value function and optimal control policy, the critic NN is updated via a novel hybrid learning scheme, where its weights are adjusted once at a sampling instant and also in a finite iterative manner within the instants to enhance the convergence rate. Moreover, to deal with the persistency of excitation (PE) condition, a replay buffer is incorporated into the critic update law through concurrent learning. To address the vanishing gradient issue, the actor and critic MNN weights are tuned using control input and temporal difference errors (TDEs), respectively. In addition, a weight consolidation scheme is incorporated into the critic MNN update law to attain lifelong learning and overcome catastrophic forgetting, thus lowering the cumulative cost. The tracking error, and the actor and critic weight estimation errors are shown to be bounded using the Lyapunov analysis. Simulation results using the proposed approach on a two-link robot manipulator show a significant reduction in tracking error by 44% and cumulative cost by 31% in a multitask environment.

Online Active Continual Learning for Robotic Lifelong Object Recognition.

In real-world applications, robotic systems collect vast amounts of new data from ever-changing environments over time. They need to continually interact and learn new knowledge from the external world to adapt to the environment. Particularly, lifelong object recognition in an online and interactive manner is a crucial and fundamental capability for robotic systems. To meet this realistic demand, in this article, we propose an online active continual learning (OACL) framework for robotic lifelong object recognition, in the scenario of both classes and domains changing with dynamic environments. First, to reduce the labeling cost as much as possible while maximizing the performance, a new online active learning (OAL) strategy is designed by taking both the uncertainty and diversity of samples into account to protect the information volume and distribution of data. In addition, to prevent catastrophic forgetting and reduce memory costs, a novel online continual learning (OCL) algorithm is proposed based on the deep feature semantic augmentation and a new loss-based deep model and replay buffer update, which can mitigate the class imbalance between the old and new classes and alleviate confusion between two similar classes. Moreover, the mistake bound of the proposed method is analyzed in theory. OACL allows robots to select the most representative new samples to query labels and continually learn new objects and new variants of previously learned objects from a nonindependent and identically distributed (i.i.d.) data stream without catastrophic forgetting. Extensive experiments conducted on real lifelong robotic vision datasets demonstrate that our algorithm, even trained with fewer labeled samples and replay exemplars, can achieve state-of-the-art performance on OCL tasks.

Cooperative game theory based coordinated scheduling of production and transportation of parallel machines with deterioration effects

Abstract Cooperative game theory is used to study the production and transportation scheduling problem of parallel machines with deterioration effects. Jobs with an initial scheduling order are processed on parallel machines and then transported to customers. Penalties are incurred for early or late completion if the jobs are not completed on the due dates. Considering that jobs can rearrange the scheduling order in the coalition through cooperation to bring about cost savings, the cooperative game model is constructed by considering the jobs as players and taking the maximum cost savings of the coalition as the characteristic function. It is proven that when jobs have the same normal processing time and deterioration rate, the cooperative game has a non-empty core, and a core allocation can be obtained from the β-rule allocation method. A Dual Replay Buffer Double DQN (DRB-DDQN) algorithm is designed to get the optimal schedule of the coalition. Experiments demonstrate the effectiveness of the DRB-DDQN and β-rule allocation method in solving the game scheduling problem.

Path Planning for Robots Combined with Zero-Shot and Hierarchical Reinforcement Learning in Novel Environments

Path planning for robots based on reinforcement learning encounters challenges in integrating semantic information about environments into the training process. In unseen or complex environmental information, agents often perform sub-optimally and require more training time. In response to these challenges, this manuscript pioneers a framework integrating zero-shot learning combined with hierarchical reinforcement learning to enhance agent decision-making in complex environments. Zero-shot learning enables agents to infer correct actions for previously unseen objects or situations based on learned semantic associations. Subsequently, the path planning component utilizes hierarchical reinforcement learning with adaptive replay buffer, directed by the insights gained from zero-shot learning, to make decisions effectively. Two parts are trained separately, so zero-shot learning is available in different and unseen environments. Through simulation experiments, we compare the traditional hierarchical reinforcement learning method with the proposed method. The results prove that this structure can make full use of environmental information to generalize across unseen environments and plan collision-free paths.

Improved exploration–exploitation trade-off through adaptive prioritized experience replay

Experience replay is an indispensable part of deep reinforcement learning algorithms that enables the agent to revisit and reuse its past and recent experiences to update the network parameters. In many baseline off-policy algorithms, such as deep Q-networks (DQN), transitions in the replay buffer are typically sampled uniformly. This uniform sampling is not optimal for accelerating the agent’s training towards learning the optimal policy. A more selective and prioritized approach to experience sampling can yield improved learning efficiency and performance. In this regard, this work is devoted to the design of a novel prioritizing strategy to adaptively adjust the sampling probabilities of stored transitions in the replay buffer. Unlike existing sampling methods, the proposed algorithm takes into consideration the exploration–exploitation trade-off (EET) to rank transitions, which is of utmost importance in learning an optimal policy. Specifically, this approach utilizes temporal difference and Bellman errors as criteria for sampling priorities. To maintain balance in EET throughout training, the weights associated with both criteria are dynamically adjusted when constructing the sampling priorities. Additionally, any bias introduced by this sample prioritization is mitigated through assigning importance-sampling weight to each transition in the buffer. The efficacy of this prioritization scheme is assessed through training the DQN algorithm across various OpenAI Gym environments. The results obtained underscore the significance and superiority of our proposed algorithm over state-of-the-art methods. This is evidenced by its accelerated learning pace, greater cumulative reward, and higher success rate.

Deep Sarsa Replay model for real Time Traffic Control

Traffic congestion is a major factor currently affecting the global economy thereby costing billions of dollars in terms of the time lost in traffic, productivity and fuel wastage. Although, various researchers have made concerted effort to proffer solution to the problems of traffic congestion, in addition, recent reinforcement learning research has also presented multiple possible solutions. However, based on the literature, most of the reinforcement learning traffic control agents do not possess the real-time traffic control capabilities to efficiently deal with the complex and ever-changing nature of traffic on an average urban city road. This paper presents a Real Time deep reinforcement learning model for Traffic Control for optimization of vehicle flow (minimizing waiting time and traffic congestion). The Deep SARSA (DSARSA) learning algorithm with experience replay applied in the training process. The DSARSA replay model is built to extract traffic information from all lanes by applying multilayer perceptron with two hidden layers. Then, at the output layer an approximated qvalue is produced and updated with SARSA algorithm. For training, a random batch of experiences is sampled from the replay buffer. This helps break the correlation between consecutive experiences, stabilizes training and enables the model to be to able deal appropriately with any traffic condition. Experiments on 1000 episode of low, medium and heavy traffic flow was conducted on SUMO traffic simulator. TraCL allows interface with the SUMO simulation using python showed that the proposed DSARSA replay model achieved a state-of-the-art performance compared to other baseline DRL models applied in traffic control in terms of learning stability and reward maximization.

Green Computation Offloading With DRL in Multi‐Access Edge Computing

ABSTRACTIn multi‐access edge computing (MEC), computational task offloading of mobile terminals (MT) is expected to provide the green applications with the restriction of energy consumption and service latency. Nevertheless, the diverse statuses of a range of edge servers and mobile terminals, along with the fluctuating offloading routes, present a challenge in the realm of computational task offloading. In order to bolster green applications, we present an innovative computational task offloading model as our initial approach. In particular, the nascent model is constrained by energy consumption and service latency considerations: (1) Smart mobile terminals with computational capabilities could serve as carriers; (2) The diverse computational and communication capacities of edge servers have the potential to enhance the offloading process; (3) The unpredictable routing paths of mobile terminals and edge servers could result in varied information transmissions. We then propose an improved deep reinforcement learning (DRL) algorithm named PS‐DDPG with the prioritized experience replay (PER) and the stochastic weight averaging (SWA) mechanisms based on deep deterministic policy gradients (DDPG) to seek an optimal offloading mode, saving energy consumption. Next, we introduce an enhanced deep reinforcement learning (DRL) algorithm named PS‐DDPG, incorporating the prioritized experience replay (PER) and stochastic weight averaging (SWA) techniques rooted in deep deterministic policy gradients (DDPG). This approach aims to identify an efficient offloading strategy, thereby reducing energy consumption. Fortunately, algorithm is proposed for each MT, which is responsible for making decisions regarding task partition, channel allocation, and power transmission control. Our developed approach achieves the ultimate estimation of observed values and enhances memory via write operations. The replay buffer holds data from previous time slots to upgrade both the actor and critic networks, followed by a buffer reset. Comprehensive experiments validate the superior performance, including stability and convergence, of our algorithm when juxtaposed with prior studies.

Nonlinear Reinforcement Learning-Based Dynamic Test Case Prioritization with Anomaly Detection for Continuous Integration

This research article introduces DynamicR-TCP, a nonlinear approach to dynamic test case prioritization (TCP) leveraging reinforcement learning and anomaly detection. Designed to enhance the efficiency of software testing in continuous integration (CI) environments, the proposed model dynamically adapts to changing conditions by learning complex patterns from historical test data. A reinforcement learning agent, employing policy and value networks, guides nonlinear prioritization by optimizing the sequence of test case executions. The model integrates a sliding window strategy for adaptive focus on smaller test subsets and uses an experience replay buffer for iterative learning. An anomaly detection layer monitors test outcomes for deviations, triggering retraining when needed to maintain reliability. Experimental evaluation using the Defects4J dataset demonstrates the model’s superior fault detection rates, faster time to first fault, and higher computational efficiency compared to contemporary methods. An ablation study highlights the nonlinear synergy among the key components—reinforcement learning, sliding windows, and anomaly detection—showing significant performance gains when combined. This comprehensive framework advances software testing by providing an adaptive, efficient, and scalable solution, ensuring robust performance in dynamic CI environments.

Continual learning with high-order experience replay for dynamic network embedding

Dynamic network embedding (DNE) poses a tough challenge in graph representation learning, especially when confronted with the frequent updates of streaming data. Conventional DNEs primarily resort to parameter updating but perform inadequately on historical networks, resulting in the problem of catastrophic forgetting. To tackle such issues, recent advancements in graph neural networks (GNNs) have explored matrix factorization techniques. However, these approaches encounter difficulties in preserving the global patterns of incremental data. In this paper, we propose CLDNE, a Continual Learning framework specifically designed for Dynamic Network Embedding. At the core of CLDNE lies a streaming graph auto-encoder that effectively captures both global and local patterns of the input graph. To further overcome catastrophic forgetting, CLDNE is equipped with an experience replay buffer and a knowledge distillation module, which preserve high-order historical topology and static historical patterns. We conduct experiments on four dynamic networks using link prediction and node classification tasks to evaluate the effectiveness of CLDNE. The outcomes demonstrate that CLDNE successfully mitigates the catastrophic forgetting problem and reduces training time by 80% without a significant loss in learning new patterns.

TD3 Algorithm of Dynamic Classification Replay Buffer Based PID Parameter Optimization

TD3 Algorithm of Dynamic Classification Replay Buffer Based PID Parameter Optimization

GWO-ANFIS-RD3PG: A reinforcement learning approach with dynamic adjustment and dual replay mechanism for building energy forecasting

Reliable prediction of building energy consumption is essential for effective and sustainable energy management. Traditional forecasting methods, however, face challenges when dealing with sudden changes in energy consumption. To address the above-mentioned issue, we introduce the GWO-ANFIS-RD3PG prediction model in this paper. This model aims to ensure global forecasting accuracy while minimizing prediction errors at sudden change points. A Grey Wolf Optimization (GWO) algorithm with the Adaptive Neuro-Fuzzy Inference System (ANFIS) is proposed to predict the energy consumption type and its fluctuation magnitude for the next time step. Within the Recurrent Dual Experience Replay Buffer DDPG (RD3PG) prediction module, we employ Long Short-Term Memory (LSTM), as the actor network in the Deep Deterministic Policy Gradient (DDPG), to account for long-term dependencies in time series data. Notably, we introduce an adaptive action modification mechanism that allows the agent to optimize actions based on the output of the aforementioned ANFIS, enabling real-time adjustments during sudden change. To further enhance the model’s performance, we adopt a dual experience replay mechanism to store data samples from sudden change points, capturing insight information during these anomalies. Experimental results on two real-world datasets demonstrate that this method reduces MAE by 5.81% and 13.47%, and RMSE by 5.69% and 15.21% compared to the state-of-art models, showing the significance of the proposed method for energy efficiency improvement in real-world building operations.

A joint optimization of resource allocation management and multi-task offloading in high-mobility vehicular multi-access edge computing networks

Vehicular communications have advanced data exchange and real-time services in intelligent transportation systems by exploiting advanced communication between vehicles and infrastructure. The emergence of Multi-access Edge Computing (MEC) has further elevated this field by utilizing distributed edge resources near vehicles for low-latency data processing and high-reliability communication. In this dynamic environment, adequate resource allocation and task offloading are pivotal to ensure superior performance, lower latency, and efficient network resource utilization, enhancing Quality of Service (QoS) and overall driving experience and safety. This paper presents a developed vehicular network and offloading mechanism, introducing a resource management model with real-time allocation and load balancing. The proposed method integrates task prioritization, multi-agent collaboration, context-aware decision-making, and distributed learning to optimize network performance. The introduced optimized algorithm initializes Q-networks and target networks, sets up an experience replay buffer, and configures agents with local state representations. Agents use an ε-greedy policy for action selection, update Q-values through experience replay, and prioritize tasks based on urgency while sharing state information for collaborative decision-making. Evaluations through simulation demonstrate optimized performance, enhancing efficiency in vehicular MEC networks compared to baseline and the other well-known algorithms.

EETS: An energy-efficient task scheduler in cloud computing based on improved DQN algorithm

The huge energy consumption of data centers in cloud computing leads to increased operating costs and high carbon emissions to the environment. Deep Reinforcement Learning (DRL) technology combines of deep learning and reinforcement learning, which has an obvious advantage in solving complex task scheduling problems. Deep Q Network(DQN)-based task scheduling has been employed for objective optimization. However, training the DQN algorithm may result in value overestimation, which can negatively impact the learning effectiveness. The replay buffer technique, while increasing sample utilization, does not distinguish between sample importance, resulting in limited utilization of valuable samples. This study proposes an enhanced task scheduling algorithm based on the DQN framework, which utilizes a more optimized Dueling-network architecture as well as Double DQN strategy to alleviate the overestimation bias and address the shortcomings of DQN. It also incorporates a prioritized experience replay technique to achieve importance sampling of experience data, which overcomes the problem of low utilization due to uniform sampling from replay memory. Based on these improved techniques, we developed an energy-efficient task scheduling algorithm called EETS (Energy-Efficient Task Scheduling). This algorithm automatically learns the optimal scheduling policy from historical data while interacting with the environment. Experimental results demonstrate that EETS exhibits faster convergence rates and higher rewards compared to both DQN and DDQN. In scheduling performance, EETS outperforms other baseline algorithms in key metrics, including energy consumption, average task response time, and average machine working time. Particularly, it has a significant advantage when handling large batches of tasks.

Human-Inspired Meta-Reinforcement Learning Using Bayesian Knowledge and Enhanced Deep Q-Network

Over the last decades, there has been growing interest in research in multiple and interdisciplinary fields of human-AI computing. In particular, approaches integrating human’s perspective and design with reinforcement learning (RL) have received more attention. However, the current research on RL may need to consider its enhancement from human-inspired approaches further. In this work, we focus on enabling a meta-reinforcement learning (meta-RL) agent to achieve adaptation and generalization, according to modeling Markov Decision Processes (MDP) using Bayesian knowledge and analysis. By introducing a novel framework called human-inspired meta-RL (HMRL), we incorporate the agent performing resilient actions to leverage the dynamic dense reward based on the knowledge and prediction of a Bayesian analysis. The proposed framework can make the agent learn generalization and prevent the agent from failing catastrophically. The experimental results show that our approach helps the agent reduce computational costs with learning adaptation. In addition to the system design, we have also extended further algorithmic improvement based on learning within a deep Q-network (DQN) implementations for more complicated future tasks, which compared replay buffers to possibly enhance the optimization process. Finally, we conclude and anticipate that integrating human-inspired meta-RL can enable learning more formulations relating to robustness and scalability, leading to promising directions and more complex AI goals in the future.

VN-MADDPG: A Variable-Noise-Based Multi-Agent Reinforcement Learning Algorithm for Autonomous Vehicles at Unsignalized Intersections

The decision-making performance of autonomous vehicles tends to be unstable at unsignalized intersections, making it difficult for them to make optimal decisions. We propose a decision-making model based on the Variable-Noise Multi-Agent Deep Deterministic Policy Gradient (VN-MADDPG) algorithm to address these issues. The variable-noise mechanism reduces noise dynamically, enabling the agent to utilize the learned policy more effectively to complete tasks. This significantly improves the stability of the decision-making model in making optimal decisions. The importance sampling module addresses the inconsistency between outdated experience in the replay buffer and current environmental features. This enhances the model’s learning efficiency and improves the robustness of the decision-making model. Experimental results on the CARLA simulation platform show that the success rate of decision making at unsignalized intersections by autonomous vehicles has significantly increased, and the pass time has been reduced. The decision-making model based on the VN-MADDPG algorithm demonstrates stable and excellent decision-making performance.

Learning Multi-Agent Cooperation via Considering Actions of Teammates.

Recently value-based centralized training with decentralized execution (CTDE) multi-agent reinforcement learning (MARL) methods have achieved excellent performance in cooperative tasks. However, the most representative method among these methods, Q-network MIXing (QMIX), restricts the joint action Q values to be a monotonic mixing of each agent's utilities. Furthermore, current methods cannot generalize to unseen environments or different agent configurations, which is known as ad hoc team play situation. In this work, we propose a novel Q values decomposition that considers both the return of an agent acting on its own and cooperating with other observable agents to address the nonmonotonic problem. Based on the decomposition, we propose a greedy action searching method that can improve exploration and is not affected by changes in observable agents or changes in the order of agents' actions. In this way, our method can adapt to ad hoc team play situation. Furthermore, we utilize an auxiliary loss related to environmental cognition consistency and a modified prioritized experience replay (PER) buffer to assist training. Our extensive experimental results show that our method achieves significant performance improvements in both challenging monotonic and nonmonotonic domains, and can handle the ad hoc team play situation perfectly.

Reinforcement learning-based satellite formation attitude control under multi-constraint

As the complexity of space missions increases, the constraints on satellite attitude control become more stringent, particularly for satellites working in orbit formation. This paper introduces a novel method, based on the categorization and modeling of different constraints, for attitude control of satellite formations under multiple constraints. The method employs the Phased Priority Reinforcement Learning (PPRL) approach, which utilizes Deep Deterministic Policy Gradient (DDPG) technology. Considering the complexity of constraints and the challenge posed by the high control dimensionality due to multi-satellite coordination, the method addresses these challenges through a two-step training strategy. The first step addresses the multi-constraint issue for individual satellites and increases the priority of single-satellite training experience data in the experience replay buffer of the second step to enhance data utilization efficiency. To address the issue of reward sparsity in complex high-dimensional constraint models, a detailed reward mechanism is proposed, incorporating both local and global constraints into the reward function, thereby achieving both efficient and effective attitude control. This approach not only meets dynamic, state, and performance constraints but also demonstrates adaptability and robustness through numerical simulations. Compared to traditional methods, this approach achieves significant improvements in control performance and constraint satisfaction, offering a novel solution pathway for high-dimensional control problems in multi-constraint satellite formations.