Experience Replay Research Articles

Multi-objective reinforcement learning-based energy management for fuel cell vehicles considering lifecycle costs

To balance the hydrogen consumption of fuel cell vehicle (FCV), the durability of the fuel cell (FC), and the life of the power battery (PB) to further reduce the whole lifecycle costs of FCV. A multi-objective reinforcement learning-based (MORL-based) energy management strategy (EMS) is proposed in this research. First, the composition mechanism of the FCV lifecycle costs is analyzed, and the equivalent hydrogen consumption model, FC durability degradation model, and PB life decay model are established; Then, a three-dimensional reward function is constructed by integrating the objectives of equivalent hydrogen consumption, FC durability degradation, and PB life decay. And the penalty terms coupled with the decay factors are introduced into the reward function to satisfy the mutual constraint characteristics between the PB and the FC system to ensure the stability of the MORL-based EMS; In addition, the prioritized experience replay technology is introduced into the MORL-based EMS to improve the learning efficiency and convergence of traditional deep Q network (DQN) algorithm; After that, the evaluation and target network of the embedded dueling network are introduced to solve the multi-objective overestimation problem encountered in the training process by generalizing the behavior learning in the presence of similar value behaviors; Finally, the performance of MORL-based EMS and DQN-based EMS is compared by numerical simulation under various driving cycles. The results show that the MORL-based EMS proposed in this paper has better convergence ability, adaptability, and lower lifecycle costs than the DQN-based EMS. In addition, the lifecycle costs of the MORL-based EMS can achieve a 99.2% control effect of the dynamic programming-based EMS.

Map-based experience replay: a memory-efficient solution to catastrophic forgetting in reinforcement learning.

Deep reinforcement learning (RL) agents often suffer from catastrophic forgetting, forgetting previously found solutions in parts of the input space when training new data. Replay memories are a common solution to the problem by decorrelating and shuffling old and new training samples. They naively store state transitions as they arrive, without regard for redundancy. We introduce a novel cognitive-inspired replay memory approach based on the Grow-When-Required (GWR) self-organizing network, which resembles a map-based mental model of the world. Our approach organizes stored transitions into a concise environment-model-like network of state nodes and transition edges, merging similar samples to reduce the memory size and increase pair-wise distance among samples, which increases the relevancy of each sample. Overall, our study shows that map-based experience replay allows for significant memory reduction with only small decreases in performance.

DisGUIDE: Disagreement-Guided Data-Free Model Extraction

Recent model-extraction attacks on Machine Learning as a Service (MLaaS) systems have moved towards data-free approaches, showing the feasibility of stealing models trained with difficult-to-access data. However, these attacks are ineffective or limited due to the low accuracy of extracted models and the high number of queries to the models under attack. The high query cost makes such techniques infeasible for online MLaaS systems that charge per query. We create a novel approach to get higher accuracy and query efficiency than prior data-free model extraction techniques. Specifically, we introduce a novel generator training scheme that maximizes the disagreement loss between two clone models that attempt to copy the model under attack. This loss, combined with diversity loss and experience replay, enables the generator to produce better instances to train the clone models. Our evaluation on popular datasets CIFAR-10 and CIFAR-100 shows that our approach improves the final model accuracy by up to 3.42% and 18.48% respectively. The average number of queries required to achieve the accuracy of the prior state of the art is reduced by up to 64.95%. We hope this will promote future work on feasible data-free model extraction and defenses against such attacks.

Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning

Offline multi-agent reinforcement learning (MARL) aims to learn effective multi-agent policies from pre-collected datasets, which is an important step toward the deployment of multi-agent systems in real-world applications. However, in practice, each individual behavior policy that generates multi-agent joint trajectories usually has a different level of how well it performs. e.g., an agent is a random policy while other agents are medium policies. In the cooperative game with global reward, one agent learned by existing offline MARL often inherits this random policy, jeopardizing the utility of the entire team. In this paper, we investigate offline MARL with explicit consideration on the diversity of agent-wise trajectories and propose a novel framework called Shared Individual Trajectories (SIT) to address this problem. Specifically, an attention-based reward decomposition network assigns the credit to each agent through a differentiable key-value memory mechanism in an offline manner. These decomposed credits are then used to reconstruct the joint offline datasets into prioritized experience replay with individual trajectories, thereafter agents can share their good trajectories and conservatively train their policies with a graph attention network (GAT) based critic. We evaluate our method in both discrete control (i.e., StarCraft II and multi-agent particle environment) and continuous control (i.e., multi-agent mujoco). The results indicate that our method achieves significantly better results in complex and mixed offline multi-agent datasets, especially when the difference of data quality between individual trajectories is large.

Finite‐time optimal tracking control for a class of nonlinear systems with prescribed performance

AbstractIn this paper, a finite‐time optimal tracking control scheme based on integral reinforcement learning is developed for partially unknown nonlinear systems. In order to realize the prescribed performance, the original system is transformed into an equivalent unconstrained system so as to a composite system is constructed. Subsequently, a modified nonlinear quadratic performance function containing the auxiliary tracking error is designed. Furthermore, the technique of experience replay is used to update the critic neural network, which eliminates the persistent of excitation condition in traditional optimal methods. By combining the prescribed performance control with the finite‐time optimization control technique, the tracking error is driven to a desired performance in finite time. Consequently, it has been shown that all signals in the partially unknown nonlinear system are semiglobally practical finite‐time stable by stability analysis. Finally, the provided comparative simulation results verify the effectiveness of the developed control scheme.

Optimal navigation for AGVs: A soft actor–critic-based reinforcement learning approach with composite auxiliary rewards

In this paper, we address the problem of real-time navigation and obstacle avoidance for automated guided vehicles (AGVs) in dynamic environments, which is a primary research area in collaborative control systems for AGVs. To overcome the computational inefficiency of recalculating optimal paths every time, we propose an improved Soft Actor–Critic (SAC)-based reinforcement learning methodology. This methodology utilizes a novel composite auxiliary reward structure and sum-tree prioritized experience replay (SAC-SP) to achieve real-time optimal feedback control. First, we formulate the navigation task as a Markov Decision Process that considers both static and dynamic obstacles. To accelerate the active learning of AGVs, we propose a novel strategy that uses composite auxiliary rewards. Next, we train the AGVs using the proposed SAC-SP methodology to handle real-time navigation with the composite auxiliary reward structure. The well-trained policy network can generate effective on-board optimal feedback actions given obstacle positions, targets, and AGV states. Simulation experiments demonstrate that our proposed method can steer AGVs to the destination with high robustness to original conditions and various obstacle restrictions, generating optimal feedback actions in the shortest amount of time.

OPTIMAL TRACKING CONTROL FOR ROBOT MANIPULATORS WITH INPUT CONSTRAINT BASED REINFORCEMENT LEARNING

This paper introduces an optimal tracking controller for robot manipulators with saturation torques. The robot model is presented as a strict-feedback nonlinear system. Firstly, the position tracking control problem is transformed into the optimal tracking control problem. Subsequently, the saturated optimal control law is designed. The optimal control law is determined through the solution of the Hamilton-Jacobi-Bellman (HJB) equation. We use a reinforcement learning algorithm with only one neural network (NN) to approximate the solution of the equation HJB. The technique of experience replay is used to relax a persistent citation condition. By Lyapunov analysis, the tracking and the approximation errors are uniformly ultimately bounded (UUB). Finally, the simulation on a robot manipulator with saturation torques is performed to verify the efficiency of the proposed controller.

Deep reinforcement learning for a multi-objective operation in a nuclear power plant

Nuclear power plant (NPP) operations with multiple objectives and devices are still performed manually by operators despite the potential for human error. These operations could be automated to reduce the burden on operators; however, classical approaches may not be suitable for these multi-objective tasks. An alternative approach is deep reinforcement learning (DRL), which has been successful in automating various complex tasks and has been applied in automation of certain operations in NPPs. But despite the recent progress, previous studies using DRL for NPP operations have limitations to handle complex multi-objective operations with multiple devices efficiently. This study proposes a novel DRL-based approach that addresses these limitations by employing a continuous action space and straightforward binary rewards supported by the adoption of a soft actor-critic and hindsight experience replay. The feasibility of the proposed approach was evaluated for controlling the pressure and volume of the reactor coolant while heating the coolant during NPP startup. The results show that the proposed approach can train the agent with a proper strategy for effectively achieving multiple objectives through the control of multiple devices. Moreover, hands-on testing results demonstrate that the trained agent is capable of handling untrained objectives, such as cooldown, with substantial success.

Efficient Multi-Goal Reinforcement Learning via Value Consistency Prioritization

Goal-conditioned reinforcement learning (RL) with sparse rewards remains a challenging problem in deep RL. Hindsight Experience Replay (HER) has been demonstrated to be an effective solution, where HER replaces desired goals in failed experiences with practically achieved states. Existing approaches mainly focus on either exploration or exploitation to improve the performance of HER. From a joint perspective, exploiting specific past experiences can also implicitly drive exploration. Therefore, we concentrate on prioritizing both original and relabeled samples for efficient goal-conditioned RL. To achieve this, we propose a novel value consistency prioritization (VCP) method, where the priority of samples is determined by the consistency of ensemble Q-values. This distinguishes the VCP method with most existing prioritization approaches which prioritizes samples based on the uncertainty of ensemble Q-values. Through extensive experiments, we demonstrate that VCP achieves significantly higher sample efficiency than existing algorithms on a range of challenging goal-conditioned manipulation tasks. We also visualize how VCP prioritizes good experiences to enhance policy learning.

Deep Reinforcement Learning for the Visual Servoing Control of UAVs with FOV Constraint

Visual servoing is a control method that utilizes image feedback to control robot motion, and it has been widely applied in unmanned aerial vehicle (UAV) motion control. However, due to field-of-view (FOV) constraints, visual servoing still faces challenges, such as easy target loss and low control efficiency. To address these issues, visual servoing control for UAVs based on the deep reinforcement learning (DRL) method is proposed, which dynamically adjusts the servo gain in real time to avoid target loss and improve control efficiency. Firstly, a Markov model of visual servoing control for a UAV under field-of-view constraints is established, which consists ofquintuplet and considers the improvement of the control efficiency. Secondly, an improved deep Q-network (DQN) algorithm with a target network and experience replay is designed to solve the Markov model. In addition, two independent agents are designed to adjust the linear and angular velocity servo gains in order to enhance the control performance, respectively. In the simulation environment, the effectiveness of the proposed method was verified using a monocular camera.

Research on adaptive obstacle avoidance algorithm of robot based on DDPG-DWA

In this paper, we propose an adaptive obstacle avoidance algorithm based on DDPG (Deep Deterministic Policy Gradient) and DWA (Dynamic-Window Approach) to study the obstacle avoidance problem of robots in complex continuous state space. First, the obstacle avoidance problem is converted into an optimal learning incentive problem, and the self-learning of the obstacle avoidance policy is realized based on DDPG; second, the DWA obstacle avoidance trajectory evaluation function is optimized using the DDPG reward incentive mechanism, and the Experience Replay mechanism; finally, the algorithm model is simulated. The experiments show that the model can significantly circumvent the deficiency of the DWA algorithm in limiting to the optimal local solution in a complex environment and solve the action output problem in the continuous velocity and turning angle value interval of the robot; through the trial and error interaction with the environment and the trajectory evaluation incentive feedback, the obstacle avoidance passing ability of the robot in a complex environment is improved.

NN-Based Parallel Model Predictive Control for a Quadrotor UAV

A novel neural network (NN)-based parallel model predictive control (PMPC) method is proposed to deal with the tracking problem of the quadrotor unmanned aerial vehicles (Q-UAVs) system in this article. It is well known that the dynamics of Q-UAVs are changeable while the system is operating in some specific environments. In this case, traditional NN-based MPC methods are not applicable because their model networks are pre-trained and kept constant throughout the process. To solve this problem, we propose the PMPC algorithm, which introduces parallel control structure and experience pool replay technology into the MPC method. In this algorithm, an NN-based artificial system runs in parallel with the UAV system to reconstruct its dynamics model. Furthermore, the experience replay technology is used to maintain the accuracy of the reconstructed model, so as to ensure the effectiveness of the model prediction algorithm. Furthermore, a convergence proof of the artificial system is also given in this paper. Finally, numerical results and analysis are given to demonstrate the effectiveness of the PMPC algorithm.

Unbalanced Class-incremental Learning for Text Classification Based on Experience Replay

While deep learning has achieved remarkable results for text classification, incremental learning for text classification is still a challenge. The main problem is that models suffer from catastrophic forgetting, which is they always forget knowledge learned before when labelled data comes sequentially and is trained in sequence. In this study, we propose methods of preventing catastrophic forgetting to handle unbalanced increased data. As an improvement over experience replay, our approaches improve the accuracy about 23.3% with 23% of all training data on Yahoo and 9.5% with 12% of all training data and on DBPedia.

Multi-Input Autonomous Driving Based on Deep Reinforcement Learning With Double Bias Experience Replay

It is still a challenge to realize safe and fast autonomous driving through deep reinforcement learning. Most autonomous driving reinforcement learning models are subject to a single experience replay approach for training agents and how to improve the driving speed and safety of agent has become the focus of research. Therefore, we present an improved Double Bias Experience Replay (DBER) approach, which enables the agent to choose its own driving learning tendency. A new loss function is proposed to ameliorate the relationship between negative loss and positive loss. The proposed approach has been applied to three algorithms to verify: Deep Q Network (DQN), Dueling Double DQN (DD-DQN) and Quantile Regression DQN (QR-DQN). Compared with the existing approaches, the proposed approach show better performance and robustness of driving policy on the driving simulator, which is implemented by the Unity ML-agents. The approach makes the vehicle agent obtain better performance, such as higher reward, faster driving speed, less lane changing and more in the same training time.

Deep Reinforcement Learning Based Link Adaptation Technique for LTE/NR Systems

Outdated channel quality indicator (CQI) feedback causes severe performance degradation of traditional link adaptation (LA) techniques in long term evolution (LTE) and new radio (NR) systems. This paper puts forth a deep reinforcement learning (DRL) based link adaptation (LA) technique, referred to as deep reinforcement learning link adaptation (DRLLA), to select efficient modulation and coding scheme (MCS) in the presence of the outdated CQI feedback. The goal of DRLLA is to maximize the link throughput while achieving a low block error rate (BLER). We first give explicit definitions of state, action, and reward in DRL paradigms, thereby realizing DRLLA. Then, to trade off the throughput against the BLER, we further develop a new experience replay mechanism called classified experience replay (CER) as the underpinning technique in DRLLA. In CER, experiences are separated into two buckets, one for successful experiences and the other for failed experiences, and then a fixed proportion from each is sampled to replay. The essence of CER is to obtain different trade-offs via adjusting the proportion among different training experiences. Furthermore, to reduce the signaling overhead and the system reconfiguration cost caused by frequent MCS switching, we propose a new action selection strategy termed as switching-controlled <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> -greedy (SC- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> -greedy) for DRLLA. Simulation results demonstrate that compared with the state-of-the-art OLLA, LTSLA, and DRLLA with other experience replay mechanisms, DRLLA with CER can achieve higher throughput and lower BLER in various time-varying scenarios, and be more robust to different CQI feedback delays and CQI reporting periods. Furthermore, with the SC- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> -greedy policy, DRLLA can capture better trade-offs between the link transmission quality and the MCS switching overhead compared with other baselines.

Deep reinforcement learning-based model-free path planning and collision avoidance for UAVs: A soft actor–critic with hindsight experience replay approach

In this paper, we propose a soft actor–critic (SAC) algorithm with hindsight experience replay (HER), called SACHER, which is a class of deep reinforcement learning (DRL) algorithm. SAC is an off-policy model-free DRL algorithm that outperforms earlier DRL algorithms in terms of exploration and robustness. However, in SAC, maximizing the entropy-augmented objective degrades the optimality of learning outcomes. We propose SACHER to improve the learning performance of SAC. We apply SACHER to the path planning and collision avoidance control of unmanned aerial vehicles (UAVs). We demonstrate the effectiveness of SACHER in terms of the success rate, learning speed, and collision avoidance performance of UAV operation.

Resilience Assessment for Power Systems Under Sequential Attacks Using Double DQN With Improved Prioritized Experience Replay

Resilience Assessment for Power Systems Under Sequential Attacks Using Double DQN With Improved Prioritized Experience Replay

Reinforcement Learning for Pass Detection and Generation of Possession Statistics in Soccer

We propose a reinforcement learning (RL) based technique to detect passes from the video of a soccer match. The detection of passes determines ball possession statistics of a soccer match. A sequence of video frames is mapped to a sequence of states, such as ball with team A or team B or ball not possessed either by team A or B. The agent of RL learns the frame-to-state mapping and the optimal policy to decide the mapping task. We propose a novel reward function by utilizing contextual information of the soccer game in order to help the agent decide the optimal policy. In this context, the advantage of RL is in the integration of a reward system in choosing an action that maps a video frame of a soccer match to one of three possible states. Unlike competing methods, we design the RL model in a way so that explicit identification of team labels of players is not required. We introduce a Deep Recurrent Q-Network (DRQN) to learn the optimal policy. For efficient training of the DRQN, we have proposed de-correlated experience replay (DER), a strategy that selects important experiences based on the correlations of the experiences stored in the replay memory. Experimental results show that at least 5.75% and 2.1% better accuracy are achieved in calculating pass detection and possession statistics, respectively, compared to similar approaches.

Target Tracking Control of UAV Through Deep Reinforcement Learning

This study presents an innovative reinforcement-learning-based control algorithm for a vertical take-off and landing (VTOL) aircraft under wind disturbances. In our approach, the tracking control problem of the VTOL aircraft is formulated as a Markov decision process, and the appropriate system state, reward function, and soft update method are presented. To improve the control accuracy under wind disturbances, three kinds of wind fields were added in the learning environment to expand the exploration space and simulate the effect of wind disturbances on the flight control. Moreover, to ensure the tracking accuracy and the practical implementation, a quantum-inspired experience replay strategy was developed based on quantum computation theory. In this strategy, the preparation operation scheme was designed to encourage the exploration and speed up the convergence. The depreciation operation method was developed to enrich the sample diversity, which increased the robustness of the controller and allowed the control strategy learned in the numerical simulations to be directly transferred into real-world environments. Numerical simulations, hardware-in-the-loop experiments, and real-world flight experiments were conducted to evaluate the performance and merits of the proposed method. The results demonstrated high accuracy and effectiveness and good robustness of the proposed control algorithm in terms of standoff target tracking control and flight stability.

NFSP-PLT: Solving Games with a Weighted NFSP-PER-Based Method

Nash equilibrium strategy is a typical goal when solving two-player imperfect-information games (IIGs). Neural fictitious self-play (NFSP) is a popular method to find the Nash equilibrium in IIGs, which is the first end-to-end method used to compute the Nash equilibrium strategy. However, the training of NFSP requires a large number of sample data and the interactive cost of obtaining such data is often very high. Realizing the efficient training of network under limited samples is an urgent problem. In this paper, we first proposed a new NFSP-based method, NFSP with prioritized experience replay (NFSP-PER), to improve the sample training efficiency. Then, a weighted NFSP-PER with learning time (NFSP-PLT) was proposed to control the utilization degree of priority-weighted samples. Furthermore, based on the NFSP-PLT, an adaptive upper-confidence-bound applied to tree (UCT) is used to solve the optimal response strategy, which makes the solving strategy more accurate. Extensive experimental results show that the proposed NFSP-PLT effectively improves the sample learning efficiency compared with the existing works.