Replay Mechanism Research Articles

Unmanned Aerial Vehicle Path Planning Algorithm Based on Deep Reinforcement Learning in Large-Scale and Dynamic Environments

Path planning is one of the key technologies for autonomous flight of Unmanned Aerial Vehicle. Traditional path planning algorithms have some limitations and deficiencies in the complex and dynamic environment. In this article, we propose a deep reinforcement learning approach for three-dimensional path planning by utilizing the local information and relative distance without global information. UAV can obtain the limited environmental information nearby in the actual scenario with limited sensor capabilities. Therefore, path planning can be formulated as a Partially Observable Markov Decision Process. The recurrent neural network with temporal memory is constructed to address the partial observability problem by extracting crucial information from historical state-action sequences. We develop an action selection strategy that combines the current reward value and the state-action value to reduce the meaningless exploration. In addition, we construct two sample memory pools and propose an adaptive experience replay mechanism based on the frequency of failure. The simulation experiment results show that our method has significant improvements over Deep Q-Network and Deep Recurrent Q-Network in terms of stability and learning efficiency. Our approach successfully plans a reasonable three-dimensional path in the large-scale and complex environment, and has the perfect ability to avoid obstacles.in the unknown environment.

Deep Deterministic Policy Gradient Based on Double Network Prioritized Experience Replay

The traditional deep deterministic policy gradient (DDPG) algorithm has the disadvantages of slow convergence velocity and ease of falling into the local optimum. From these two perspectives, a DDPG algorithm based on the double network prioritized experience replay mechanism (DNPER-DDPG) is proposed in this paper. Firstly, the value function is approximated by introducing the idea of two neural networks, and the minimum of the action value functions generated by the two networks is selected as the updated value of the actor policy network, which reduces the incidence of local optimal policy. Then, the Q values obtained by the two networks and the immediate reward obtained by the environment are used as the criteria for prioritization, and the importance of the samples in the experience replay mechanism is divided to improve the convergence speed of the algorithm. Finally, the improved method is demonstrated in the classic control environment of OpenAI Gym, and the results show that the proposed method has increased convergence speed and cumulative reward compared with the comparison algorithm.

Adaptive Controller of PEMFC Output Voltage Based on Ambient Intelligence Large-Scale Deep Reinforcement Learning

In this article, an adaptive Proportion integration (PI) controller for varying the output voltage of a proton exchange membrane fuel cell (PEMFC) is proposed. The PI controller operates on the basis of ambient intelligence large-scale deep reinforcement learning. It functions as a coefficient tuner based on an ambient intelligence exploration multi-delay deep deterministic policy gradient (AIEM-DDPG) algorithm. This algorithm is an improvement on the original deep deterministic police gradient (DDPG) algorithm, which incorporates ambient intelligence exploration. The DDPG algorithm serves as the core, and the AIEM-DDPG algorithm runs on a variety of deep reinforcement learning algorithms, including soft actor-critic (SAC), deep deterministic policy gradient (DDPG), proximal policy optimization (PPO) and double deep Q-network (DDQN) algorithms, to attain distributed exploration in the environment. In addition, a classified priority experience replay mechanism is introduced to improve the exploration efficiency. Clipping multi-Q learning, policy delayed updating, target policy smooth regularization and other methods are utilized to solve the problem of Q-value overestimation. A model-free algorithm with good global searching ability and optimization speed is demonstrated. Simulation results show that the AIEM-DDPG adaptive PI controller attains better robustness and adaptability, as well as a good control effect.

RRPOT: A Record and Replay Based Honeypot System

The honeypot system is a critical defense facility in cybersecurity. It captures and analyzes the attack behavior for constructing a more robust defense system. However, considering the overhead caused by analysis, the system cannot support fine-grained analysis. This paper proposes RRPOT, a novel honeypot system based on the record and replay (RnR) framework of virtual machine (VM). By combining the native honeypot system with an RnR framework, the system supports to postpone the time-consuming analysis to the replay stage, thereby enables fine-grained analysis. Moreover, we propose an on-demand record mechanism to reduce performance and storage costs during the record stage and employ an on-demand replay mechanism to improve replay efficiency. Leveraging these approaches, we develop a flexible and efficient honeypot system, and it was proven useful in the practical scenario.

The impact of copycat attack on RPL based 6LoWPAN networks in Internet of Things

IPv6 Routing Protocol for Low-Power and Lossy Networks (RPL) is the standard network layer protocol for achieving efficient routing in IPv6 over Low-Power Wireless Personal Area Networks (6LoWPAN). Resource-constrained and non-tamper resistant nature of smart sensor nodes makes RPL protocol susceptible to different threats. An attacker may use insider or outsider attack strategy to perform Denial-of-Service (DoS) attacks against RPL based networks. Security and Privacy risks associated with RPL protocol may limit its global adoption and worldwide acceptance. A proper investigation of RPL specific attacks and their impacts on an underlying network needs to be done. In this paper, we present and investigate one of the catastrophic attacks named as a copycat attack, a type of replay based DoS attack against the RPL protocol. An in-depth experimental study for analyzing the impacts of the copycat attack on RPL has been done. The experimental results show that the copycat attack can significantly degrade network performance in terms of packet delivery ratio, average end-to-end delay, and average power consumption. To the best of our knowledge, this is the first paper that extensively studies the impact of RPL specific replay mechanism based DoS attack on 6LoWPAN networks.

Logistics-involved QoS-aware service composition in cloud manufacturing with deep reinforcement learning

• Deep reinforcement learning is used to solve the QoS-aware service composition problem in cloud manufacturing with the consideration of logistics. • A DRL algorithm that combines the basic DQN algorithm, the dueling architecture, and the prioritized replay mechanism is proposed. • Experimental results demonstrate the effectiveness, high efficiency, adaptability, and scalability of the proposed DRL algorithm. Cloud manufacturing is a new manufacturing model that aims to provide on-demand manufacturing services to consumers over the Internet. Service composition is an essential issue as well as an important technique in cloud manufacturing (CMfg) that supports construction of larger-granularity, value-added services by combining a number of smaller-granularity services to satisfy consumers’ complex requirements. Meta-heuristics algorithms such as genetic algorithm, particle swarm optimization, and ant colony algorithm are frequently employed for addressing service composition issues in cloud manufacturing. These algorithms, however, require complex design flows and painstaking parameter tuning, and lack adaptability to dynamic environment. Deep reinforcement learning (DRL) provides an alternative approach for solving cloud manufacturing service composition (CMfg-SC) issues. DRL as model-free artificial intelligent methods enables a system to learn optimal service composition solutions through training, which can therefore circumvent the aforementioned problems with meta-heuristics algorithms. This paper is dedicated to exploring possible applications of DRL in CMfg-SC. A logistics-involved QoS-aware DRL-based CMfg-SC is proposed. A dueling Deep Q-Network (DQN) with prioritized replay named PD-DQN is designed as the DRL algorithm. Effectiveness, robustness, adaptability, and scalability of PD-DQN are investigated, and compared with that of the basic DQN and Q-learning. Experimental results indicate that PD-DQN is able to effectively address the CMfg-SC problem.

Episodic memory retrieval success is associated with rapid replay of episode content.

Retrieval of everyday experiences is fundamental for informing our future decisions. The fine-grained neurophysiological mechanisms that support such memory retrieval are largely unknown. We studied participants who first experienced, without repetition, unique multicomponent 40-80-s episodes. One day later, they engaged in cued retrieval of these episodes while undergoing magnetoencephalography. By decoding individual episode elements, we found that trial-by-trial successful retrieval was supported by the sequential replay of episode elements, with a temporal compression factor of >60. The direction of replay supporting retrieval, either backward or forward, depended on whether the task goal was to retrieve elements of an episode that followed or preceded, respectively, a retrieval cue. This sequential replay was weaker in very-high-performing participants, in whom instead we found evidence for simultaneous clustered reactivation. Our results demonstrate that memory-mediated decisions are supported by a rapid replay mechanism that can flexibly shift in direction in response to task goals.

Behavioral evidence for memory replay of video episodes in the macaque.

Humans recall the past by replaying fragments of events temporally. Here, we demonstrate a similar effect in macaques. We trained six rhesus monkeys with a temporal-order judgement (TOJ) task and collected 5000 TOJ trials. In each trial, the monkeys watched a naturalistic video of about 10 s comprising two across-context clips, and after a 2 s delay, performed TOJ between two frames from the video. The data are suggestive of a non-linear, time-compressed forward memory replay mechanism in the macaque. In contrast with humans, such compression of replay is, however, not sophisticated enough to allow these monkeys to skip over irrelevant information by compressing the encoded video globally. We also reveal that the monkeys detect event contextual boundaries, and that such detection facilitates recall by increasing the rate of information accumulation. Demonstration of a time-compressed, forward replay-like pattern in the macaque provides insights into the evolution of episodic memory in our lineage.

Combination of Amplitude and Frequency Modulation Features for Presentation Attack Detection

In this paper, we propose the combination of Amplitude Modulation and Frequency Modulation (AM-FM) features for replay Spoof Speech Detection (SSD) task. The AM components are known to be affected by noise (in this case, due to replay mechanism). In particular, we exploit this damage in AM component to corresponding Instantaneous Frequency (IF) for SSD task. Thus, the novelty of proposed Amplitude Weighted Frequency Cepstral Coefficients (AWFCC) feature set lies in using frequency components along with squared weighted amplitude components that are degraded due to replay noise. The AWFCC feature set contains the information of both AM and FM components together and hence, gave discriminatory information in the spectral characteristics. The experiments were performed on publicly available ASVspoof 2017 challenge version 1.0 and 2.0 databases using AWFCC feature set. We have compared results of proposed feature set with the other state-of-the-art feature set, such as Constant Q Cepstral Coefficients (CQCC), Linear Frequency Cepstral Coefficients (LFCC), Mel Frequency Cepstral Coefficients (MFCC) and using a simple Gaussian Mixture Model (GMM) classifier. The individual performance of AWFCC feature set obtained lower % EER than the other feature sets on both version 1.0 and 2.0 databases. Furthermore, we used score-level fusion in order to obtain the possible complementary information of two feature sets to reduce the % EER further. To that effect, the score-level fusion of CQCC and AWFCC feature sets gave 5.75 % and 10.42 % EER on development and evaluation sets, respectively, of ASVspoof 2017 version 2.0 database. Moreover, for evaluation dataset, we have also studied the performance of proposed feature set on different Replay Configurations (RC), namely, acoustic environments, playback, and recording devices. For all the levels of threat conditions (i.e., low, medium, and high) to the ASV system, the proposed feature set performed better compared to the existing state-of-the-art feature sets.

RETRACTED ARTICLE: Solving the protein folding problem in hydrophobic-polar model using deep reinforcement learning

The present article focuses on solving the protein folding problem with deep reinforcement learning (DRL) approach. The protein folding problem is an NP-hard problem and as we are proposing our approach in the hydrophobic-polar model, we deal with an NP-complete problem. Also, the protein folding problem is a combinatorial optimization problem. Combinatorial optimization problems are hard to solve optimally, that is why any attempt to improve their solutions is beneficent. Generally, this problem refers to the process of predicting the structure of a protein from its amino acids sequence. During recent years, the protein folding problem has attracted a lot of attention. The amount of time and expenses of using nuclear magnetic resonance imaging and crystallography for identifying the three-dimensional structure is the main reason of many proposed approaches. In this study, our approach models the problem as a DRL problem, and for enhancing its performance, we adopt long short-term memory networks for the approximation phase in the reinforcement learning algorithm. Using deep Q-learning approach and actor–critic algorithm with an experience replay mechanism overcomes the complexity of other proposed approaches which leads to better accuracy in less time. In addition, we analyzed the efficiency and effectiveness of the dueling deep Q-network technique for solving the protein folding problem. Providing a step-by-step implementation and modeling for solving the bi-dimensional protein folding problem with the DRL approach is the purpose of the present study which could be helpful for solving other omics and computational biology problems. However, a comparison between the DRL approach and other notable approaches (as it is available in Sect. 6) shows that our approach outperforms other approaches in finding the minimum value of the free energy, which is the main factor in the protein folding problem, in less time in any available case.

Multi-Robot Flocking Control Based on Deep Reinforcement Learning

In this paper, we apply deep reinforcement learning (DRL) to solve the flocking control problem of multi-robot systems in complex environments with dynamic obstacles. Starting from the traditional flocking model, we propose a DRL framework for implementing multi-robot flocking control, eliminating the tedious work of modeling and control designing. We adopt the multi-agent deep deterministic policy gradient (MADDPG) algorithm, which additionally uses the information of multiple robots in the learning process to better predict the actions that robots will take. To address the problems such as low learning efficiency and slow convergence speed of the MADDPG algorithm, this paper studies a prioritized experience replay (PER) mechanism and proposes the Prioritized Experience Replay-MADDPG (PER-MADDPG) algorithm. Based on the temporal difference (TD) error, a priority evaluation function is designed to determine which experiences are sampled preferentially from the replay buffer. In the end, the simulation results verify the effectiveness of the proposed algorithm. It has a faster convergence speed and enables the robot group to complete the flocking task in the environment with obstacles.

Efficient Exploration in Side-Scrolling Video Games with Trajectory Replay

Deep Reinforcement learning agent has outperformed human players in many games, such as the Atari 2600 games. In more complicated games, previous related works proposed a curiosity-driven exploration for learning. Nevertheless, it generally requires substantial computational resources to train the agent. We attempt to design a method to assist with our agent to explore the environment. By utilizing prior learned experience more effectively, we develop a new memory replay mechanism, which consists of two modules: Trajectory Replay Module to record the agent moving trajectory information with much less space, and the Trajectory Optimization Module to formulate the state information as a reward. We evaluate our approach with two popular side-scrolling video games: Super Mario Bros and Sonic the Hedgehog. The experiment results show that our method can help the agent explore the environment efficiently, pass through various tough scenarios and successfully reach the goal in most of the testing game levels with merely four workers and of ordinary CPU computational resources for training. The demo videos are at Super Mario Bros and Sonic the Hedgehog.

Heuristic Q-learning based on experience replay for three-dimensional path planning of the unmanned aerial vehicle.

In order to solve the problem that the existing reinforcement learning algorithm is difficult to converge due to the excessive state space of the three-dimensional path planning of the unmanned aerial vehicle, this article proposes a reinforcement learning algorithm based on the heuristic function and the maximum average reward value of the experience replay mechanism. The knowledge of track performance is introduced to construct heuristic function to guide the unmanned aerial vehicles' action selection and reduce the useless exploration. Experience replay mechanism based on maximum average reward increases the utilization rate of excellent samples and the convergence speed of the algorithm. The simulation results show that the proposed three-dimensional path planning algorithm has good learning efficiency, and the convergence speed and training performance are significantly improved.

Human-Like Time-Compressed Forward Replay of Video Episodes in Macaque Monkeys

Humans can perform episodic memory replay dynamically by replaying fragments of fine-grained temporal patterns of events and skipping flexibly across subevents. Here, we demonstrate a similar effect in macaque monkeys. We report evidence that the monkeys apply a time-compressed, forward replay mechanism during their judgement the order of cinematic events. We trained six macaque monkeys with a temporal order judgement (TOJ) task and collected 5000 TOJ trials with naturalistic videos. In each trial, they watched a video of about 10 s comprising two across-context clips, and after a 2-s retention delay, performed a temporal order judgement between two frames extracted from the video. The results show that the monkeys adopt a self-terminating search of ordered frames in their mnemonic representation. The memory replay is forward and temporally compressed, paralleling evidence in humans. Such compression of replay is however not sophisticated enough to allow them to skip over irrelevant information by compressing the encoded video globally. Moreover, we also reveal that the monkeys can segment events using contextual boundaries like humans and such contextual segmentation facilitates their memory recall by an increased rate of information accumulation in a drift diffusion model framework. Memory replay is an elaborate mental process and our demonstration of a time-compressed, forward replay mechanism in the macaque monkeys provides insights into mapping the mechanisms and evolution of episodic memory in our lineage.

Toward a Brain-Inspired System: Deep Recurrent Reinforcement Learning for a Simulated Self-Driving Agent.

An effective way to achieve intelligence is to simulate various intelligent behaviors in the human brain. In recent years, bio-inspired learning methods have emerged, and they are different from the classical mathematical programming principle. From the perspective of brain inspiration, reinforcement learning has gained additional interest in solving decision-making tasks as increasing neuroscientific research demonstrates that significant links exist between reinforcement learning and specific neural substrates. Because of the tremendous research that focuses on human brains and reinforcement learning, scientists have investigated how robots can autonomously tackle complex tasks in the form of making a self-driving agent control in a human-like way. In this study, we propose an end-to-end architecture using novel deep-Q-network architecture in conjunction with a recurrence to resolve the problem in the field of simulated self-driving. The main contribution of this study is that we trained the driving agent using a brain-inspired trial-and-error technique, which was in line with the real world situation. Besides, there are three innovations in the proposed learning network: raw screen outputs are the only information which the driving agent can rely on, a weighted layer that enhances the differences of the lengthy episode, and a modified replay mechanism that overcomes the problem of sparsity and accelerates learning. The proposed network was trained and tested under a third-party OpenAI Gym environment. After training for several episodes, the resulting driving agent performed advanced behaviors in the given scene. We hope that in the future, the proposed brain-inspired learning system would inspire practicable self-driving control solutions.

Double Q-PID algorithm for mobile robot control

Many expert systems have been developed for self-adaptive PID controllers of mobile robots. However, the high computational requirements of the expert systems layers, developed for the tuning of the PID controllers, still require previous expert knowledge and high efficiency in algorithmic and software execution for real-time applications. To address these problems, in this paper we propose an expert agent-based system, based on a reinforcement learning agent, for self-adapting multiple low-level PID controllers in mobile robots. For the formulation of the artificial expert agent, we develop an incremental model-free algorithm version of the double Q-Learning algorithm for fast on-line adaptation of multiple low-level PID controllers. Fast learning and high on-line adaptability of the artificial expert agent is achieved by means of a proposed incremental active-learning exploration-exploitation procedure, for a non-uniform state space exploration, along with an experience replay mechanism for multiple value functions updates in the double Q-learning algorithm. A comprehensive comparative simulation study and experiments in a real mobile robot demonstrate the high performance of the proposed algorithm for a real-time simultaneous tuning of multiple adaptive low-level PID controllers of mobile robots in real world conditions.

A Two-Step Environment-Learning-Based Method for Optimal UAV Deployment

Unmanned aerial vehicles (UAVs) can be used as low-altitude flight base stations to satisfy the coverage requirements of wireless users in various scenarios. In practical applications, since the transmitted power and energy resources of the UAVs are limited and the propagation environments are complicated and time-variant, it is challenging to control a group of UAVs to ensure coverage performance while preserving the connectivity and safety of the UAV networks. To this end, a two-step environment-learning-based method is proposed for the intelligent deployment of the UAVs. First, a machine learning algorithm is used to establish an accurate prediction model of the link qualities from the UAVs to the users under a specific scenario for the next step. Then, a modified deep deterministic policy gradient (DDPG) algorithm is employed to control the movements of the UAVs according to the predicted link qualities and to maximize the proportion of covered users. The prioritized experience replay mechanism is introduced to the standard DDPG algorithm to accelerate the deployment procedure. The coverage performance is analyzed in both the interference-free situation and the situation with co-channel interference. Simulation results have shown that the proposed method has a higher convergence speed than the standard DDPG method. Additionally, the proposed deployment method can achieve higher coverage performance and better adaptability to the dynamic environment than three commonly used methods, the random method, the K-means-based method, and the statistical-channel-model-based method.

Deep Q-learning From Demonstrations

Deep reinforcement learning (RL) has achieved several high profile successes in difficult decision-making problems. However, these algorithms typically require a huge amount of data before they reach reasonable performance. In fact, their performance during learning can be extremely poor. This may be acceptable for a simulator, but it severely limits the applicability of deep RL to many real-world tasks, where the agent must learn in the real environment. In this paper we study a setting where the agent may access data from previous control of the system. We present an algorithm, Deep Q-learning from Demonstrations (DQfD), that leverages small sets of demonstration data to massively accelerate the learning process even from relatively small amounts of demonstration data and is able to automatically assess the necessary ratio of demonstration data while learning thanks to a prioritized replay mechanism. DQfD works by combining temporal difference updates with supervised classification of the demonstrator’s actions. We show that DQfD has better initial performance than Prioritized Dueling Double Deep Q-Networks (PDD DQN) as it starts with better scores on the first million steps on 41 of 42 games and on average it takes PDD DQN 83 million steps to catch up to DQfD’s performance. DQfD learns to out-perform the best demonstration given in 14 of 42 games. In addition, DQfD leverages human demonstrations to achieve state-of-the-art results for 11 games. Finally, we show that DQfD performs better than three related algorithms for incorporating demonstration data into DQN.

Local Energy Trading Behavior Modeling With Deep Reinforcement Learning

In this paper, we model prosumers’ energy trading behavior, with the operation of an energy storage system, in a proposed event-driven local energy market. Through modeling local energy trading strategies of a prosumer in the proposed holistic market model, the prosumer’s decision-making process will be built as a Markov decision process with many continuous variables. Then, this decision-making process of local market participation will be solved by deep reinforcement learning technology with experience replay mechanism. Specifically, a deep Q-learning for local energy trading algorithm is modified from deep Q-network to facilitate such a decision-making within an intelligent energy system and promote prosumers’ willingness to participate in the localized energy ecosystem.

A Data-Consistency Scheme for the Distributed-Cache Storage of the Memcached System

Memcached, commonly used to speed up the data access in big-data and Internet-web applications, is a system software of the distributed-cache mechanism. But it is subject to the severe challenge of the loss of recently uncommitted updates in the case where the Memcached servers crash due to some reason. Although the replica scheme and the disk-log-based replay mechanism have been proposed to overcome this problem, they generate either the overhead of the replica synchronization or the persistent-storage overhead that is caused by flushing related logs. This paper proposes a scheme of backing up the write requests (i.e., set and add) on the Memcached client side, to reduce the overhead resulting from the making of disk-log records or performing the replica consistency. If the Memcached server fails, a timestamp-based recovery mechanism is then introduced to replay the write requests (buffered by relevant clients), for regaining the lost-data updates on the rebooted Memcached server, thereby meeting the data-consistency requirement. More importantly, compared with the mechanism of logging the write requests to the persistent storage of the master server and the server-replication scheme, the newly proposed approach of backing up the logs on the client side can greatly decrease the time overhead by up to 116.8% when processing the write workloads.