Deep Recurrent Q-network Research Articles

Network defense decision-making based on a stochastic game system and a deep recurrent Q-network

Defense decision-making in cybersecurity has increasingly relied upon stochastic game processes that combine game theory with a Markov decision process (MDP). However, the MDP presumes that both attackers and defenders are perfectly rational and have complete information, which greatly limits the scope of application and guidance value of MDP to the defense decision-making process. The present study addresses this issue by applying a partially observable MDP to analyze attack-defense behaviors, and a deep Q-network (DQN) algorithm based on a recurrent neural network for solving game equilibria dynamically and intelligently under conditions of partial rationality and incomplete information. The proposed DQN method enables network defense strategies to leverage online learning to gradually approach an optimal defense strategy. The rationality and convergence of the proposed approach are demonstrated by conducting simulations and comparative analyses of both the attacking and defending parties engaged in distributed reflection denial of service attacks.

Application of Deep Recurrent Q Network with Dueling Architecture for Optimal Sepsis Treatment Policy

Sepsis is one of the leading causes of mortality globally, and it costs billions of dollars annually. However, treating septic patients is currently highly challenging, and more research is needed into a general treatment method for sepsis. Therefore, in this work, we propose a reinforcement learning method for learning the optimal treatment strategies for septic patients. We model the patient physiological time series data as the input for a deep recurrent Q-network that learns reliable treatment policies. We evaluate our model using an off-policy evaluation method, and the experimental results indicate that it outperforms the physicians’ policy, reducing patient mortality up to 3.04%. Thus, our model can be used as a tool to reduce patient mortality by supporting clinicians in making dynamic decisions.

Application of Machine Learning on Crop Yield Prediction in Agriculture Enforcement

Yield forecasting is based totally entirely on soil, water and vegetation to be a possible subject. Deep-based depth-based fashions are widely accustomed extract important plant functions for predictive purposes. Although such strategies are necessary to resolve the matter of predicting yields there are the subsequent abnormalities: they can't create an indirect or indirect map between raw facts and yield values; and also the full functionality of this excess is explained within the high satisfaction of the published works. Deep durability provides guidance and motivation for the above-mentioned errors. Combining master intensity and deep mastering, deep reinforcing mastering creates a comprehensive yield prediction framework which will plan the uncooked facts in crop prediction rates. The proposed project creates a version of the Deep Recurrent Q-Network Support Vector Machine deep mastering set of rules over Q-Learning to strengthen the mastering set of rules for predicting yield. Sequential downloads of the Recurrent Neural community are fed by fact parameters. The Q-mastering community creates a predictive yield environment based totally on input criteria. The precise layer displays the discharge values of the Support Vector Machine on the Q values. The reinforcement master component contains a mix of parametric functions on the sting that helps predict the yield. Finally, the agent obtains a measure of the mixture of steps performed by minimizing the error and increasing the accuracy of the forecast. The proposed model successfully predicts this crop yield that's hip by keeping the initial distribution of facts with 93.7% accuracy.

Aerial filming with synchronized drones using reinforcement learning

Usage of multiple drones is necessary for aerial filming applications to ensure redundancy. However, this could inevitably contribute to higher risks of collisions, especially when the number of drones increases. Hence, this motivates us to explore various autonomous flight formation control methods that have the potential to enable multiple drones to effectively track a specific target at the same time. In this paper, we designed a model-free deep reinforcement learning algorithm, which is mainly based on the Deep Recurrent Q-Network concept, for the aforementioned purposes. The proposed algorithm was expanded into single and multi-agent types that enable multiple drones tracking while maintaining formation and preventing collision. The involved rewards in these approaches are two-dimensional in nature and are dependent on the communication system. Using Microsoft AirSim simulator, a virtual environment that includes four virtual drones was developed for experimental purposes. A comparison was made among various methods during the simulations, and the results concluded that the recurrent, single-agent model is the most effective method, being 33% more effective than its recurrent, multi-agent counterparts. The poor performance of the non-recurrent, single-agent baseline model also suggests that the recurrent elements in the network are essential to enable desirable multiple-drones flight.

A numerical study of fish adaption behaviors in complex environments with a deep reinforcement learning and immersed boundary\u2013lattice Boltzmann method

Fish adaption behaviors in complex environments are of great importance in improving the performance of underwater vehicles. This work presents a numerical study of the adaption behaviors of self-propelled fish in complex environments by developing a numerical framework of deep learning and immersed boundary–lattice Boltzmann method (IB–LBM). In this framework, the fish swimming in a viscous incompressible flow is simulated with an IB–LBM which is validated by conducting two benchmark problems including a uniform flow over a stationary cylinder and a self-propelled anguilliform swimming in a quiescent flow. Furthermore, a deep recurrent Q-network (DRQN) is incorporated with the IB–LBM to train the fish model to adapt its motion to optimally achieve a specific task, such as prey capture, rheotaxis and Kármán gaiting. Compared to existing learning models for fish, this work incorporates the fish position, velocity and acceleration into the state space in the DRQN; and it considers the amplitude and frequency action spaces as well as the historical effects. This framework makes use of the high computational efficiency of the IB–LBM which is of crucial importance for the effective coupling with learning algorithms. Applications of the proposed numerical framework in point-to-point swimming in quiescent flow and position holding both in a uniform stream and a Kármán vortex street demonstrate the strategies used to adapt to different situations.

Unmanned Aerial Vehicle Path Planning Algorithm Based on Deep Reinforcement Learning in Large-Scale and Dynamic Environments

Path planning is one of the key technologies for autonomous flight of Unmanned Aerial Vehicle. Traditional path planning algorithms have some limitations and deficiencies in the complex and dynamic environment. In this article, we propose a deep reinforcement learning approach for three-dimensional path planning by utilizing the local information and relative distance without global information. UAV can obtain the limited environmental information nearby in the actual scenario with limited sensor capabilities. Therefore, path planning can be formulated as a Partially Observable Markov Decision Process. The recurrent neural network with temporal memory is constructed to address the partial observability problem by extracting crucial information from historical state-action sequences. We develop an action selection strategy that combines the current reward value and the state-action value to reduce the meaningless exploration. In addition, we construct two sample memory pools and propose an adaptive experience replay mechanism based on the frequency of failure. The simulation experiment results show that our method has significant improvements over Deep Q-Network and Deep Recurrent Q-Network in terms of stability and learning efficiency. Our approach successfully plans a reasonable three-dimensional path in the large-scale and complex environment, and has the perfect ability to avoid obstacles.in the unknown environment.

Double Deep Recurrent Reinforcement Learning for Centralized Dynamic Multichannel Access

We consider the problem of dynamic multichannel access for transmission maximization in multiuser wireless communication networks. The objective is to find a multiuser strategy that maximizes global channel utilization with a low collision in a centralized manner without any prior knowledge. Obtaining an optimal solution for centralized dynamic multichannel access is an extremely difficult problem due to the large‐state and large‐action space. To tackle this problem, we develop a centralized dynamic multichannel access framework based on double deep recurrent Q‐network. The centralized node first maps current state directly to channel assignment actions, which can overcome prohibitive computation compared with reinforcement learning. Then, the centralized node can be easy to select multiple channels by maximizing the sum of value functions based on a trained neural network. Finally, the proposed method avoids collisions between secondary users through centralized allocation policy.

Deep Reinforcement Learning Control for Radar Detection and Tracking in Congested Spectral Environments

In this paper, dynamic non-cooperative coexistence between a cognitive pulsed radar and a nearby communications system is addressed by applying nonlinear value function approximation via deep reinforcement learning (Deep RL) to develop a policy for optimal radar performance. The radar learns to vary the bandwidth and center frequency of its linear frequency modulated (LFM) waveforms to mitigate mutual interference with other systems and improve target detection performance while also maintaining sufficient utilization of the available frequency bands required for a fine range resolution. We demonstrate that our approach, based on the Deep Q-Learning (DQL) algorithm, enhances important radar metrics, including SINR and bandwidth utilization, more effectively than policy iteration or sense-and-avoid (SAA) approaches in a variety of realistic coexistence environments. We also extend the DQL-based approach to incorporate Double Q-learning and a recurrent neural network to form a Double Deep Recurrent Q-Network (DDRQN). We demonstrate the DDRQN results in favorable performance and stability compared to DQL and policy iteration. Finally, we demonstrate the practicality of our proposed approach through a discussion of experiments performed on a software defined radar (SDRadar) prototype system. Our experimental results indicate that the proposed Deep RL approach significantly improves radar detection performance in congested spectral environments when compared to policy iteration and SAA.

Heterogeneous Task Offloading and Resource Allocations via Deep Recurrent Reinforcement Learning in Partial Observable Multifog Networks

As wireless services and applications become more sophisticated and require faster and higher-capacity networks, there is a need for an efficient management of the execution of increasingly complex tasks based on the requirements of each application. In this regard, fog computing enables the integration of virtualized servers into networks and brings cloud services closer to end devices. In contrast to the cloud server, the computing capacity of fog nodes is limited and thus a single fog node might not be capable of computing-intensive tasks. In this context, task offloading can be particularly useful at the fog nodes by selecting the suitable nodes and proper resource management while guaranteeing the Quality-of-Service (QoS) requirements of the users. This paper studies the design of a joint task offloading and resource allocation control for heterogeneous service tasks in multi-fog nodes systems. This problem is formulated as a partially observable stochastic game, in which each fog node cooperates to maximize the aggregated local rewards while the nodes only have access to local observations. To deal with partial observability, we apply a deep recurrent Q-network (DRQN) approach to approximate the optimal value functions. The solution is then compared to a deep Q-network (DQN) and deep convolutional Q-network (DCQN) approach to evaluate the performance of different neural networks. Moreover, to guarantee the convergence and accuracy of the neural network, an adjusted exploration-exploitation method is adopted. Provided numerical results show that the proposed algorithm can achieve a higher average success rate and lower average overflow than baseline methods.

Deep Q-Network with Predictive State Models in Partially Observable Domains

While deep reinforcement learning (DRL) has achieved great success in some large domains, most of the related algorithms assume that the state of the underlying system is fully observable. However, many real-world problems are actually partially observable. For systems with continuous observation, most of the related algorithms, e.g., the deep Q-network (DQN) and deep recurrent Q-network (DRQN), use history observations to represent states; however, they often make computation-expensive and ignore the information of actions. Predictive state representations (PSRs) can offer a powerful framework for modelling partially observable dynamical systems with discrete or continuous state space, which represents the latent state using completely observable actions and observations. In this paper, we present a PSR model-based DQN approach which combines the strengths of the PSR model and DQN planning. We use a recurrent network to establish the recurrent PSR model, which can fully learn dynamics of the partially continuous observable environment. Then, the model is used for the state representation and update of DQN, which makes DQN no longer rely on a fixed number of history observations or recurrent neural network (RNN) to represent states in the case of partially observable environments. The strong performance of the proposed approach is demonstrated on a set of robotic control tasks from OpenAI Gym by comparing with the technique with the memory-based DRQN and the state-of-the-art recurrent predictive state policy (RPSP) networks. Source code is available at https://github.com/RPSR-DQN/paper-code.git.

The Application of Deep Reinforcement Learning to Distributed Spectrum Access in Dynamic Heterogeneous Environments With Partial Observations

This papera <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> investigates deep reinforcement learning (DRL) based on a Recurrent Neural Network (RNN) for Dynamic Spectrum Access (DSA) under partial observations, referred to as a Deep Recurrent Q-Network (DRQN). Specifically, we consider a scenario with multiple independent channels and multiple heterogeneous Primary Users (PUs). Two key challenges in our problem formulation are that we assume our DRQN node does not have any prior knowledge of the other nodes' behavior patterns and attempts to predict the future channel state based on previous observations. The goal of the DRQN is to learn a channel access strategy with a low collision rate but a high channel utilization rate. With proper definitions of the state, action and rewards, our extensive simulation results show that a DRQN-based approach can handle a variety of communication environments including dynamic environments. Further, our results show that the DRQN node is also able to cope with multi-rate and multi-agent scenarios. Importantly, we show the following benefits of using recurrent neural networks in DSA: (i) the ability to learn the optimal strategy in different environments under partial observations; (ii) robustness to imperfect observations and (iii) the ability to utilize multiple channels, and (iv) robustness in the presence of multiple agents. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> A parton of this work was presented at MILCOM 2018 in [1].

Dynamic Computation Offloading in IoT Fog Systems With Imperfect Channel-State Information: A POMDP Approach

Driven by the growing popularity of mobile applications, such as the Internet of Things (IoT), fog computing has been envisioned as a promising approach to enhance the computation capability of mobile devices and reduce the energy consumption. In this article, we aim to investigate the dynamic computation offloading problem in the IoT fog system under the fast time-varying wireless channel conditions. Our work differs from the existing work, which is based on the assumption that the channel-state information can be perfectly obtained by the offloading agent (e.g., the IoT device). In reality, due to hardware limitation, short sensing time, and network connectivity issues in IoT fog systems, it is difficult for the IoT device to have the perfect knowledge of a dynamic channel environment. Therefore, in this article, we propose a partially observable offloading scheme to enable the IoT device to make the optimal offloading decision with imperfect channel-state information. The optimization problem is formulated as a partially observable Markov decision process (POMDP) formulation, with the objective of minimizing the IoT device's energy consumption while meeting its requirement on task processing delay. To find the optimal offloading solution, an offline algorithm based on the deep recurrent $Q$ -network (DRQN) is developed. Finally, extensive simulation experiments are performed to evaluate the effectiveness of the proposed offloading scheme.

Crop Yield Prediction Using Deep Reinforcement Learning Model for Sustainable Agrarian Applications

Predicting crop yield based on the environmental, soil, water and crop parameters has been a potential research topic. Deep-learning-based models are broadly used to extract significant crop features for prediction. Though these methods could resolve the yield prediction problem there exist the following inadequacies: Unable to create a direct non-linear or linear mapping between the raw data and crop yield values; and the performance of those models highly relies on the quality of the extracted features. Deep reinforcement learning provides direction and motivation for the aforementioned shortcomings. Combining the intelligence of reinforcement learning and deep learning, deep reinforcement learning builds a complete crop yield prediction framework that can map the raw data to the crop prediction values. The proposed work constructs a Deep Recurrent Q-Network model which is a Recurrent Neural Network deep learning algorithm over the Q-Learning reinforcement learning algorithm to forecast the crop yield. The sequentially stacked layers of Recurrent Neural network is fed by the data parameters. The Q- learning network constructs a crop yield prediction environment based on the input parameters. A linear layer maps the Recurrent Neural Network output values to the Q-values. The reinforcement learning agent incorporates a combination of parametric features with the threshold that assist in predicting crop yield. Finally, the agent receives an aggregate score for the actions performed by minimizing the error and maximizing the forecast accuracy. The proposed model efficiently predicts the crop yield outperforming existing models by preserving the original data distribution with an accuracy of 93.7%.

Reinforcement learning-based cell selection in sparse mobile crowdsensing

Sparse Mobile Crowdsensing (MCS) is a novel MCS paradigm which allows us to use the mobile devices to collect sensing data from only a small subset of cells (sub-areas) in the target sensing area while intelligently inferring the data of other cells with quality guarantee. Since selecting sensed data from different cell sets will probably lead to diverse levels of inference data quality, cell selection (i.e., choosing which cells in the target area to collect sensed data from participants) is a critical issue that will impact the total amount of data that requires to be collected (i.e., data collection costs) for ensuring a certain level of data quality. To address this issue, this paper proposes the reinforcement learning-based cell selection algorithm for Sparse MCS. First, we model the key concepts in reinforcement learning including state, action, and reward, and then propose a Q-learning based cell selection algorithm. To deal with the large state space, we employ the deep Q-network to learn the Q-function that can help decide which cell is a better choice under a certain state during cell selection. Then, we modify the Q-network to a deep recurrent Q-network with LSTM to catch the temporal patterns and handle partial observability. Furthermore, we leverage the transfer learning techniques to relieve the dependency on a large amount of training data. Experiments on various real-life sensing datasets verify the effectiveness of our proposed algorithms over the state-of-the-art mechanisms in Sparse MCS by reducing up to 20% of sensed cells with the same data inference quality guarantee.

Exploring Deep Recurrent Q-Learning for Navigation in a 3D Environment

Learning to navigate in 3D environments from raw sensory input is an important step towards bridging the gap between human players and artificial intelligence in digital games. Recent advances in deep reinforcement learning have seen success in teaching agents to play Atari 2600 games from raw pixel information where the environment is always fully observable by the agent. This is not true for first-person 3D navigation tasks. Instead, the agent is limited by its field of view which limits its ability to make optimal decisions in the environment. This paper explores using a Deep Recurrent Q-Network implementation with a long short-term memory layer for dealing with such tasks by allowing an agent to process recent frames and gain a memory of the environment. An agent was trained in a 3D first-person labyrinth-like environment for 2 million frames. Informal observations indicate that the trained agent navigated in the right direction but was unable to find the target of the environment.