Deep Recurrent Q-network Research Articles

DRQN-based global optimal control of air conditioning water system

The intelligent control of the air conditioning water system, a critical component of HVAC systems, plays a crucial role in reducing building energy consumption. Reinforcement learning (RL), an emerging interactive learning method capable of self-optimization according to environmental changes, has been widely applied in the field of HVAC control. This paper focuses on an air conditioning water system with chiller group control. Initially, a load-based sequencing control strategy is adopted, and deadband ranges are set based on empirical knowledge to coordinate the sequencing control of multiple devices. Subsequently, cooling load, wet-bulb temperature, and the number of chillers are used as the state space. By incorporating the implicit temporal information associated with these changes, the Deep Recurrent Q-Network (DRQN) is introduced to effectively leverage historical data for optimizing operational parameters such as chiller supply water temperature, pump frequency, and cooling tower fan speed. Experimental results show that compared with traditional rule-based control and Deep Q-Network (DQN) control, the proposed control framework achieves energy savings of 20.38%-24.59% and 1.82%-3.48%, respectively, under different sequencing control strategies.

Customer Acquisition via Explainable Deep Reinforcement Learning

Effective customer acquisition is crucial for digital platforms, with sequential targeting ensuring that marketing messages are both timely and relevant. The proposed deep recurrent Q-network with attention (DRQN-attention) model enhances this process by optimizing long-term rewards and increasing decision-making transparency. Tested with a data set from a digital bank, the DRQN-attention model has proven to enhance clarity in decision making and outperform traditional methods in boosting long-term rewards. Its attention mechanism acts as a strategic tool for forward planning, pinpointing crucial ad marketing channels that are likely to engage and convert prospects. This capability enables marketers to understand the dynamic targeting strategies of the proposed model that align with customer profiles, dynamic behaviors, and the seasonality of the markets, thereby boosting confidence and effectiveness in their customer acquisition strategies.

Dealing with uncertainty: Balancing exploration and exploitation in deep recurrent reinforcement learning

Incomplete knowledge of the environment leads an agent to make decisions under uncertainty. One of the major dilemmas in Reinforcement Learning (RL) where an autonomous agent has to balance two contrasting needs in making its decisions is: exploiting the current knowledge of the environment to maximize the cumulative reward as well as exploring actions that allow improving the knowledge of the environment, hopefully leading to higher reward values (exploration–exploitation trade-off). Concurrently, another relevant issue regards the full observability of the states, which may not be assumed in all applications. For instance, when 2D images are considered as input in an RL approach used for finding the best actions within a 3D simulation environment. In this work, we address these issues by deploying and testing several techniques to balance exploration and exploitation trade-off on partially observable systems for predicting steering wheels in autonomous driving scenarios. More precisely, the final aim is to investigate the effects of using both adaptive and deterministic exploration strategies coupled with a Deep Recurrent Q-Network. Additionally, we adapted and evaluated the impact of a modified quadratic loss function to improve the learning phase of the underlying Convolutional Recurrent Neural Network. We show that adaptive methods better approximate the trade-off between exploration and exploitation and, in general, Softmax and Max-Boltzmann strategies outperform ϵ-greedy techniques.

Deadline-aware task offloading in vehicular networks using deep reinforcement learning

Smart vehicles have a rising demand for computation resources, and recently vehicular edge computing has been recognized as an effective solution. Edge servers deployed in roadside units are capable of accomplishing tasks beyond the capacity which is embedded inside the vehicles. However, the main challenge is to carefully select the tasks to be offloaded considering the deadlines, and in order to reduce energy consumption, while delivering a good performance. In this paper, we consider a vehicular edge computing network in which multiple cars are moving at non-constant speed and produce tasks at each time slot. Then, we propose a task offloading algorithm, aware of the vehicle’s direction, based on Rainbow, a deep Q-learning algorithm combining several independent improvements to the deep Q-network algorithm. This is to overcome the conventional limits and to reach an optimal offloading policy, by effectively incorporating the computation resources of edge servers to jointly minimize average delay and energy consumption. Real-world traffic data is used to evaluate the performance of the proposed approach compared to other algorithms, in particular deep Q-network, double deep Q-network, and deep recurrent Q-network. Results of the experiments show an average reduction of 18% and 15% in energy consumption and delay, respectively, when using the proposed Rainbow deep Q-network based algorithm in comparison to the state-of-the-art. Moreover, the stability and convergence of the learning process have significantly improved by adopting the Rainbow algorithm.

Reinforcement Learning for Pass Detection and Generation of Possession Statistics in Soccer

We propose a reinforcement learning (RL) based technique to detect passes from the video of a soccer match. The detection of passes determines ball possession statistics of a soccer match. A sequence of video frames is mapped to a sequence of states, such as ball with team A or team B or ball not possessed either by team A or B. The agent of RL learns the frame-to-state mapping and the optimal policy to decide the mapping task. We propose a novel reward function by utilizing contextual information of the soccer game in order to help the agent decide the optimal policy. In this context, the advantage of RL is in the integration of a reward system in choosing an action that maps a video frame of a soccer match to one of three possible states. Unlike competing methods, we design the RL model in a way so that explicit identification of team labels of players is not required. We introduce a Deep Recurrent Q-Network (DRQN) to learn the optimal policy. For efficient training of the DRQN, we have proposed de-correlated experience replay (DER), a strategy that selects important experiences based on the correlations of the experiences stored in the replay memory. Experimental results show that at least 5.75% and 2.1% better accuracy are achieved in calculating pass detection and possession statistics, respectively, compared to similar approaches.

Logistics-involved task scheduling in cloud manufacturing with offline deep reinforcement learning

As an application of industrial information integration engineering (IIIE) in manufacturing, cloud manufacturing (CMfg) integrates enterprises’ manufacturing information and provides an open and sharing platform for processing manufacturing tasks with distributed manufacturing services. Assigning tasks to manufacturing enterprises in the CMfg platform calls for effective scheduling algorithms. In recent years, deep reinforcement learning (DRL) has been widely applied to tackle cloud manufacturing scheduling problems (CMfg-SPs) because of its high generalization and fast-responding capability. However, the current DRL algorithms need to be trial-and-error through online interaction with the environment, which is costly and not allowed in the real CMfg platform. This paper proposes a novel offline DRL scheduling algorithm that alleviates the online trial-and-error issue while retaining DRL’s original advantages. First, we describe the system model of CMfg-SPs and propose the sequential Markov decision process modeling strategy, where all tasks are regarded as one agent. Then, we introduce the framework of the decision transformer (DT), which converts the online scheduling decision-making problem into an offline classification problem. Finally, we construct an attention-based model as the agent’s policy and train it offline under the DT architecture. Experimental results indicate that the proposed method consistently matches or exceeds online DRL algorithms, including deep double q-network (DDQN), deep recurrent q-network (DRQN), proximal policy optimization (PPO), and the offline learning algorithm behavior cloning (BC) in terms of scheduling performance and model generalization.

Distributed dynamic spectrum access through multi-agent deep recurrent Q-learning in cognitive radio network

This paper addresses the problem of distributed dynamic spectrum access in a cognitive radio (CR) environment utilizing deep recurrent reinforcement learning. Specifically, the network consists of multiple primary users (PU) transmitting intermittently in their respective channels, while the secondary users (SU) attempt to access the channels when PUs are not transmitting. The problem is challenging considering the decentralized nature of CR network where each SU attempts to access a vacant channel, without coordination with other SUs, which result in collision and throughput loss. To address this issue, a multi-agent environment is considered where each of the SUs perform independent reinforcement learning to learn the appropriate policy to transmit opportunistically so as to minimize collisions with other users. In this article, we propose two long short-term memory (LSTM) based deep recurrent Q-network (DRQN) architectures for exploiting the temporal correlation in the transmissions by various nodes in the network. Furthermore, we investigate the effect of the architecture on success rate with varying number of users in the network and partial channel observations. Simulation results are compared with other existing reinforcement learning based techniques to establish the superiority of the proposed method.

Dynamic Spectrum Sharing Based on Deep Reinforcement Learning in Mobile Communication Systems.

The rapid development of mobile communication services in recent years has resulted in a scarcity of spectrum resources. This paper addresses the problem of multi-dimensional resource allocation in cognitive radio systems. Deep reinforcement learning (DRL) combines deep learning and reinforcement learning to enable agents to solve complex problems. In this study, we propose a training approach based on DRL to design a strategy for secondary users in the communication system to share the spectrum and control their transmission power. The neural networks are constructed using the Deep Q-Network and Deep Recurrent Q-Network structures. The results of the conducted simulation experiments demonstrate that the proposed method can effectively improve the user's reward and reduce collisions. In terms of reward, the proposed method outperforms opportunistic multichannel ALOHA by about 10% and about 30% for the single SU scenario and the multi-SU scenario, respectively. Furthermore, we explore the complexity of the algorithm and the influence of parameters in the DRL algorithm on the training.

Modeling and optimization of epidemiological control policies through reinforcement learning

Pandemics involve the high transmission of a disease that impacts global and local health and economic patterns. The impact of a pandemic can be minimized by enforcing certain restrictions on a community. However, while minimizing infection and death rates, these restrictions can also lead to economic crises. Epidemiological models help propose pandemic control strategies based on non-pharmaceutical interventions such as social distancing, curfews, and lockdowns, reducing the economic impact of these restrictions. However, designing manual control strategies while considering disease spread and economic status is non-trivial. Optimal strategies can be designed through multi-objective reinforcement learning (MORL) models, which demonstrate how restrictions can be used to optimize the outcome of a pandemic. In this research, we utilized an epidemiological Susceptible, Exposed, Infected, Recovered, Deceased (SEIRD) model – a compartmental model for virtually simulating a pandemic day by day. We combined the SEIRD model with a deep double recurrent Q-network to train a reinforcement learning agent to enforce the optimal restriction on the SEIRD simulation based on a reward function. We tested two agents with unique reward functions and pandemic goals to obtain two strategies. The first agent placed long lockdowns to reduce the initial spread of the disease, followed by cyclical and shorter lockdowns to mitigate the resurgence of the disease. The second agent provided similar infection rates but an improved economy by implementing a 10-day lockdown and 20-day no-restriction cycle. This use of reinforcement learning and epidemiological modeling allowed for both economic and infection mitigation in multiple pandemic scenarios.

Deep Recurrent Q-Network Methods for mmWave Beam Tracking systems

This article studies a reinforcement learning (RL) approach for beam tracking problems in millimeter-wave massive multiple-input multiple-output (MIMO) systems. Entire beam sweeping in traditional beam training problems is intractable due to prohibitive search overheads. To solve this issue, a partially observable Markov decision process (POMDP) formulation can be applied where decisions are made with partial beam sweeping. However, the POMDP cannot be straightforwardly addressed by existing RL approaches which are intended for fully observable environments. In this paper, we propose a deep recurrent Q-learning (DRQN) method which provides an efficient beam decision policy only with partial observations. Numerical results validate the superiority of the proposed method over conventional schemes.

Novel Reinforcement Learning Research Platform for Role-Playing Games

The latest achievements in the field of reinforcement learning have encouraged the development of vision-based learning methods that compete with human-provided results obtained on various games and training environments. Convolutional neural networks together with Q-learning-based approaches have managed to solve and outperform human players in environments such as Atari 2600, Doom or StarCraft II, but the niche of 3D realistic games with a high degree of freedom of movement and rich graphics remains unexplored, despite having the highest resemblance to real-world situations. In this paper, we propose a novel testbed to push the limits of deep learning methods, namely an OpenAI Gym-like environment based on Dark Souls III, a notoriously difficult role-playing game, where even human players have reportedly struggled. We explore two types of architectures, Deep Q-Network and Deep Recurrent Q-Network, providing the results of a first incursion into this new problem class. The source code for the training environment and baselines is made available.

Deadline-Aware Deep-Recurrent-Q-Network Governor for Smart Energy Saving

Complex cyber-physical-social systems (CPSS) consist of battery-supplied devices with low energy consumption requirements. It is essential to maintain the timing performance of computing or communication tasks while saving the device energy. Linux OS provides built-in frequency governors for power management. However, these governors are not able to incorporate the timing requirements of the application for decision-making and cannot adapt the power management decision to the specific application on target devices. This paper presents an intelligent Linux frequency Deep-Recurrent-Q-Network (DRQN) governor for dedicated applications with deadline requirements running on CPSS devices through machine learning. Although machine learning algorithms have made considerable breakthroughs in recent years, deploying them to real small devices is challenging because of the computational overhead. To tackle the computation overhead problem, an interactive learning framework is designed where the DRQN model performs only the lightest inference in the kernel while using the online data (time-series) to learn and update itself in real-time at the user level. The governor is tested on both standalone devices and networked devices. The experiment shows that DRQN can self-develop tradeoff policy to meet the user's need with low overhead. The energy saved by DRQN ranges from 7% to 33% for various deadlines.

An Adaptive MAC Protocol Based on Time-Domain Interference Alignment for UWANs

Abstract The spatial and temporal uncertainty caused by large propagation delays is a fundamental feature of Underwater Acoustic Networks (UWANs), which seriously affects the performance of the UWANs and also brings challenges to the design of MAC protocols. In this paper, we develop an adaptive MAC protocol based on deep reinforcement learning for UWANs, called ARL-MAC protocol, to intelligently allocate time slots for nodes. Firstly, we design a reward mechanism based on the idea of Time-Domain Interference Alignment (TDIA). We determine the reward according to the combination of the node action and the feedback corresponding to the action. Then, we propose a flexible training mechanism to deal with the ever-changing underwater environment, which improves the fairness of time slot allocation. In addition, we introduce the Deep Recurrent Q-Network (DRQN) algorithm to solve the partially observable information issue. Finally, we evaluate the ARL-MAC protocol with the different number of nodes and changing network environment. Simulation results reveal that the ARL-MAC protocol outperforms other MAC protocols for UWANs in terms of throughput, collision rate and service fairness.

Improved Deep Recurrent Q-Network of POMDPs for Automated Penetration Testing

With the development of technology, people’s daily lives are closely related to networks. The importance of cybersecurity protection draws global attention. Automated penetration testing is the novel method to protect the security of networks, which enhances efficiency and reduces costs compared with traditional manual penetration testing. Previous studies have provided many ways to obtain a better policy for penetration testing paths, but many studies are based on ideal penetration testing scenarios. In order to find potential vulnerabilities from the perspective of hackers in the real world, this paper models the process of black-box penetration testing as a Partially Observed Markov Decision Process (POMDP). In addition, we propose a new algorithm named ND3RQN, which is applied to the automated black-box penetration testing. In the POMDP model, an agent interacts with a network environment to choose a better policy without insider information about the target network, except for the start points. To handle this problem, we utilize a Long Short-Term Memory (LSTM) structure empowering agent to make decisions based on historical memory. In addition, this paper enhances the current algorithm using the structure of the neural network, the calculation method of the Q-value, and adding noise parameters to the neural network to advance the generalization and efficiency of this algorithm. In the last section, we conduct comparison experiments of the ND3RQN algorithm and other recent state-of-the-art (SOTA) algorithms. The experimental results vividly show that this novel algorithm is able to find a greater attack-path strategy for all vulnerable hosts in the automated black-box penetration testing. Additionally, the generalization and robustness of this algorithm are far superior to other SOTA algorithms in different size simulation scenarios based on the CyberBattleSim simulation developed by Microsoft.

Model-Free Deep Recurrent Q-Network Reinforcement Learning for Quantum Circuit Architectures Design

Artificial intelligence (AI) technology leads to new insights into the manipulation of quantum systems in the Noisy Intermediate-Scale Quantum (NISQ) era. Classical agent-based artificial intelligence algorithms provide a framework for the design or control of quantum systems. Traditional reinforcement learning methods are designed for the Markov Decision Process (MDP) and, hence, have difficulty in dealing with partially observable or quantum observable decision processes. Due to the difficulty of building or inferring a model of a specified quantum system, a model-free-based control approach is more practical and feasible than its counterpart of a model-based approach. In this work, we apply a model-free deep recurrent Q-network (DRQN) reinforcement learning method for qubit-based quantum circuit architecture design problems. This paper is the first attempt to solve the quantum circuit design problem from the recurrent reinforcement learning algorithm, while using discrete policy. Simulation results suggest that our long short-term memory (LSTM)-based DRQN method is able to learn quantum circuits for entangled Bell–Greenberger–Horne–Zeilinger (Bell–GHZ) states. However, since we also observe unstable learning curves in experiments, suggesting that the DRQN could be a promising method for AI-based quantum circuit design application, more investigation on the stability issue would be required.

Deep reinforcement learning based trajectory optimization for magnetometer-mounted UAV to landmine detection

Unmanned aerial vehicles (UAVs) have emerged as a viable choice for data collection and landmine (LM) detection. The LM buried under the dirt or sand is detected using a UAV-mounted magnetometer in this paper. A UAV is deployed to gather data along the intended route when the magnetometer receives a signal from the LMs. During a whole round of data collection, we want to reduce the total energy consumption of the UAV-Magnetometer-LM system. To do this, we turn the energy consumption reduction issue into a limited combinatorial optimization problem by concurrently picking time slots and arranging the UAV’s visitation sequence to identify the LM. The problem of minimizing energy usage is NP-hard, making it difficult to solve optimally. In order to tackle this challenge, we used the deep reinforcement learning (DRL) based deep deterministic policy gradient (DDPG) scheme. DDPG is used to enhance the convergence speed and eliminate redundant computations. Furthermore, to improve the detection in real-time, we proposed the proximal online policy technique (POPT). Numerical results demonstrate that the proposed scheme consumes 37.14%, 31.25%, and 21.42% better results than synthetic aperture radar (SAR), convolution neural network (CNN), and double deep recurrent Q-network (DDRQN).

Joint Air-to-Ground Scheduling in UAV-Aided Vehicular Communication: A DRL Approach With Partial Observations

The unmanned aerial vehicle (UAV)-aided vehicular communication can be greatly facilitated by joint optimization of UAV trajectory design and vehicle assignment. While most existing works are based on the full observation of system state, we consider the partial observability with predicting the vehicles’ trajectories. The vehicle trajectory prediction and joint optimization problem are modeled as a Partially-Observable Markov Decision Process (POMDP). To deal with the non-Markovian of the POMDP, we construct a new deep recurrent Q-network (DRQN) framework based on deep Q-network (DQN) algorithm and Long Short Term Memory (LSTM) layer. Simulation results demonstrate that the proposed DRQN-based scheme is fast convergent and outperforms the baseline schemes in terms of the sum spectral efficiency.

강화학습 기반 안벽 배치 계획 알고리즘 개발

In the shipbuilding, outfitting processes are mainly performed on ships moored by quays of the shipyard. Quays in shipyards have different lengths, depths, and facilities so that the allowable quays for each ship are limited. In addition, the working efficiency of the quay process depends on the kinds of the quays where the outfitting processes are performed. As a result, it is inevitable for ships to move around the quays to avoid delay in schedule while maximizing the working efficiency. The movements of ships should be minimized because it is a waste of cost and time. In most shipyards, the quay allocation is manually determined by planners using their own implicit rules. However, the optimal plan can’t be obtained by those heuristics which only considers limited problem space. In this study, the scheduling algorithm for the quay allocation problem is developed using a reinforcement learning approach. Based on the Markov decision process model using discrete-event simulation, the scheduling agent is trained by DRQN (Deep Recurrent Q-network) algorithm and tested with 10 scenarios. The proposed algorithm outperforms the reference value set by heuristic in terms of three KPIs-unallocated duration, the number of ship movements, and ship-quay priority ratio.

Basketball Motion Posture Recognition Based on Recurrent Deep Learning Model

In order to improve the training effect of athletes and effectively identify the movement posture of basketball players, we propose a basketball motion posture recognition method based on recurrent deep learning. A one-dimensional convolution layer is added to the neural network structure of the deep recurrent Q network (DRQN) to extract the athlete pose feature data before the long short-term memory (LSTM) layer. The acceleration and angular velocity data of athletes are collected by inertial sensors, and the multi-dimensional motion posture features are extracted from the time domain and frequency domain, respectively, and the posture recognition of basketball is realized by DRQN. Finally, the new reinforcement learning algorithm is trained and tested in a time-series-related environment. The experimental results show that the method can effectively recognize the basketball motion posture, and the average accuracy of posture recognition reaches 99.3%.

Neural Policy Style Transfer

Style Transfer has been proposed in a number of fields: fine arts, natural language processing, and fixed trajectories. We scale this concept up to control policies within a Deep Reinforcement Learning infrastructure. Each network is trained to maximize the expected reward, which typically encodes the goal of an action, and can be described as the content. The expressive power of deep neural networks enables encoding a secondary task, which can be described as the style. The Neural Policy Style Transfer (NPST)11NPST: Neural Policy Style Transfer. algorithm is proposed to transfer the style of one policy to another, while maintaining the content of the latter. Different policies are defined via Deep Q-Network architectures. These models are trained using demonstrations through Inverse Reinforcement Learning. Two different sets of user demonstrations are performed, one for content and other for style. Different styles are encoded as defined by user demonstrations. The generated policy is the result of feeding a content policy and a style policy to the NPST algorithm. Experiments are performed in a catch-ball game inspired by the Deep Reinforcement Learning classical Atari games; and a real-world painting scenario with a full-sized humanoid robot, based on previous works of the authors. The implementation of three different Q-Network architectures (Shallow, Deep and Deep Recurrent Q-Network) to encode the policies within the NPST framework is proposed and the results obtained in the experiments with each of these architectures compared.