Recurrent Reinforcement Learning Research Articles

Twin-system recurrent reinforcement learning for optimizing portfolio strategy

Portfolio management is important for sequential investment decisions in response to fluctuating financial markets. As portfolio management can be formulated as a sequential decision-making problem, it has been addressed using reinforcement learning in recent years. However, reinforcement learning methods face challenges in addressing portfolio management problems considering practical constraints. To overcome the limitations, this study proposes a twin-system approach, establishing a tractable twin that mirrors the original problem but with more manageable constraints and system dynamics. Once an optimized portfolio strategy is achieved within the tractable twin, the proposed mapping function translates it back to the original problem, ensuring the retention of optimized performance. Unlike the previous study, the proposed recurrent reinforcement learning method optimizes the portfolio strategy for a single agent managing all candidate assets. This method allows for comprehensive investment decisions by incorporating the features of candidate assets, leading to a more globally optimized portfolio strategy. Experimental studies demonstrate that the proposed method consistently outperforms benchmark strategies on the US sector and foreign exchange portfolios.

A Deep Recurrent Reinforcement Learning Approach for Enhanced MPPT in PV Systems

Under partial shading condition (PSC), for efficient usage, solar photovoltaic (PV) systems are required to operate at global maximum power point (GMPP). A maximum power point tracking (MPPT) controller is usually employed for this task. In this work, for MPPT under PSC, a recurrent deep reinforcement learning (DRL) approach has been developed and investigated. The optimized neural network, augmented by long short-term memory (LSTM), is trained using the proximal policy optimization (PPO) DRL algorithm. The MPPT performance of the proposed scheme was assessed using the GMPP attainment accuracy (%), on several realistic static and dynamic test cases. The presented investigations revealed the proposed controller's average MPPT accuracy of 97.79% for the considered random static PSCs. Similarly, average MPPT accuracies of 95.69% under changing irradiance, 95.70% under change in temperature and 95.77% under changes in both irradiance and temperature were recorded. These accuracies are found to be significantly superior to recent reported works. The resulting performance enhancement is attributed to the inclusion of LSTM, which allowed the agent to retain information about past states and actions, enabling it to make informed decisions. Therefore, based on the presented investigations, it is concluded that the proposed PPO-LSTM approach is an excellent MPPT alternative for PV systems.

Dealing with uncertainty: Balancing exploration and exploitation in deep recurrent reinforcement learning

Incomplete knowledge of the environment leads an agent to make decisions under uncertainty. One of the major dilemmas in Reinforcement Learning (RL) where an autonomous agent has to balance two contrasting needs in making its decisions is: exploiting the current knowledge of the environment to maximize the cumulative reward as well as exploring actions that allow improving the knowledge of the environment, hopefully leading to higher reward values (exploration–exploitation trade-off). Concurrently, another relevant issue regards the full observability of the states, which may not be assumed in all applications. For instance, when 2D images are considered as input in an RL approach used for finding the best actions within a 3D simulation environment. In this work, we address these issues by deploying and testing several techniques to balance exploration and exploitation trade-off on partially observable systems for predicting steering wheels in autonomous driving scenarios. More precisely, the final aim is to investigate the effects of using both adaptive and deterministic exploration strategies coupled with a Deep Recurrent Q-Network. Additionally, we adapted and evaluated the impact of a modified quadratic loss function to improve the learning phase of the underlying Convolutional Recurrent Neural Network. We show that adaptive methods better approximate the trade-off between exploration and exploitation and, in general, Softmax and Max-Boltzmann strategies outperform ϵ-greedy techniques.

RRIoT: Recurrent reinforcement learning for cyber threat detection on IoT devices

To address the recent worldwide proliferation of cybersecurity attacks across computing systems, especially internet-of-things devices, new robust and automated methods are needed to detect and mitigate the attacks in real time, ensuring the confidentiality, integrity, and availability of systems. Machine Learning (ML) techniques have shown promise in detecting some types of attacks. However, they are not universally successful in detecting sequential attacks. In this study, we propose RRIoT, to mitigate this issue. RRIoT applies a Deep Deterministic Policy Gradient reinforcement learning (RL) algorithm in conjunction with an LSTM layer within an adversarial environment to detect and identify attacks. We evaluate our method against novel and state-of-the-art ML/RL algorithms which build upon previous RL algorithms such as deep Q-networks (DQN), dueling deep Q-networks (DDQN), and deep deterministic policy gradient (DDPG). Our results indicate that our proposed RRIoT generally performs better than existing ML algorithms and performs as well as or better than novel RL algorithms with new network architectures. We leverage Shapley Additive Global Importance (SAGE) to provide additional insight into which features contribute most to a model's performance and verify feature importance through the implementation of an ablation study across three IoT telemetry datasets.

Multi-User Delay-Constrained Scheduling With Deep Recurrent Reinforcement Learning

Multi-User Delay-Constrained Scheduling With Deep Recurrent Reinforcement Learning

Deep Recurrent Reinforcement Learning for Partially Observable User Association in a Vertical Heterogenous Network

Deep Recurrent Reinforcement Learning for Partially Observable User Association in a Vertical Heterogenous Network

Imperfect CSI-Based Resource Management in Cognitive IoT Networks: A Deep Recurrent Reinforcement Learning Framework

Imperfect CSI-Based Resource Management in Cognitive IoT Networks: A Deep Recurrent Reinforcement Learning Framework

A new hybrid method of recurrent reinforcement learning and BiLSTM for algorithmic trading

Recently, the algorithmic trading of financial assets is rapidly developing with the rise of deep learning. In particular, deep reinforcement learning, as a combination of deep learning and reinforcement learning, stands out among many approaches in the field of decision-making because of its high performance, strong generalization, and high fitting ability. In this paper, we attempt to propose a hybrid method of recurrent reinforcement learning (RRL) and deep learning to figure out the algorithmic trading problem of determining the optimal trading position in the daily trading activities of the stock market. We adopt deep neural network (DNN), long short-term memory neural network (LSTM), and bidirectional long short-term memory neural network (BiLSTM) to automatically extract higher-level abstract feature information from sequential trading data, respectively, and then generate optimal trading strategies by interacting with the environment in a reinforcement learning framework. In particular, the BiLSTM consisting of two LSTM models with opposite directions is able to make full use of the information from both directions in attempting to capture more effective information. In experiments, the daily data of Dow Jones, S&P500, and NASDAQ (from Jan-01, 2005 to Dec-31, 2020) are applied to verify the performance of the newly proposed DNN-RL, LSTM-RL, and BiLSTM-RL trading systems. Experimental results show that the proposed methods significantly outperform the benchmark methods, such as RRL and Buy and Hold, with higher scalability and better robustness. Especially, BiLSTM-RL performs better than other methods.

On Deep Recurrent Reinforcement Learning for Active Visual Tracking of Space Noncooperative Objects

Active tracking of space noncooperative object that merely relies on vision camera is greatly significant for autonomous rendezvous and debris removal. Considering its Partial Observable Markov Decision Process (POMDP) property, this paper proposes a novel deep recurrent neural network architecture, named as RAMAVT, incorporating Multi-Head Attention (MHA) module and Squeeze-and-Excitation (SE) layer that remarkably improve the representative ability of neural network with almost no extra computational cost. It has been successfully applied to value-based and policy gradient-based deep reinforcement learning algorithm, and learned to drive the chasing spacecraft to follow arbitrary space noncooperative object with high-frequency and near-optimal velocity control commands. Extensive experiments and robustness evaluations implemented on SNCOAT benchmark show the betterment and robustness of our method compared with other state-of-the-art active visual trackers. In addition, we make further ablation study and interpretability research on RAMAVT of which validity and rationality have been demonstrated. The source codes are available on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Dongzhou-1996/RAMAVT</uri> .

Spatial–temporal recurrent reinforcement learning for autonomous ships

This paper proposes a spatial–temporal recurrent neural network architecture for deep Q-networks that can be used to steer an autonomous ship. The network design makes it possible to handle an arbitrary number of surrounding target ships while offering robustness to partial observability. Furthermore, a state-of-the-art collision risk metric is proposed to enable an easier assessment of different situations by the agent. The COLREG rules of maritime traffic are explicitly considered in the design of the reward function. The final policy is validated on a custom set of newly created single-ship encounters called ‘Around the Clock’ problems and the commonly used Imazu (1987) problems, which include 18 multi-ship scenarios. Performance comparisons with artificial potential field and velocity obstacle methods demonstrate the potential of the proposed approach for maritime path planning. Furthermore, the new architecture exhibits robustness when it is deployed in multi-agent scenarios and it is compatible with other deep reinforcement learning algorithms, including actor-critic frameworks.

Deep Recurrent Reinforcement Learning-Based Distributed Dynamic Spectrum Access in Multichannel Wireless Networks With Imperfect Feedback

This paper investigates Dynamic Spectrum Access (DSA) paradigm with imperfect feedback for multiuser wireless network. Each user selects an orthogonal channel in particular time slot to transmit packet with a certain transmission probability. In next time slot, the user who has transmitted a packet receives an ACK signal based on local observation. Bearing in mind the dynamic nature of wireless networks, it is appealing to develop a blended strategy to perform effective DSA. This paper aims to design a distributed Deep Reinforcement Learning (DRL) based scheme with an objective of maximizing network utility. In conventional DRL framework, it is assumed that the feedback received (ACK packet) is always correct but in wireless networks, it may be lost or corrupted due to noise. Furthermore, it is challenging to promise that the multiple agents will cooperate to make coherent decisions in order to accomplish the same objective, particularly under imperfect feedback. To tackle these challenges, this work proposes (i) Deep Recurrent Reinforcement Learning network with integrated GRU layer to optimize the network utility function and (ii) a feedback recovery mechanism using complete and incomplete replay buffers. Extensive simulations corroborate the success of proposed scheme in complex multiuser scenario and exhibit robustness against the detrimental effects of the imperfect feedback.

Distributed dynamic spectrum access through multi-agent deep recurrent Q-learning in cognitive radio network

This paper addresses the problem of distributed dynamic spectrum access in a cognitive radio (CR) environment utilizing deep recurrent reinforcement learning. Specifically, the network consists of multiple primary users (PU) transmitting intermittently in their respective channels, while the secondary users (SU) attempt to access the channels when PUs are not transmitting. The problem is challenging considering the decentralized nature of CR network where each SU attempts to access a vacant channel, without coordination with other SUs, which result in collision and throughput loss. To address this issue, a multi-agent environment is considered where each of the SUs perform independent reinforcement learning to learn the appropriate policy to transmit opportunistically so as to minimize collisions with other users. In this article, we propose two long short-term memory (LSTM) based deep recurrent Q-network (DRQN) architectures for exploiting the temporal correlation in the transmissions by various nodes in the network. Furthermore, we investigate the effect of the architecture on success rate with varying number of users in the network and partial channel observations. Simulation results are compared with other existing reinforcement learning based techniques to establish the superiority of the proposed method.

Asset correlation based deep reinforcement learning for the portfolio selection

Portfolio selection is an important application of AI in the financial field, which has attracted considerable attention from academia and industry alike. One of the great challenges in this application is modeling the correlation among assets in the portfolio. However, current studies cannot deal well with this challenge because it is difficult to analyze complex nonlinearity in the correlation. This paper proposes a policy network that models the nonlinear correlation by utilizing the self-attention mechanism to better tackle this issue. In addition, a deterministic policy gradient recurrent reinforcement learning method based on Monte Carlo sampling is constructed with the objective function of cumulative return to train the policy network. In most existing reinforcement learning-based studies, the state transition probability is generally regarded as unknown, so the value function of the policy can only be estimated. Based on financial backtest experiments, we analyze that the state transition probability is known in the portfolio, and value function can be directly obtained by sampling, further theoretically proving the optimality of the proposed reinforcement learning method in the portfolio. Finally, the superiority and generality of our approach are demonstrated through comprehensive experiments on the cryptocurrency dataset, S&P 500 stock dataset, and ETF dataset.

Self-attention based deep direct recurrent reinforcement learning with hybrid loss for trading signal generation

Algorithmic trading based on machine learning has the advantage of using intrinsic features and embedded causality in complex stock price time series. We propose a novel algorithmic trading model based on recurrent reinforcement learning, optimized for making consecutive trading signals. This paper elaborates on how temporal features from complex observation are optimally extracted to maximize the expected rewards of the reinforcement learning model. Our model incorporates the hybrid learning loss to allow sequences of hidden features for reinforcement learning to contain the original state’s characteristics fully. The self-attention mechanism is also introduced to our model for learning the temporal importance of the hidden representation series, which helps the reinforcement learning model to be aware of temporal dependence for its decision-making. In this paper, we verify the effectiveness of proposed model using some major market indices and the representative stocks in each sector of S&P500. The augmented structure that we propose has a significant dominance on trading performance. Our proposed model, self-attention based deep direct recurrent reinforcement learning with hybrid loss (SA-DDR-HL), shows superior performance over well-known baseline benchmark models, including machine learning and time series models.

Error-related potential-based shared autonomy via deep recurrent reinforcement learning

Objective. Error-related potential (ErrP)-based brain–computer interfaces (BCIs) have received a considerable amount of attention in the human–robot interaction community. In contrast to traditional BCI, which requires continuous and explicit commands from an operator, ErrP-based BCI leverages the ErrP, which is evoked when an operator observes unexpected behaviours from the robot counterpart. This paper proposes a novel shared autonomy model for ErrP-based human–robot interaction. Approach. We incorporate ErrP information provided by a BCI as useful observations for an agent and formulate the shared autonomy problem as a partially observable Markov decision process. A recurrent neural network-based actor-critic model is used to address the uncertainty in the ErrP signal. We evaluate the proposed framework in a simulated human-in-the-loop robot navigation task with both simulated users and real users. Main results. The results show that the proposed ErrP-based shared autonomy model enables an autonomous robot to complete navigation tasks more efficiently. In a simulation with 70% ErrP accuracy, agents completed the task 14.1% faster than in the no ErrP condition, while with real users, agents completed the navigation task 14.9% faster. Significance. The evaluation results confirmed that the shared autonomy via deep recurrent reinforcement learning is an effective way to deal with uncertain human feedback in a complex human–robot interaction task.

Model-Free Deep Recurrent Q-Network Reinforcement Learning for Quantum Circuit Architectures Design

Artificial intelligence (AI) technology leads to new insights into the manipulation of quantum systems in the Noisy Intermediate-Scale Quantum (NISQ) era. Classical agent-based artificial intelligence algorithms provide a framework for the design or control of quantum systems. Traditional reinforcement learning methods are designed for the Markov Decision Process (MDP) and, hence, have difficulty in dealing with partially observable or quantum observable decision processes. Due to the difficulty of building or inferring a model of a specified quantum system, a model-free-based control approach is more practical and feasible than its counterpart of a model-based approach. In this work, we apply a model-free deep recurrent Q-network (DRQN) reinforcement learning method for qubit-based quantum circuit architecture design problems. This paper is the first attempt to solve the quantum circuit design problem from the recurrent reinforcement learning algorithm, while using discrete policy. Simulation results suggest that our long short-term memory (LSTM)-based DRQN method is able to learn quantum circuits for entangled Bell–Greenberger–Horne–Zeilinger (Bell–GHZ) states. However, since we also observe unstable learning curves in experiments, suggesting that the DRQN could be a promising method for AI-based quantum circuit design application, more investigation on the stability issue would be required.

Online Partial Offloading and Task Scheduling in SDN-Fog Networks With Deep Recurrent Reinforcement Learning

Smart industries enabling automation and data exchange in manufacturing technologies demanding real-time processing, nearby storage, and reliability, all of which can be satisfied by the fog computing architecture. With the emergence of smart devices coupled with a diverse range of application requirements, it is essential to have an intelligent fog network where intelligence is spread across all network segments, taking network nodes self-aware and self-decision making. In fog networks, an optimal distribution decision faces challenges due to uncertainties associated with user workload and available resources at the fog nodes and also the wide range of node’s computing power. Given this challenge, a computational offloading and CPU resource scheduling method for minimizing energy consumption is proposed. To investigate the characteristics for offloading and optimizing their allocation, we consider two types of tasks, namely, offloadable and nonoffloadable tasks. The independent fog nodes adopt the same strategy without prior knowledge of the dynamic statistics and global observations, aiming to maximize a common goal with cooperative behaviors. Then, the deep recurrent <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -network (DRQN) is applied to deal with the partial-observability from limited information. The proposed DRQN-based method requires comparatively less computational complexity than the conventional <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning algorithm. The simulation results show that the proposed method can effectively deal with both transmission and CPU energy consumptions while guaranteeing convergence in a limited time.

Context-aware reinforcement learning for course recommendation

Online course recommendation is an extremely relevant ingredient for the efficiency of e-learning. The current recommendation methods cannot guarantee the effectiveness and accuracy of course recommendation, especially when a user has enrolled in many different courses. Because these methods fail to distinguish the most relevant historical courses, which can contribute to predicting the target course that indeed reflects the user’s interests from her sequential learning behaviors. In this paper, we propose a context-aware reinforcement learning method, named Hierarchical and Recurrent Reinforcement Learning (HRRL), to efficiently reconstruct user profiles for course recommendation. The key ingredient of our scheme is the novel interaction between an attention-based recommendation model and a profile reviser with Recurrent Reinforcement Learning (RRL) that exploits temporal context. To this aim, a contextual policy gradient with approximation is proposed for RRL. By employing RRL in hierarchical tasks of revising user profiles, the proposed HRRL model enables reliable convergence in revising policy learning and improves the recommendation accuracy. We demonstrate the effectiveness of our proposed method by experiments on two open online courses datasets. Empirical results show that HRRL significantly outperforms state-of-the-art baselines.

Outperforming algorithmic trading reinforcement learning systems: A supervised approach to the cryptocurrency market

The interdisciplinary relationship between machine learning and financial markets has long been a theme of great interest among both research communities. Recently, reinforcement learning and deep learning methods gained prominence in the active asset trading task, aiming to achieve outstanding performances compared with classical benchmarks, such as the Buy and Hold strategy. This paper explores both the supervised learning and reinforcement learning approaches applied to active asset trading, drawing attention to the benefits of both approaches. This work extends the comparison between the supervised approach and reinforcement learning by using state-of-the-art strategies with both techniques. We propose adopting the ResNet architecture, one of the best deep learning approaches for time series classification, into the ResNet-LSTM actor (RSLSTM-A). We compare RSLSTM-A against classical and recent reinforcement learning techniques, such as recurrent reinforcement learning, deep Q-network, and advantage actor–critic. We simulated a currency exchange market environment with the price time series of the Bitcoin, Litecoin, Ethereum, Monero, Nxt, and Dash cryptocurrencies to run our tests. We show that our approach achieves better overall performance, confirming that supervised learning can outperform reinforcement learning for trading. We also present a graphic representation of the features extracted from the ResNet neural network to identify which type of characteristics each residual block generates.

Double Deep Recurrent Reinforcement Learning for Centralized Dynamic Multichannel Access

We consider the problem of dynamic multichannel access for transmission maximization in multiuser wireless communication networks. The objective is to find a multiuser strategy that maximizes global channel utilization with a low collision in a centralized manner without any prior knowledge. Obtaining an optimal solution for centralized dynamic multichannel access is an extremely difficult problem due to the large-state and large-action space. To tackle this problem, we develop a centralized dynamic multichannel access framework based on double deep recurrent Q-network. The centralized node first maps current state directly to channel assignment actions, which can overcome prohibitive computation compared with reinforcement learning. Then, the centralized node can be easy to select multiple channels by maximizing the sum of value functions based on a trained neural network. Finally, the proposed method avoids collisions between secondary users through centralized allocation policy.