Q-learning Agent Research Articles

Wildfire Front Monitoring with Multiple UAVs using Deep Q-Learning

Wildfires destroy thousands of hectares every summer all over the globe. To provide an effective response and to mitigate wildfires impact, firefighters require a real-time monitoring of the fire front. This paper proposes a cooperative reinforcement learning (RL) framework that allows a team of autonomous unmanned aerial vehicles (UAVs) to learn how to monitor a fire front. In the literature, independent Q-learners were proposed to solve a wildfire monitoring task with two UAVs. Here we propose a framework that can be easily extended to a larger number of UAVs. Our framework builds on two methods: multiple single trained Q-learning agents (MSTA) and value decomposition networks (VDN). MSTA trains a single UAV controller, which is then "copied" to each of the UAVs in the team. In contrast, VDN trains agents to learn how to cooperate. We benchmarked in simulations our two considered methods – MSTA and VDN – against two state-of-the-art approaches: independent Q-learners and a joint Q-learner. Simulation results show that our considered methods outperform state-of-the-art approaches in a wildfire front monitoring task with up to 9 fixed-wing and multi-copter UAVs.

Multi-DQN: An ensemble of Deep Q-learning agents for stock market forecasting

The stock market forecasting is one of the most challenging application of machine learning, as its historical data are naturally noisy and unstable. Most of the successful approaches act in a supervised manner, labeling training data as being of positive or negative moments of the market. However, training machine learning classifiers in such a way may suffer from over-fitting, since the market behavior depends on several external factors like other markets trends, political events, etc. In this paper, we aim at minimizing such problems by proposing an ensemble of reinforcement learning approaches which do not use annotations (i.e. market goes up or down) to learn, but rather learn how to maximize a return function over the training stage. In order to achieve this goal, we exploit a Q-learning agent trained several times with the same training data and investigate its ensemble behavior in important real-world stock markets. Experimental results in intraday trading indicate better performance than the conventional Buy-and-Hold strategy, which still behaves well in our setups. We also discuss qualitative and quantitative analyses of these results.

A Q-learning agent for automated trading in equity stock markets

Trading strategies play a vital role in Algorithmic trading, a computer program that takes and executes automated trading decisions in the stock market. The conventional wisdom is that the same trading strategy is not profitable for all stocks all the time. The selection of a trading strategy for the stock at a particular time instant is the major research problem in the stock market trading. An optimal dynamic trading strategy generated from the current pattern of the stock price trend can attempt to solve this problem. Reinforcement Learning can find this optimal dynamic trading strategy by interacting with the actual stock market as its environment. The representation of the state of the environment is crucial for performance. We have proposed two different ways to represent the discrete states of the environment. In this work, we trained the trading agent using the Q-learning algorithm of Reinforcement Learning to find optimal dynamic trading strategies. We experimented with the two proposed models on real stock market data from the Indian and American stock markets. The proposed models outperformed the Buy-and-Hold and Decision-Tree based trading strategy in terms of profitability.

Reinforcement Learning using EEG signals for Therapeutic Use of Music in Emotion Management.

Prolonged influence of negative emotions can result in clinical depression or anxiety, and while many prescribed techniques exist, music therapy approaches, coupled with psychotherapy, have shown to help lower depressive symptoms, supplementing traditional treatment approaches. Identifying the appropriate choice of music, therefore, is of utmost importance. Selecting appropriate playlists, however, are challenged by user feedback that may inadvertently select songs that amplify the negative effects. Therefore, this work uses electroencephalogram (EEG) that automatically identifies the emotional impact of music and trains a reinforcement-learning approach to identify an adaptive personalized playlist of music to lead to improved emotional states. This work uses data from 32 users, collected in the publicly available DEAP dataset, to select songs for users that guide them towards joyful emotional states. Using a domain-specific reward-shaping function, a Q-learning agent is able to correctly guide a majority of users to the target emotional states, represented in a common emotion wheel. The average angular error of all users is 57°, with a standard deviation of 2.8 and the target emotional state is achieved.Clinical relevance- Music therapy for improving clinical depression and anxiety can be supplemented by additional emotion-guided music decisions in remote and personal settings by using automated techniques to capture emotional state and identify music that best guides users to target joyful states.

Intelligent virtual agent, learning how to reach a goal by making the least number of compromises

The learning process in the Q-learning algorithm is characterized by maximizing a single, numerical reward signal. However, there are tasks for which the requirements toward the way to reach a goal are complex. This paper proposes a modification to the Q-learning algorithm. In order to make the Q-learning agent find the optimal path to the goal by meeting particular complex criteria, the use of measures model (a model of environment criteria), represented as a new memory matrix, is introduced. If the goal cannot be reached by following the pre-set criteria, the learning agent can compromise a given criterion. The agent makes the least possible number of tradeoffs in order to reach the goal. If the criteria are arranged by their level of importance, then the agent can choose more in number and more acceptable compromises. The aim of the modification is to empower the learning agent to control the way of reaching a goal. The modified algorithm has been applied to training smart shopping-cart learning agents. The tests show improvement in their behaviour.

Reinforced Deterministic and Probabilistic Load Forecasting via $Q$ -Learning Dynamic Model Selection

Both deterministic and probabilistic load forecasting (DLF and PLF) are of critical importance to reliable and economical power system operations. However, most of the widely used statistical machine learning (ML) models are trained by optimizing the global performance, without considering the local behaviour. This paper develops a two-step short-term load forecasting (STLF) model with Q-learning based dynamic model selection (QMS), which provides reinforced deterministic and probabilistic load forecasts (DLFs and PLFs). First, a deterministic forecasting model pool (DMP) and a probabilistic forecasting model pool (PMP) are built based on 10 state-of-the-art ML DLF models and 4 predictive distribution models. Then, in the first-step of each time stamp, a Q-learning agent selects the locally-best DLF model from the DMP to provide an enhanced DLF. At last, the DLF is input to the best PLF model selected from the PMP by another Q-learning agent to perform PLF in the second-step. Numerical simulations on two-year weather and smart meter data show that the developed STLF-QMS method improves DLF and PLF by 50% and 60%, respectively, compared to the state-of-the-art benchmarks.

A Q-Learning Approach With Collective Contention Estimation for Bandwidth-Efficient and Fair Access Control in IEEE 802.11p Vehicular Networks

Vehicular ad hoc networks (VANETs) are wireless networks formed of moving vehicle stations, that enable safety-related packet exchanges among them. Their infrastructure-less, unbounded nature allows the formation of dense networks that present a channel sharing issue, which is harder to tackle than in conventional WLANs, due to fundamental differences of the protocol stack. Optimizing channel access strategies is important for the efficient usage of the available wireless bandwidth and the successful deployment of VANETs. We present a Q-learning-based approach to wirelessly network a big number of vehicles and enable the efficient exchange of data packets among them. More specifically, this work focuses on a IEEE 802.11p-compatible contention-based medium access control protocol for efficiently sharing the wireless channel among multiple vehicular stations. The stations feature algorithms that “learn” how to act optimally in a network in order to maximize their achieved packet delivery and minimise bandwidth wastage. Additionally, via a collective contention estimation mechanism, which we embed on the Q-learning agent, faster convergence, higher throughput, and short-term fairness are achieved.

Automatic Design of Deep Networks with Neural Blocks

In recent years, deep neural networks (DNNs) have achieved great successes in many areas, such as cognitive computation, pattern recognition, and computer vision. Although many hand-crafted deep networks have been proposed in the literature, designing a well-behaved neural network for a specific application requires high-level expertise yet. Hence, the automatic architecture design of DNNs has become a challenging and important problem. In this paper, we propose a new reinforcement learning method, whose action policy is to select neural blocks and construct deep networks. We define the action search space with three types of neural blocks, i.e., dense block, residual block, and inception-like block. Additionally, we have also designed several variants for the residual and inception-like blocks. The optimal network is automatically learned by a Q-learning agent, which is iteratively trained to generate well-performed deep networks. To evaluate the proposed method, we have conducted experiments on three datasets, MNIST, SVHN, and CIFAR-10, for image classification applications. Compared with existing hand-crafted and auto-generated neural networks, our auto-designed neural network delivers promising results. Moreover, the proposed reinforcement learning algorithm for deep networks design only runs on one GPU, demonstrating much higher efficiency than most of the previous deep network search approaches.

Learning to steer nonlinear interior-point methods

Interior-point or barrier methods handle nonlinear programs by sequentially solving barrier subprograms with a decreasing sequence of barrier parameters. The specific barrier update rule strongly influences the theoretical convergence properties as well as the practical efficiency. While many global and local convergence analyses consider a monotone update that decreases the barrier parameter for every approximately solved subprogram, computational studies show a superior performance of more adaptive strategies. In this paper we interpret the adaptive barrier update as a reinforcement learning task. A deep Q-learning agent is trained by both imitation and random action selection. Numerical results based on an implementation within the nonlinear programming solver WORHP show that the agent successfully learns to steer the barrier parameter and additionally improves WORHP’s performance on the CUTEst test set.

An improved fuzzy ARTMAP and Q-learning agent model for pattern classification

The Fuzzy ARTMAP (FAM) network is an online supervised neural network that operates by computing the similarity level between the new sample and those prototype nodes stored in its network against a threshold. In our previous study, we have developed a multi-agent system consisting of an ensemble of FAM networks and Q-learning, known as QMACS, for data classification. In this paper, an Improved QMACS (IQMACS) model with trust measurement using a combination of Q-learning and Bayesian formalism is proposed. A number of benchmark and real-world problems, i.e., motor fault detection and human motion detection, are conducted to evaluate the effectiveness of IQMACS. Statistical features are extracted from real-world case studies and utilized for classification with IQMACS, QMACS, and their constituents. The experimental results indicate that IQMACS produces better classification performance by combining the outcomes of its constituents as compared with those of QMACS and other related methods.

Reinforcement Learning for Real-Time Optimization in NB-IoT Networks

NarrowBand Internet of Things (NB-IoT) is an emerging cellular-based technology that offers a range of flexible configurations for massive IoT radio access from groups of devices with heterogeneous requirements. A configuration specifies the amount of radio resource allocated to each group of devices for random access and for data transmission. Assuming no knowledge of the traffic statistics, there exists an important challenge in “how to determine the configuration that maximizes the long-term average number of served IoT devices at each transmission time interval (TTI) in an online fashion.” Given the complexity of searching for optimal configuration, we first develop real-time configuration selection based on the tabular Q-learning (tabular-Q), the linear approximation-based Q-learning (LA-Q), and the deep neural network-based Q-learning (DQN) in the single-parameter single-group scenario. Our results show that the proposed reinforcement learning-based approaches considerably outperform the conventional heuristic approaches based on load estimation (LE-URC) in terms of the number of served IoT devices. This result also indicates that LA-Q and DQN can be good alternatives for tabular-Q to achieve almost the same performance with much less training time. We further advance LA-Q and DQN via actions aggregation (AA-LA-Q and AA-DQN) and via cooperative multi-agent learning (CMA-DQN) for the multi-parameter multi-group scenario, thereby solve the problem that Q-learning agents do not converge in high-dimensional configurations. In this scenario, the superiority of the proposed Q-learning approaches over the conventional LE-URC approach significantly improves with the increase of configuration dimensions, and the CMA-DQN approach outperforms the other approaches in both throughput and training efficiency.

Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control

Reinforcement learning (RL) is a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, and deep neural networks further enhance its learning power. However, the centralized RL is infeasible for large-scale ATSC due to the extremely high dimension of the joint action space. The multi-agent RL (MARL) overcomes the scalability issue by distributing the global control to each local RL agent, but it introduces new challenges: now, the environment becomes partially observable from the viewpoint of each local agent due to limited communication among agents. Most existing studies in MARL focus on designing efficient communication and coordination among traditional Q-learning agents. This paper presents, for the first time, a fully scalable and decentralized MARL algorithm for the state-of-the-art deep RL agent, advantage actor critic (A2C), within the context of ATSC. In particular, two methods are proposed to stabilize the learning procedure, by improving the observability and reducing the learning difficulty of each local agent. The proposed multi-agent A2C is compared against independent A2C and independent Q-learning algorithms, in both a large synthetic traffic grid and a large real-world traffic network of Monaco city, under simulated peak-hour traffic dynamics. The results demonstrate its optimality, robustness, and sample efficiency over the other state-of-the-art decentralized MARL algorithms.

Fuzzy Q-Learning Agent for Online Tuning of PID Controller for DC Motor Speed Control

This paper proposes a hybrid Zeigler-Nichols (Z-N) reinforcement learning approach for online tuning of the parameters of the Proportional Integral Derivative (PID) for controlling the speed of a DC motor. The PID gains are set by the Z-N method, and are then adapted online through the fuzzy Q-Learning agent. The fuzzy Q-Learning agent is used instead of the conventional Q-Learning, in order to deal with the continuous state-action space. The fuzzy Q-Learning agent defines its state according to the value of the error. The output signal of the agent consists of three output variables, in which each one defines the percentage change of each gain. Each gain can be increased or decreased from 0% to 50% of its initial value. Through this method, the gains of the controller are adjusted online via the interaction of the environment. The knowledge of the expert is not a necessity during the setup process. The simulation results highlight the performance of the proposed control strategy. After the exploration phase, the settling time is reduced in the steady states. In the transient states, the response has less amplitude oscillations and reaches the equilibrium point faster than the conventional PID controller.

BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems

We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems. Our agents explore via Thompson sampling, drawing Monte Carlo samples from a Bayes-by-Backprop neural network. Our algorithm learns much faster than common exploration strategies such as ε-greedy, Boltzmann, bootstrapping, and intrinsic-reward-based ones. Additionally, we show that spiking the replay buffer with experiences from just a few successful episodes can make Q-learning feasible when it might otherwise fail.

Machine Learning for Trading

In multi-period trading with realistic market impact, determining the dynamic trading strategy that optimizes expected utility of final wealth is a hard problem. In this paper we show that, with an appropriate choice of the reward function, reinforcement learning techniques (specifically, Q-learning) can successfully handle the risk-averse case. We provide a proof of concept in the form of a simulated market which permits a statistical arbitrage even with trading costs. The Q-learning agent finds and exploits this arbitrage.

How network monitoring and reinforcement learning can improve tcp fairness in wireless multi-hop networks

Wireless mesh network (WMN) is an emerging technology for the last-mile Internet access. Despite extensive research and the commercial implementations of WMNs, there are still serious fairness issues in the transport layer, where the transmission control protocol (TCP) favors flows with a smaller number of hops to flows with a larger number of hops. TCP unfair behavior is a known anomaly over WMN that attracts much attention in recent years and is the focus of this paper. In this article, we propose a distributed network monitoring mechanism using a cross-layer approach that deploys reinforcement learning techniques (RL) to achieve fair resource allocation for nodes within the wireless mesh setting. In our approach, we deploy Q-learning, a reinforcement learning mechanism, to monitor the dynamics of the network. The Q-learning agent creates a state map of the network based on the medium access control (MAC) parameters and takes actions to enhance TCP fairness and throughput of the starved flows in the network. The proposal creates a distributed cooperative mechanism where each node hosting a TCP source monitors the network and adjusts its TCP parameters based on the network dynamics. Extensive simulation results and testbed analysis demonstrate that the proposed method significantly improves the TCP fairness in a multi-hop wireless environment.

Queue Based Q-Learning for Efficient Resource Provisioning in Cloud Data Centers

Cloud Computing is a novel paradigm that offers virtual resources on demand through internet. Due to rapid demand to cloud resources, it is difficult to estimate the user's demand. As a result, the complexity of resource provisioning increases, which leads to the requirement of an adaptive resource provisioning. In this paper, the authors address the problem of efficient resource provisioning through Queue based Q-learning algorithm using reinforcement learning agent. Reinforcement learning has been proved in various domains for automatic control and resource provisioning. In the absence of complete environment model, reinforcement learning can be used to define optimal allocation policies. The proposed Queue based Q-learning agent analyses the CPU utilization of all active Virtual Machines (VMs) and detects the least loaded virtual machine for resource provisioning. It detects the least loaded virtual machines through Inter Quartile Range. Using the queue size of virtual machines it looks ahead by one time step to find the optimal virtual machine for provisioning.

Cognitively inspired reinforcement learning architecture and its application to giant-swing motion control

Many algorithms and methods in artificial intelligence or machine learning were inspired by human cognition. As a mechanism to handle the exploration–exploitation dilemma in reinforcement learning, the loosely symmetric (LS) value function that models causal intuition of humans was proposed (Shinohara et al., 2007). While LS shows the highest correlation with causal induction by humans, it has been reported that it effectively works in multi-armed bandit problems that form the simplest class of tasks representing the dilemma. However, the scope of application of LS was limited to the reinforcement learning problems that have K actions with only one state (K-armed bandit problems). This study proposes LS-Q learning architecture that can deal with general reinforcement learning tasks with multiple states and delayed reward. We tested the learning performance of the new architecture in giant-swing robot motion learning, where uncertainty and unknown-ness of the environment is huge. In the test, the help of ready-made internal models or functional approximation of the state space were not given. The simulations showed that while the ordinary Q-learning agent does not reach giant-swing motion because of stagnant loops (local optima with low rewards), LS-Q escapes such loops and acquires giant-swing. It is confirmed that the smaller number of states is, in other words, the more coarse-grained the division of states and the more incomplete the state observation is, the better LS-Q performs in comparison with Q-learning. We also showed that the high performance of LS-Q depends comparatively little on parameter tuning and learning time. This suggests that the proposed method inspired by human cognition works adaptively in real environments.

Mobile Robot Navigation Using ARTQL Algorithm with Novelty Driven Mechanism

Q-learning algorithm is usually used for traditional mobile robot navigation. The traditional Q-learning methods have the problem of dimension disaster, which may be produced by applying Q-learning to intelligent system of continuous state-space. Besides, the learning activity and efficiency are low. In order to solve these problems, a new method called ARTQL is proposed, which combined ART2 network with the traditional Q-learning algorithm. Then, a learning mechanism called novelty driven is proposed to lead the ARTQL algorithm to learn more actively and efficiently. Through the ARTQL with novelty driven mechanism algorithm, Q-learning Agent in view of the duty which needs to complete to learn an appropriate incremental clustering of state-space model, so Agent can carry out decision-making and a two-tiers online learning of state-space model cluster in unknown environment, without any priori knowledge, through interaction with the environment unceasingly alternately to improve the control strategies, increase the learning accuracy, activity and efficiency. Finally through the mobile robot navigation simulation experiments, we show that, using the proposed algorithm, mobile robot can improve its navigation performance continuously by interactive learning with the environment with high autonomous.

Achieving autonomous power management using reinforcement learning

System level power management must consider the uncertainty and variability that come from the environment, the application and the hardware. A robust power management technique must be able to learn the optimal decision from past events and improve itself as the environment changes. This article presents a novel on-line power management technique based on model-free constrained reinforcement learning (Q-learning). The proposed learning algorithm requires no prior information of the workload and dynamically adapts to the environment to achieve autonomous power management. We focus on the power management of the peripheral device and the microprocessor, two of the basic components of a computer. Due to their different operating behaviors and performance considerations, these two types of devices require different designs of Q-learning agent. The article discusses system modeling and cost function construction for both types of Q-learning agent. Enhancement techniques are also proposed to speed up the convergence and better maintain the required performance (or power) constraint in a dynamic system with large variations. Compared with the existing machine learning based power management techniques, the Q-learning based power management is more flexible in adapting to different workload and hardware and provides a wider range of power-performance tradeoff.