Advanced Reinforcement Learning Research Articles

DM-CSAT: a LTE-U/Wi-Fi coexistence solution based on reinforcement learning

Recent literature demonstrated promising results of Long-Term Evolution (LTE) deployments over unlicensed bands when coexisting with Wi-Fi networks via the Duty-Cycle (DC) approach. However, it is known that performance in coexistence is strongly dependent on traffic patterns and on the duty-cycle ON–OFF rate of LTE. Most DC solutions rely on static coexistence parameters configuration, hence real-life performance in dynamically varying scenarios might be affected. Advanced reinforcement learning techniques may be used to adjust DC parameters towards efficient coexistence, and we propose a Q-learning Carrier-Sensing Adaptive Transmission mechanism which adapts LTE duty-cycle ON–OFF time ratio to the transmitted data rate, aiming at maximizing the Wi-Fi and LTE-Unlicensed (LTE-U) aggregated throughput. The problem is formulated as a Markov decision process, and the Q-learning solution for finding the best LTE-U ON–OFF time ratio is based on the Bellman’s equation. We evaluate the performance of the proposed solution for different traffic load scenarios using the ns-3 simulator. Results demonstrate the benefits from the adaptability to changing circumstances of the proposed method in terms of Wi-Fi/LTE aggregated throughput, as well as achieving a fair coexistence.

SCARL: Attentive Reinforcement Learning-Based Scheduling in a Multi-Resource Heterogeneous Cluster

Advanced reinforcement learning (RL) technologies have recently increased the opportunity for automating several tasks in cluster management at scale by exploiting repetitive logs of cluster operation and building a learning model for resource allocation and job scheduling. Yet, this trend of adopting RL in the domain of cluster management has not fully addressed the diversity and heterogeneity of jobs and machines in modern cluster environments. In this paper, we present an RL-based scheduler for a multi-resource cluster, namely SCARL (SCheduler with Attentive Reinforcement Learning), concentrating on intricate cluster operating conditions with different resource requirements and capabilities. Specifically, we employ attentive embedding and factored-action scheduling that together efficiently incorporate time-varying interdependency of jobs and machines in RL processing; they enable an end-to-end scalable policy for scheduling diverse jobs on heterogeneous machines. To the best of our knowledge, we are the first to employ attention mechanism in RL-based cluster resource management. Through experiments, we demonstrate that our approach is competitive with existing heuristic methods under various cluster simulation configurations, e.g., an average 9.2 % enhancement in slowdown over the shortest job first algorithm. Additionally, the approach yields stable performance with our test cluster for running synthetic workloads based on real traces.

Deep-Reinforcement-Learning-Based Optimization for Cache-Enabled Opportunistic Interference Alignment Wireless Networks

Both caching and interference alignment (IA) are promising techniques for next-generation wireless networks. Nevertheless, most of the existing works on cache-enabled IA wireless networks assume that the channel is invariant, which is unrealistic considering the time-varying nature of practical wireless environments. In this paper, we consider realistic time-varying channels. Specifically, the channel is formulated as a finite-state Markov channel (FSMC). The complexity of the system is very high when we consider realistic FSMC models. Therefore, in this paper, we propose a novel deep reinforcement learning approach, which is an advanced reinforcement learning algorithm that uses a deep $Q$ network to approximate the $Q$ value-action function. We use Google TensorFlow to implement deep reinforcement learning in this paper to obtain the optimal IA user selection policy in cache-enabled opportunistic IA wireless networks. Simulation results are presented to show that the performance of cache-enabled opportunistic IA networks in terms of the network's sum rate and energy efficiency can be significantly improved by using the proposed approach.

High-Frequency Trading Meets Online Learning

We propose an optimization framework for market-making in a limit order book, based on the theory of stochastic approximation. The idea is to take advantage of the iterative nature of the process of updating bid and ask quotes in order to make the algorithm optimize its strategy on a trial-and-error basis (i.e., online learning) using a variation of the stochastic gradient-descent algorithm. An advantage of this approach is that the exploration of the system by the algorithm is performed in run-time, so explicit specifications of price dynamics are not necessary, as is the case in the stochastic-control approach [(Gueant et al., 2013, Dealing with the Inventory Risk: A Solution to the Market Making Problem, Mathematics and Financial Economics 7(4), 477–507)]. For price/liquidity modeling, we consider a discrete-time variant of the Avellaneda–Stoikov model [(Avellaneda, M. and S. Stoikov, 2008, Liquidation in Limit Order Books with Controlled Intensity, Mathematical Finance 24(4), 627–650)] similar to its developent in the paper of Laruelle et al. [(Laruelle et al., 2013, Optimal Posting Price of Limit Orders: Learning by trading, Mathematics and Financial Economics 7(3), 359–403)] in the context of optimal liquidation tactics. Our aim is to set the ground for more advanced reinforcement learning techniques and to argue that the rationale of our method is generic enough to be extended to other classes of trading problems besides market-making.

Role of the medial prefrontal cortex in impaired decision making in juvenile attention-deficit/hyperactivity disorder.

Attention-deficit/hyperactivity disorder (ADHD) has been associated with deficient decision making and learning. Models of ADHD have suggested that these deficits could be caused by impaired reward prediction errors (RPEs). Reward prediction errors are signals that indicate violations of expectations and are known to be encoded by the dopaminergic system. However, the precise learning and decision-making deficits and their neurobiological correlates in ADHD are not well known. To determine the impaired decision-making and learning mechanisms in juvenile ADHD using advanced computational models, as well as the related neural RPE processes using multimodal neuroimaging. Twenty adolescents with ADHD and 20 healthy adolescents serving as controls (aged 12-16 years) were examined using a probabilistic reversal learning task while simultaneous functional magnetic resonance imaging and electroencephalogram were recorded. Learning and decision making were investigated by contrasting a hierarchical Bayesian model with an advanced reinforcement learning model and by comparing the model parameters. The neural correlates of RPEs were studied in functional magnetic resonance imaging and electroencephalogram. Adolescents with ADHD showed more simplistic learning as reflected by the reinforcement learning model (exceedance probability, Px = .92) and had increased exploratory behavior compared with healthy controls (mean [SD] decision steepness parameter β: ADHD, 4.83 [2.97]; controls, 6.04 [2.53]; P = .02). The functional magnetic resonance imaging analysis revealed impaired RPE processing in the medial prefrontal cortex during cue as well as during outcome presentation (P < .05, family-wise error correction). The outcome-related impairment in the medial prefrontal cortex could be attributed to deficient processing at 200 to 400 milliseconds after feedback presentation as reflected by reduced feedback-related negativity (ADHD, 0.61 [3.90] μV; controls, -1.68 [2.52] μV; P = .04). The combination of computational modeling of behavior and multimodal neuroimaging revealed that impaired decision making and learning mechanisms in adolescents with ADHD are driven by impaired RPE processing in the medial prefrontal cortex. This novel, combined approach furthers the understanding of the pathomechanisms in ADHD and may advance treatment strategies.