Multi-armed Bandit Framework Research Articles

Spectrum Blind Recovery and Application in Non-uniform Sampling Based Automatic Modulation Classifier

Multi-standard wireless communication radios (MWCRs) capable of digitizing wideband signal to support wide variety of data-intensive services are desired. Limited reconfigurability of the analog front end along with hardware and cost constraints of high-speed analog-to-digital converters have generated significant interest in non-uniform (sub-Nyquist) sampling (NUS) and digital reconstruction-based MWCRs. Existing reconstruction approaches require prior knowledge of sparsity which may not be available in the dynamic spectrum environment. To alleviate this problem, a blind and adaptive reconstruction approach has been proposed in this paper. The proposed approach employs multi-armed Bandit framework to estimate the spectrum occupancy. Simulation results show that the average normalized mean square error of the proposed approach is 10–20% lower than other reconstruction approaches. Next, cumulant and machine learning-based automatic modulation classifier (AMC) is designed to validate the usefulness of the proposed approach in practical applications. Simulation results show that the classification accuracy of NUS-based AMC approaches, uniform sampling-based AMC with increase in signal-to-noise ratio and proposed approach is superior to others. The simulation results are further verified on the proposed universal software radio peripheral testbed in real radio environment. Experimental results demonstrate the close resemblance with simulation results.

Optimizing Infill Drilling Decisions Using Multi-Armed Bandits: Application in a Long-Term, Multi-Element Stockpile

Mining operations face a decision regarding additional drilling several times during their lifetime. The two questions that always arise upon making this decision are whether more drilling is required and, if so, where the additional drill holes should be located. The method presented in this paper addresses both of these questions through an optimization in a multi-armed bandit (MAB) framework. The MAB optimizes the best infill drilling pattern while taking geological uncertainty into account by using multiple conditional simulations for the deposit under consideration. The proposed method is applied to a long-term, multi-element stockpile, which is a part of a gold mining complex. The stockpiles in this mining complex are of particular interest due to difficult-to-meet blending requirements. In several mining periods grade targets of deleterious elements at the processing plant can only be met by using high amounts of stockpiled material. The best pattern is defined in terms of causing the most material type changes for the blocks in the stockpile. Material type changes are the driver for changes in the extraction sequence, which ultimately defines the value of a mining operation. The results of the proposed method demonstrate its practical aspects and its effectiveness towards the optimization of infill drilling schemes.

Decentralized Blind Spectrum Selection in Cognitive Radio Networks Considering Handoff Cost

Due to the spectrum varying nature of cognitive radio networks, secondary users are required to perform spectrum handoffs when the spectrum is occupied by primary users, which will lead to a handoff delay. In this paper, based on the multi-armed bandit framework of medium access in decentralized cognitive radio networks, we investigate blind spectrum selection problem of secondary users whose sensing ability of cognitive radio is limited and the channel statistics are a priori unknown, taking the handoff delay as a fixed handoff cost into consideration. In this scenario, secondary users have to make the choice of either staying foregoing spectrum with low availability or handing off to another spectrum with higher availability. We model the problem and investigate the performance of three representative policies, i.e., ρPRE, SL(K), kth-UCB1. The simulation results show that, despite the inclusion of the fixed handoff cost, these policies achieve the same asymptotic performance as that without handoff cost. Moreover, through comparison of these policies, we found the kth-UCB1 policy has better overall performance.

Efficient Ordered Combinatorial Semi-Bandits for Whole-Page Recommendation

Multi-Armed Bandit (MAB) framework has been successfully applied in many web applications. However, many complex real-world applications that involve multiple content recommendations cannot fit into the traditional MAB setting. To address this issue, we consider an ordered combinatorial semi-bandit problem where the learner recommends S actions from a base set of K actions, and displays the results in S (out of M) different positions. The aim is to maximize the cumulative reward with respect to the best possible subset and positions in hindsight. By the adaptation of a minimum-cost maximum-flow network, a practical algorithm based on Thompson sampling is derived for the (contextual) combinatorial problem, thus resolving the problem of computational intractability.With its potential to work with whole-page recommendation and any probabilistic models, to illustrate the effectiveness of our method, we focus on Gaussian process optimization and a contextual setting where click-through rate is predicted using logistic regression. We demonstrate the algorithms’ performance on synthetic Gaussian process problems and on large-scale news article recommendation datasets from Yahoo! Front Page Today Module.

Multiview Boosting With Information Propagation for Classification.

Multiview learning has shown promising potential in many applications. However, most techniques are focused on either view consistency, or view diversity. In this paper, we introduce a novel multiview boosting algorithm, called Boost.SH, that computes weak classifiers independently of each view but uses a shared weight distribution to propagate information among the multiple views to ensure consistency. To encourage diversity, we introduce randomized Boost.SH and show its convergence to the greedy Boost.SH solution in the sense of minimizing regret using the framework of adversarial multiarmed bandits. We also introduce a variant of Boost.SH that combines decisions from multiple experts for recommending views for classification. We propose an expert strategy for multiview learning based on inverse variance, which explores both consistency and diversity. Experiments on biometric recognition, document categorization, multilingual text, and yeast genomic multiview data sets demonstrate the advantage of Boost.SH (85%) compared with other boosting algorithms like AdaBoost (82%) using concatenated views and substantially better than a multiview kernel learning algorithm (74%).

Multi-armed bandits with application to 5G small cells

Due to the pervasive demand for mobile services, next generation wireless networks are expected to be able to deliver high date rates while wireless resources become more and more scarce. This requires the next generation wireless networks to move towards new networking paradigms that are able to efficiently support resource-demanding applications such as personalized mobile services. Examples of such paradigms foreseen for the emerging fifth generation (5G) cellular networks include very densely deployed small cells and device-to-device communications. For 5G networks, it will be imperative to search for spectrum and energy-efficient solutions to the resource allocation problems that i) are amenable to distributed implementation, ii) are capable of dealing with uncertainty and lack of information, and iii) can cope with users' selfishness. The core objective of this article is to investigate and to establish the potential of multi-armed bandit (MAB) framework to address this challenge. In particular, we provide a brief tutorial on bandit problems, including different variations and solution approaches. Furthermore, we discuss recent applications as well as future research directions. In addition, we provide a detailed example of using an MAB model for energy-efficient small cell planning in 5G networks.

Optimal dynamic resource allocation to prevent defaults

Optimal dynamic resource allocation to prevent defaults

Jamming Bandits—A Novel Learning Method for Optimal Jamming

Can an intelligent jammer learn and adapt to unknown environments in an electronic warfare-type scenario? In this paper, we answer this question in the positive, by developing a cognitive jammer that adaptively and optimally disrupts the communication between a victim transmitter–receiver pair. We formalize the problem using a multiarmed bandit framework where the jammer can choose various physical layer parameters such as the signaling scheme, power level and the on-off/pulsing duration in an attempt to obtain power efficient jamming strategies. We first present online learning algorithms to maximize the jamming efficacy against static transmitter–receiver pairs and prove that these algorithms converge to the optimal (in terms of the error rate inflicted at the victim and the energy used) jamming strategy. Even more importantly, we prove that the rate of convergence to the optimal jamming strategy is sublinear, i.e., the learning is fast in comparison to existing reinforcement learning algorithms, which is particularly important in dynamically changing wireless environments. Also, we characterize the performance of the proposed bandit-based learning algorithm against multiple static and adaptive transmitter–receiver pairs.

A Cost-Sensitive Decision Tree Learning Algorithm Based on a Multi-Armed Bandit Framework

This paper develops a new algorithm for inducing cost-sensitive decision trees that is inspired by the multi-armed bandit problem, in which a player in a casino has to decide which slot machine (bandit) from a selection of slot machines is likely to pay out the most. Game Theory proposes a solution to this multi-armed bandit problem by using a process of exploration and exploitation in which reward is maximized. This paper utilizes these concepts to develop a new algorithm by viewing the rewards as a reduction in costs, and utilizing the exploration and exploitation techniques so that a compromise between decisions based on accuracy and decisions based on costs can be found. The algorithm employs the notion of lever pulls in the multi-armed bandit game to select the attributes during decision tree induction, using a look-ahead methodology to explore potential attributes and exploit the attributes which maximizes the reward. The new algorithm is evaluated on fifteen datasets and compared to six well-known algorithms J48, EG2, MetaCost, AdaCostM1, ICET and ACT. The results obtained show that the new multi-armed based algorithm can produce more cost-effective trees without compromising accuracy. The paper also includes a critical appraisal of the limitations of the new algorithm and proposes avenues for further research.

Joint Channel Selection and Power Control in Infrastructureless Wireless Networks: A Multiplayer Multiarmed Bandit Framework

This paper deals with the problem of efficient resource allocation in dynamic infrastructureless wireless networks. Assuming a reactive interference-limited scenario, each transmitter is allowed to select one frequency channel (from a common pool) together with a power level at each transmission trial; hence, for all transmitters, not only the fading gain, but also the number of interfering transmissions and their transmit powers are varying over time. Due to the absence of a central controller and time-varying network characteristics, it is highly inefficient for transmitters to acquire global channel and network knowledge. Therefore a reasonable assumption is that transmitters have no knowledge of fading gains, interference, and network topology. Each transmitting node selfishly aims at maximizing its average reward (or minimizing its average cost), which is a function of the action of that specific transmitter as well as those of all other transmitters. This scenario is modeled as a multi-player multi-armed adversarial bandit game, in which multiple players receive an a priori unknown reward with an arbitrarily time-varying distribution by sequentially pulling an arm, selected from a known and finite set of arms. Since players do not know the arm with the highest average reward in advance, they attempt to minimize their so-called regret, determined by the set of players' actions, while attempting to achieve equilibrium in some sense. To this end, we design in this paper two joint power level and channel selection strategies. We prove that the gap between the average reward achieved by our approaches and that based on the best fixed strategy converges to zero asymptotically. Moreover, the empirical joint frequencies of the game converge to the set of correlated equilibria. We further characterize this set for two special cases of our designed game.

Response-adaptive designs for clinical trials: Simultaneous learning from multiple patients

Response-adaptive designs for clinical trials: Simultaneous learning from multiple patients

Frequency Diversity-Aware Wi-Fi Using OFDM-Based Bloom Filters

With the increasing move towards wide band operation in recent Wi-Fi networks, the frequency diversity awareness has become critical for throughput optimization. To exploit frequency diversity in Wi-Fi channels, the access point should measure the channel quality and coordinate the channel contention for all the stations. However, there is a tradeoff between achieving the frequency diversity gain and sustaining protocol efficiency because the channel estimation and coordination consume time and frequency resource that ideally should be used for data transfer. In this paper, we present Diversity-aware Wi-Fi (D-Fi), a novel PHY/MAC protocol, that efficiently exploits frequency diversity. In particular, D-Fi leverages an OFDM-based Bloom filter that synergistically integrates two operations: (i) the channel quality estimation and (ii) the contention based channel allocation. D-Fi also employs a machine learning (ML) method to resolve the false-positive ambiguity caused by the Bloom filter. Furthermore, we develop a decentralized algorithm, called $K\epsilon$ -greedy , based on the Multi-Armed Bandit (MAB) framework, so that it achieves sub-optimal performance by studying the gain for exploring new channel quality information. We implement the prototype of D-Fi on the USRP/GNURadio to validate the feasibility of our work. The experiments and trace-driven simulations show that D-Fi provides up to 3 $\times$ throughput improvement compared to the existing solutions.

CrowdUtility: A Recommendation System for Crowdsourcing Platforms

Crowd workers exhibit varying work patterns, expertise, and quality leading to wide variability in the performance of crowdsourcing platforms. The onus of choosing a suitable platform to post tasks is mostly with the requester, often leading to poor guarantees and unmet requirements due to the dynamism in performance of crowd platforms. Towards this end, we demonstrate CrowdUtility, a statistical modelling based tool for evaluating multiple crowdsourcing platforms and recommending a platform that best suits the requirements of the requester. CrowdUtility uses an online Multi-Armed Bandit framework, to schedule tasks while optimizing platform performance. We demonstrate an end-to end system starting from requirements specification, to platform recommendation, to real-time monitoring.

Learning State Selection for Reconfigurable Antennas: A Multi-Armed Bandit Approach

Reconfigurable antennas are capable of dynamically re-shaping their radiation patterns in response to the needs of a wireless link or a network. In order to utilize the benefits of reconfigurable antennas, selecting an optimal antenna state for communication is essential and depends on the availability of full channel state information for all the available antenna states. We consider the problem of reconfigurable antenna state selection in a single user MIMO system. We first formulate the state selection as a multi-armed bandit problem that aims to optimize arbitrary link quality metrics. We then show that by using online learning under a multi-armed bandit framework, a sequential decision policy can be employed to learn optimal antenna states without instantaneous full CSI and without a priori knowledge of wireless channel statistics. Our objective is to devise an adaptive state selection technique when the channels corresponding to all the states are not directly observable and compare our results against the case of a known model or genie with full information. We evaluate the performance of the proposed antenna state selection technique by identifying key link quality metrics and using measured channels in a 2 × 2 MIMO OFDM system. We show that the proposed technique maximizes long term link performance with reduced channel training frequency.

Optimal Deployment of Parallel Teams in New Product Development

Optimal Deployment of Parallel Teams in New Product Development

Sensing and Probing Cardinalities for Active Cognitive Radios

In a cognitive radio network, opportunistic spectrum access (OSA) to the underutilized spectrum involves not only sensing the spectrum occupancy but also probing the channel quality in order to identify an idle and good channel for data transmission-particularly if a large number of channels is open for secondary spectrum reuse. Although such a joint mechanism, referred to as active sensing, may improve the OSA performance due to diversity, it inevitably incurs additional energy consumption. In this paper, we consider a wideband cognitive radio network with limited available frame energy and treat a fundamental energy allocation problem: how available energy should be optimally allocated for sensing, probing, and data transmission to maximize the achievable average OSA throughput. By casting this problem into the multiarmed bandit framework under probably approximately correct (PAC) learning, we put forth a proactive strategy for determining the optimal sensing cardinality (the number of channels chosen to sense) and probing cardinality (the number of channels chosen to probe) that maximize the average throughput of the secondary user with limited available frame energy. This framework determines the optimal amount of pure exploration for the active sensing OSA bandit problem in which we refine the action (median) elimination algorithm for channel probing to minimize the sample complexity in PAC learning. Numerical results show that the optimal active sensing achieves a significant throughput gain over the (even optimal) sensing alone. Therefore, this work provides an energy allocation policy to optimally balance the available energy between exploration (sensing and probing) and exploitation (data transmission), giving the optimal diversity-energy tradeoff for the average OSA throughput.

Optimal deployment of parallel teams in new product development

Speed in product development is the thin line that determines success or failure in many fast growing industries today. Strategies like set-based concurrent engineering and parallel team deployment have been used to accelerate new product introduction by successful companies like Toyota. In this paper, these two strategies are combined to investigate the impact of generating sets of alternatives using parallel teams on the revenue generated from new product development. The trade-off between cost and speed of development due to deployment of parallel teams in product development using the multi-armed bandit (MAB) framework are also explored. Quantitative models to determine the optimum number of product development teams at different stages of development have been constructed using the Gittins Index strategy. These models are tested using hypothetical data and the results show that the higher flexibility induced quality results in higher rewards, despite the higher cost of deploying multiple teams.

Opportunistic spectrum access based on a constrained multi-armed bandit formulation

Tracking and exploiting instantaneous spectrum opportunities are fundamental challenges in opportunistic spectrum access (OSA) in presence of the bursty traffic of primary users and the limited spectrum sensing capability of secondary users. In order to take advantage of the history of spectrum sensing and access decisions, a sequential decision framework is widely used to design optimal policies. However, many existing schemes, based on a partially observed Markov decision process (POMDP) framework, reveal that optimal policies are non-stationary in nature which renders them difficult to calculate and implement. Therefore, this work pursues stationary OSA policies, which are thereby efficient yet low-complexity, while still incorporating many practical factors, such as spectrum sensing errors and a priori unknown statistical spectrum knowledge. First, with an approximation on channel evolution, OSA is formulated in a multi-armed bandit (MAB) framework. As a result, the optimal policy is specified by the well-known Gittins index rule, where the channel with the largest Gittins index is always selected. Then, closed-form formulas are derived for the Gittins indices with tunable approximation, and the design of a reinforcement learning algorithm is presented for calculating the Gittins indices, depending on whether the Markovian channel parameters are available a priori or not. Finally, the superiority of the scheme is presented via extensive experiments compared to other existing schemes in terms of the quality of policies and optimality.