Contextual Multi-armed Bandit Research Articles

Contextual Multi-Armed Bandit for Cache-Aware Decoupled Multiple Association in UDNs: A Deep Learning Approach

The new trends in network densification and heterogeneity introduce new challenges for user association. Currently, user association is typically coupled, with which a user is constrained to associate with the same base station in both uplink (UL) and downlink (DL). In homogeneous networks, this is simple and effective. However, it could restrict the performance in heterogeneous ultra dense networks (UDNs), wherein BSs are densely deployed with highly variable transmit powers and topologies. Besides, the backhaul capacity could be the bottleneck in UDNs, and caching popular contents at BSs becomes an effective method for alleviating the backhauling traffic. In this paper, we propose a novel cache-aware decoupled multiple association mechanism for full-duplex UDNs, which allows a user to associate with multiple BSs in UL and DL, in a decoupled manner. Considering that users can form a self-learning system, a contextual multi-armed bandit (CMAB) problem is formulated where network states are unknown random variables. For obtaining the optimal strategy, an SINR-based linear upper confidence bound algorithm is developed. And deep learning is adopted when considering a large-scale network with expansion of the dimension in state space. The convergence of the algorithm is proven. Simulation results validate the feasibility and superiority of the proposed approach by comparisons.

Context-Aware TDD Configuration and Resource Allocation for Mobile Edge Computing

Mobile edge computing (MEC) supporting localized context awareness creates a new technological frontier for 5G and beyond. Due to very asymmetric traffic related to MEC and the time division duplexing (TDD) system, we efficiently exploit the networking and computing functionalities for TDD orthogonal frequency division multiple access (TDD-OFDMA) technology supporting multiple services. The primary technical challenge of TDD-OFDMA systems lies in dynamic configuring based on the unknown characteristics of future traffic, i.e., the information lag. Therefore, a model-free online TDD configuration scheme is proposed based on context analysis and multi-armed bandit (MAB) optimization. The characteristics of future traffic are predicted by the context-aware MEC computing, so that TDD configuration is novelly modeled as a contextual MAB problem. Solving MAB by the contextual upper-confidence-bound, TDD configuration can be dynamically adjusted according to network traffic. To simultaneously reduce the energy consumption and makespan of mobile devices (MDs), a greedy resource allocation (GRA) embedded in the TDD configuration is further developed to select MDs and allocate resources. GRA algorithm decomposes the complex multi-factor coupling non-convex problem into a series of convex sub-problems, thereby asymptotically obtaining the selection and allocation with polynomial time complexity. Simulations justify significant performance gain in mobile networking and MEC.

Privacy-Preserving and Residential Context-Aware Online Learning for IoT-Enabled Energy Saving With Big Data Support in Smart Home Environment

Energy-saving (ES) systems developed on the basis of the Internet-of-Things (IoT) by heavily relying on automated understanding of human behaviors and activities recognition is of paramount importance in smart home. However, classic approaches are incapable to understand the relations among users’ contexts and ES of appliances very well, and they cannot handle massive metering and time-varying user context datasets. Moreover, privacy concern is thoroughly aroused from both the residential and utility provider sides as to its essentiality. To tackle these problems, we propose a privacy-preserving and residential context-aware online ES (PRCOES) system in an IoT-enabled smart home environment. We model the repeated interaction of ES of appliances and the activity recognition of user context as a contextual multiarmed bandits (CMAB) problem, where the context-aware online learning algorithm can predict appropriate energy offers (EOs) that could meet the users’ satisfaction, task completion rate, and ES purposes for appliances. We utilize a tree-based structure expanding from top to bottom to recommend EOs, which supports ever-increasing big metering datasets with user context-awareness. Theoretical analysis shows that our proposal achieves sublinear regret and differential privacy for both residents and utility provider. Experiments results validate that PRCOES could enhance users’ experience and prolong users’ engagement in everyday ES while guarantee the privacy for both residents and utility provider.

Incorporating Behavioral Constraints in Online AI Systems

AI systems that learn through reward feedback about the actions they take are increasingly deployed in domains that have significant impact on our daily life. However, in many cases the online rewards should not be the only guiding criteria, as there are additional constraints and/or priorities imposed by regulations, values, preferences, or ethical principles. We detail a novel online agent that learns a set of behavioral constraints by observation and uses these learned constraints as a guide when making decisions in an online setting while still being reactive to reward feedback. To define this agent, we propose to adopt a novel extension to the classical contextual multi-armed bandit setting and we provide a new algorithm called Behavior Constrained Thompson Sampling (BCTS) that allows for online learning while obeying exogenous constraints. Our agent learns a constrained policy that implements the observed behavioral constraints demonstrated by a teacher agent, and then uses this constrained policy to guide the reward-based online exploration and exploitation. We characterize the upper bound on the expected regret of the contextual bandit algorithm that underlies our agent and provide a case study with real world data in two application domains. Our experiments show that the designed agent is able to act within the set of behavior constraints without significantly degrading its overall reward performance.

Using multi-armed bandits to learn ethical priorities for online AI systems

AI systems that learn through reward feedback about the actions they take are deployed in domains that have significant impact on our daily life. However, in many cases the online rewards should not be the only guiding criteria, as there are additional constraints and/or priorities imposed by regulations, values, preferences, or ethical principles. We detail a novel online agent that learns a set of behavioral constraints by observation and uses these learned constraints when making decisions in an online setting, while still being reactive to reward feedback. We propose a novel extension to the contextual multi-armed bandit setting and provide a new algorithm called Behavior Constrained Thompson Sampling (BCTS) that allows for online learning while obeying exogenous constraints. Our agent learns a constrained policy that implements observed behavioral constraints demonstrated by a teacher agent, and uses this constrained policy to guide its online exploration and exploitation. We characterize the upper bound on the expected regret of BCTS and provide a case study with real-world data in two application domains. Our experiments show that the designed agent is able to act within the set of behavior constraints without significantly degrading its overall reward performance.

Selecting multiple web adverts: A contextual multi-armed bandit with state uncertainty

We present a method to solve the problem of choosing a set of adverts to display to each of a sequence of web users. The objective is to maximise user clicks over time and to do so we must learn about the quality of each advert in an online manner by observing user clicks. We formulate the problem as a novel variant of a contextual combinatorial multi-armed bandit problem. The context takes the form of a probability distribution over the user's latent topic preference, and rewards are a particular nonlinear function of the selected set and the context. These features ensure that optimal sets of adverts are appropriately diverse. We give a flexible solution method which combines submodular optimisation with existing bandit index policies. User state uncertainty creates ambiguity in interpreting user feedback which prohibits exact Bayesian updating, but we give an approximate method that is shown to work well.

Training a Quantum Neural Network to Solve the Contextual Multi-Armed Bandit Problem

Artificial intelligence has permeated all aspects of our lives today. However, to make AI behave like real AI, the critical bottleneck lies in the speed of computing. Quantum computers employ the peculiar and unique properties of quantum states such as superposition, entanglement, and interference to process information in ways that classical computers cannot. As a new paradigm of computation, quantum computers are capable of performing tasks intractable for classical processors, thus providing a quantum leap in AI research and making the development of real AI a possibility. In this regard, quantum machine learning not only enhances the classical machine learning approach but more importantly it provides an avenue to explore new machine learning models that have no classical counterparts. The qubit-based quantum computers cannot naturally represent the continuous variables commonly used in machine learning, since the measurement outputs of qubit-based circuits are generally discrete. Therefore, a continuous-variable (CV) quantum architecture based on a photonic quantum computing model is selected for our study. In this work, we employ machine learning and optimization to create photonic quantum circuits that can solve the contextual multi-armed bandit problem, a problem in the domain of reinforcement learning, which demonstrates that quantum reinforcement learning algorithms can be learned by a quantum device.

An Online Context-Aware Machine Learning Algorithm for 5G mmWave Vehicular Communications

Millimeter-Wave (mmWave) bands have become the de-facto candidate for 5G vehicle-to-everything (V2X) since future vehicular systems demand Gbps links to acquire the necessary sensory information for (semi)-autonomous driving. Nevertheless, the directionality of mmWave communications and its susceptibility to blockage raise severe questions on the feasibility of mmWave vehicular communications. The dynamic nature of 5G vehicular scenarios and the complexity of directional mmWave communication calls for higher context-awareness and adaptability. To this aim, we propose an online learning algorithm addressing the problem of beam selection with environment-awareness in mmWave vehicular systems. In particular, we model this problem as a contextual multi-armed bandit problem. Next, we propose a lightweight context-aware online learning algorithm, namely fast machine learning (FML), with proven performance bound and guaranteed convergence. FML exploits coarse user location information and aggregates the received data to learn from and adapt to its environment. Furthermore, we demonstrate the feasibility of a real-world implementation of FML by proposing a standard-compliant protocol based on the existing architecture of cellular networks and the forthcoming features of 5G. We also perform an extensive evaluation using realistic traffic patterns derived from Google Maps. Our evaluation shows that FML enables mmWave base stations to achieve near-optimal performance on average within 33 mins of deployment by learning from the available context. Moreover, FML remains within ~ 5% of the optimal performance by swift adaptation to system changes (i.e., blockage, traffic).

Accurate Content Push for Content-Centric Social Networks: A Big Data Support Online Learning Approach

With the rapid growth of the social network, information overload becomes a critical issue. Service providers push a lot of redundant contents and advertisements to users every day. Thus, users’ interests and the probability of reading them have dropped considerably and the network load is wasted. To address this issue, accurate content push is needed, where the main challenges are proving precise descriptions of users and supporting the big data nature of users and contents. Content-centric networking (CCN) has emerged as a new network architecture to meet today's requirement for content access and delivery. By using the named content, CCN makes it possible to track users’ real-time interests and motivates us studying a novel content accurate push (or called content recommendation) system. In this paper, we model this issue as a novel contextual multiarmed bandit based Monte Carlo tree search problem and propose a big data support online learning algorithm to meet the demand of content push with low cost. To avoid destroying CCN's energy efficient feature, the energy consumption is considered into our module. Then, we theoretically prove that our online learning algorithm achieves sublinear regret bound and sublinear storage, which is very efficient in the big data context and do not increase the network burden. Experiments in an offline collected dataset show that our approach significantly increases the accuracy and convergence speed against other state-of-the-art bandit algorithms and can overcome the cold start problem as well.

Asymptotically Optimal Contextual Bandit Algorithm Using Hierarchical Structures.

We propose an online algorithm for sequential learning in the contextual multiarmed bandit setting. Our approach is to partition the context space and, then, optimally combine all of the possible mappings between the partition regions and the set of bandit arms in a data-driven manner. We show that in our approach, the best mapping is able to approximate the best arm selection policy to any desired degree under mild Lipschitz conditions. Therefore, we design our algorithm based on the optimal adaptive combination and asymptotically achieve the performance of the best mapping as well as the best arm selection policy. This optimality is also guaranteed to hold even in adversarial environments since we do not rely on any statistical assumptions regarding the contexts or the loss of the bandit arms. Moreover, we design an efficient implementation for our algorithm using various hierarchical partitioning structures, such as lexicographical or arbitrary position splitting and binary trees (BTs) (and several other partitioning examples). For instance, in the case of BT partitioning, the computational complexity is only log-linear in the number of regions in the finest partition. In conclusion, we provide significant performance improvements by introducing upper bounds (with respect to the best arm selection policy) that are mathematically proven to vanish in the average loss per round sense at a faster rate compared to the state of the art. Our experimental work extensively covers various scenarios ranging from bandit settings to multiclass classification with real and synthetic data. In these experiments, we show that our algorithm is highly superior to the state-of-the-art techniques while maintaining the introduced mathematical guarantees and a computationally decent scalability.

Multi-objective Contextual Multi-armed Bandit With a Dominant Objective

In this paper, we propose a new multi-objective contextual multi-armed bandit (MAB) problem with two objectives, where one of the objectives dominates the other objective. Unlike single-objective MAB problems in which the learner obtains a random scalar reward for each arm it selects, in the proposed problem, the learner obtains a random reward vector, where each component of the reward vector corresponds to one of the objectives and the distribution of the reward depends on the context that is provided to the learner at the beginning of each round. We call this problem contextual multi-armed bandit with a dominant objective (CMAB-DO). In CMAB-DO, the goal of the learner is to maximize its total reward in the non-dominant objective while ensuring that it maximizes its total reward in the dominant objective. In this case, the optimal arm given a context is the one that maximizes the expected reward in the non-dominant objective among all arms that maximize the expected reward in the dominant objective. First, we show that the optimal arm lies in the Pareto front. Then, we propose the multi-objective contextual multi-armed bandit algorithm (MOC-MAB), and define two performance measures: the 2-dimensional (2D) regret and the Pareto regret. We show that both the 2D regret and the Pareto regret of MOC-MAB are sublinear in the number of rounds. We also compare the performance of the proposed algorithm with other state-of-the-art methods in synthetic and real-world datasets. The proposed model and the algorithm have a wide range of real-world applications that involve multiple and possibly conflicting objectives ranging from wireless communication to medical diagnosis and recommender systems.

Putting bandits into context: How function learning supports decision making.

The authors introduce the contextual multi-armed bandit task as a framework to investigate learning and decision making in uncertain environments. In this novel paradigm, participants repeatedly choose between multiple options in order to maximize their rewards. The options are described by a number of contextual features which are predictive of the rewards through initially unknown functions. From their experience with choosing options and observing the consequences of their decisions, participants can learn about the functional relation between contexts and rewards and improve their decision strategy over time. In three experiments, the authors explore participants' behavior in such learning environments. They predict participants' behavior by context-blind (mean-tracking, Kalman filter) and contextual (Gaussian process and linear regression) learning approaches combined with different choice strategies. Participants are mostly able to learn about the context-reward functions and their behavior is best described by a Gaussian process learning strategy which generalizes previous experience to similar instances. In a relatively simple task with binary features, they seem to combine this learning with a probability of improvement decision strategy which focuses on alternatives that are expected to lead to an improvement upon a current favorite option. In a task with continuous features that are linearly related to the rewards, participants seem to more explicitly balance exploration and exploitation. Finally, in a difficult learning environment where the relation between features and rewards is nonlinear, some participants are again well-described by a Gaussian process learning strategy, whereas others revert to context-blind strategies. (PsycINFO Database Record

Adapting to Concept Drift in Credit Card Transaction Data Streams Using Contextual Bandits and Decision Trees

Credit card transactions predicted to be fraudulent by automated detection systems are typically handed over to human experts for verification. To limit costs, it is standard practice to select only the most suspicious transactions for investigation. We claim that a trade-off between exploration and exploitation is imperative to enable adaptation to changes in behavior (concept drift). Exploration consists of the selection and investigation of transactions with the purpose of improving predictive models, and exploitation consists of investigating transactions detected to be suspicious. Modeling the detection of fraudulent transactions as rewarding, we use an incremental Regression Tree learner to create clusters of transactions with similar expected rewards. This enables the use of a Contextual Multi-Armed Bandit (CMAB) algorithm to provide the exploration/exploitation trade-off. We introduce a novel variant of a CMAB algorithm that makes use of the structure of this tree, and use Semi-Supervised Learning to grow the tree using unlabeled data. The approach is evaluated on a real dataset and data generated by a simulator that adds concept drift by adapting the behavior of fraudsters to avoid detection. It outperforms frequently used offline models in terms of cumulative rewards, in particular in the presence of concept drift.

Customized Nonlinear Bandits for Online Response Selection in Neural Conversation Models

Dialog response selection is an important step towards natural response generation in conversational agents. Existing work on neural conversational models mainly focuses on offline supervised learning using a large set of context-response pairs. In this paper, we focus on online learning of response selection in retrieval-based dialog systems. We propose a contextual multi-armed bandit model with a nonlinear reward function that uses distributed representation of text for online response selection. A bidirectional LSTM is used to produce the distributed representations of dialog context and responses, which serve as the input to a contextual bandit. In learning the bandit, we propose a customized Thompson sampling method that is applied to a polynomial feature space in approximating the reward. Experimental results on the Ubuntu Dialogue Corpus demonstrate significant performance gains of the proposed method over conventional linear contextual bandits. Moreover, we report encouraging response selection performance of the proposed neural bandit model using the Recall@k metric for a small set of online training samples.

Functional Contour-following via Haptic Perception and Reinforcement Learning.

Many tasks involve the fine manipulation of objects despite limited visual feedback. In such scenarios, tactile and proprioceptive feedback can be leveraged for task completion. We present an approach for real-time haptic perception and decision-making for a haptics-driven, functional contour-following task: the closure of a ziplock bag. This task is challenging for robots because the bag is deformable, transparent, and visually occluded by artificial fingertip sensors that are also compliant. A deep neural net classifier was trained to estimate the state of a zipper within a robot's pinch grasp. A Contextual Multi-Armed Bandit (C-MAB) reinforcement learning algorithm was implemented to maximize cumulative rewards by balancing exploration versus exploitation of the state-action space. The C-MAB learner outperformed a benchmark Q-learner by more efficiently exploring the state-action space while learning a hard-to-code task. The learned C-MAB policy was tested with novel ziplock bag scenarios and contours (wire, rope). Importantly, this work contributes to the development of reinforcement learning approaches that account for limited resources such as hardware life and researcher time. As robots are used to perform complex, physically interactive tasks in unstructured or unmodeled environments, it becomes important to develop methods that enable efficient and effective learning with physical testbeds.

Actionable intelligence and online learning for semantic computing

As the world becomes more connected and instrumented, high dimensional, heterogeneous and time-varying data streams are collected and need to be analyzed on the fly to extract the actionable intelligence from the data streams and make timely decisions based on this knowledge. This requires that appropriate classifiers are invoked to process the incoming streams and find the relevant knowledge. Thus, a key challenge becomes choosing online, at run-time, which classifier should be deployed to make the best possible predictions on the incoming streams. In this paper, we survey a class of methods capable to perform online learning in stream-based semantic computing tasks: multi-armed bandits (MABs). Adopting MABs for stream mining poses, numerous new challenges requires many new innovations. Most importantly, the MABs will need to explicitly consider and track online the time-varying characteristics of the data streams and to learn fast what is the relevant information out of the vast, heterogeneous and possibly highly dimensional data streams. In this paper, we discuss contextual MAB methods, which use similarities in context (meta-data) information to make decisions, and discuss their advantages when applied to stream mining for semantic computing. These methods can be adapted to discover in real-time the relevant contexts guiding the stream mining decisions, and tract the best classifier in presence of concept drift. Moreover, we also discuss how stream mining of multiple data sources can be performed by deploying cooperative MAB solutions and ensemble learning. We conclude the paper by discussing the numerous other advantages of MABs that will benefit semantic computing applications.

Decision Support in Intelligent Maintenance-planning Systems Based on Contextual Multi-armed Bandit Algorithm

In this paper we focus on two essential problems of maintenance decision support systems, namely, 1) detection of potential dangerous situation, and 2) classification of this situation in order to recommend an appropriate repair action. The former task is usually solved with the known statistical process control techniques. The latter problem can be reduced to the contextual multi- armed bandit problem. We propose a novel algorithm with Bayesian classification of abnormal situation and the softmax rule to explore the decision space. The dangerous situations are detected with the Shewhart control charts for the distances between the current and the normal situations. It is experimentally shown, that our algorithm is more accurate than the known contextual multi-armed methods with stochastic search strategies.

LinUCB applied to Monte Carlo tree search

UCT is a standard method of Monte Carlo tree search (MCTS) algorithms, which have been applied to various domains and have achieved remarkable success. This study proposes a family of LinUCT algorithms that incorporate LinUCB into MCTS algorithms. LinUCB is a recently developed method that generalizes past episodes by ridge regression with feature vectors and rewards. LinUCB outperforms UCB1 in contextual multi-armed bandit problems. We introduce a straightforward application of LinUCB, LinUCTPLAIN by substituting UCB1 with LinUCB in UCT. We show that it does not work well owing to the minimax structure of game trees. To better handle such tree structures, we present LinUCTRAVE and LinUCTFP by further incorporating two existing techniques, rapid action value estimation (RAVE) and feature propagation, which recursively propagates the feature vector of a node to that of its parent. Experiments were conducted with a synthetic model, which is an extension of the standard incremental random tree model in which each node has a feature vector that represents the characteristics of the corresponding position, and Finnsson's shock step game which is used to empirically analyze the performance of UCT with respect to the distribution of suboptimal moves. The experiments results indicate that LinUCTRAVE and LinUCTFP outperform UCT, especially when the branching factor is relatively large.

The adaptive approach to abnormal situations recognition using images from condition monitoring systems

Decision support in equipment condition monitoring systems with image processing is analyzed. Long-run accumulation of information about earlier made decisions is used to realize the adaptiveness of the proposed approach. It is shown that unlike conventional classification problems, the recognition of abnormalities uses training samples supplemented with reward estimates of earlier decisions and can be tackled using reinforcement learning algorithms. We consider the basic stages of contextual multi-armed bandit algorithms during which the probabilistic distributions of each state are evaluated to evaluate the current knowledge of the states, and the decision space is explored to increase the decision-making efficiency. We propose a new decision-making method, which uses the probabilistic neural network to classify abnormal situation and the softmax rule to explore the decision space. A modelling experiment in image processing was carried out to show that our approach allows a higher accuracy of abnormality detection than other known methods, especially for small-size initial training samples.

Online Learning of Robot Soccer Free Kick Plans Using a Bandit Approach

This paper presents an online learning approach for teams of autonomous soccer robots to select free kick plans. In robot soccer, free kicks present an opportunity to execute plans with relatively controllable initial conditions. However, the effectiveness of each plan is highly dependent on the adversary, and there are few free kicks during each game, making it necessary to learn online from sparse observations. To achieve learning, we first greatly reduce the planning space by framing the problem as a contextual multi-armed bandit problem, in which the actions are a set of pre-computed plans, and the state is the position of the free kick on the field. During execution, we model the reward function for different free kicks using Gaussian Processes, and perform online learning using the Upper Confidence Bound algorithm. Results from a physics-based simulation reveal that the robots are capable of adapting to various different realistic opponents to maximize their expected reward during free kicks.