Contextual Multi-armed Bandit Research Articles

Costly exploration produces stereotypes with dimensions of warmth and competence.

Traditional explanations for stereotypes assume that they result from deficits in humans (ingroup-favoring motives, cognitive biases) or their environments (majority advantages, real group differences). An alternative explanation recently proposed that stereotypes can emerge when exploration is costly. Even optimal decision makers in an ideal environment can inadvertently form incorrect impressions from arbitrary encounters. However, all these existing theories essentially describe shortcuts that fail to explain the multidimensionality of stereotypes. Stereotypes of social groups have a canonical multidimensional structure, organized along dimensions of warmth and competence. We show that these dimensions and the associated stereotypes can result from feature-based exploration: When individuals make self-interested decisions based on past experiences in an environment where exploring new options carries an implicit cost and when these options share similar attributes, they are more likely to separate groups along multiple dimensions. We formalize this theory via the contextual multiarmed bandit problem, use the resulting model to generate testable predictions, and evaluate those predictions against human behavior. We evaluate this process in incentivized decisions involving as many as 20 real jobs and successfully recover the classic dimensions of warmth and competence. Further experiments show that intervening on the cost of exploration effectively mitigates bias, further demonstrating that exploration cost per se is the operating variable. Future diversity interventions may consider how to reduce exploration cost, in ways that parallel our manipulations. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Observation-Augmented Contextual Multi-Armed Bandits for Robotic Search and Exploration

Observation-Augmented Contextual Multi-Armed Bandits for Robotic Search and Exploration

Improving Work Skills in People with Disabilities in a Long-Term, In-Field Study on Robotic Tutors

Inclusion of people with disabilities in the open labor market using robotic assistance is a promising new and important field of research, albeit challenging. People with disabilities are severely underrepresented in the open labor market, although inclusion adds significant value on both financial and social levels. Here, collaborative industrial robots offer great potential for support. This work conducted a month-long, in-field user study in a workshop for people with disabilities to improve learning progress through collaboration with an innovative intelligent robotic tutoring system. Seven workers with a wide variety of disabilities solved assembly tasks while being supervised by the system. In case of errors or hesitations, different modes of assistance were automatically offered. Modes of assistance included robotic pointing gestures, speech prompts, and calling a supervisor. Which assistance to offer the different participants during the study was personalized by a shared policy using reinforcement learning. Here, new, non-stationary Contextual Multi-Armed Bandit algorithms were developed during the prior simulation-based study planning to include the workers contextual information. Pioneering results were obtained in three main areas. The participants significantly improved their skills in terms of time required per task. The algorithm learned within only one session per participant which modes of assistance were preferred. Finally, a comparison between simulation and re-simulation, including the study results, revealed the underlying basic assumptions to be correct but individual variation led to strong performance differences in the real-world setting. Looking ahead, the innovative system developed could pave the way for many people with disabilities to enter the open labor market.

A Contextual Multi-Armed Bandit approach for NDN forwarding

Named Data Networking (NDN) is a promising Internet architecture that aims to supersede the current IP-based architecture and shift the host-centric model to a data-centric one. Within NDN, forwarding Interest packets remains a significant challenge and has attracted considerable recent research attention. The momentum behind machine learning techniques, especially reinforcement learning, is steadily growing, offering the potential to deliver intelligent, adaptable, and reliable NDN forwarding algorithms. In this context, this paper proposes efficient NDN forwarding strategies based on Contextual Multi-Armed Bandit (CMAB). Initially, we employ CMAB to address the challenge of forwarding Interest packets and introduce a new CMAB model tailored for NDN, dubbed CMAB4NDN. Subsequently, we construct the CMAB context using information derived from the content name and the network congestion state, which are then fed into the CMAB4NDN learning algorithm to decide on the best forwarding action. Further, we develop three CMAB strategies, namely Lin-ɛ-Greedy, Linear Upper Confidence Bound, and Contextual Thompson Sampling, and deploy them within our proposal. CMAB4NDN was implemented in ndnSIM, thoroughly evaluated, and compared with multiple state-of-the-art NDN forwarding algorithms across various scenarios. The obtained results confirm the relevance and superiority of our approach in terms of delay, throughput, and packet loss.

Enhancing conversational recommendation systems through the integration of KNN with ConLinUCB contextual bandits

In recommender system research, contextual multi-armed bandits have shown promise in delivering tailored recommendations by utilizing contextual data. However, their effectiveness is often curtailed by the cold start problem, arising from the lack of initial user data. This necessitates extensive exploration to ascertain user preferences, consequently impeding the speed of learning. The advent of conversational recommendation systems offers a solution. Through these systems, the conversational contextual bandit algorithm swiftly learns user preferences for specific key-terms via interactive dialogues, thereby enhancing the learning pace. Despite these advancements, there are limitations in current methodologies. A primary issue is the suboptimal integration of data from key-term-centric dialogues and arm-level recommendations, which could otherwise expedite the learning process. Another crucial aspect is the strategic suggestion of exploratory key phrases. These phrases are essential in quickly uncovering users potential interests in various domains, thus accelerating the convergence of accurate user preference models. Addressing these challenges, the ConLinUCB framework emerges as a groundbreaking solution. It ingeniously combines feedback from both arm-level and key-term-level interactions, significantly optimizing the learning trajectory. Building upon this, the framework integrates a K-nearest neighbour (KNN) approach to refine key-term selection and arm recommendations. This integration hinges on the similarity of user preferences, further hastening the convergence of the parameter vectors.

Bandit approach for unmanned aerial vehicle-centric low earth orbit satellite selection

In this paper, the problem of optimal low earth orbit satellite (LEO-Sat) selection by unmanned aerial vehicle (UAV) in space air ground integrated networks (SAGINs) will be investigated. The primary objective is to maximize the UAV's achievable data rate while ensuring a prolonged remaining visible time for the selected LEO-Sat. This problem poses significant challenges due to the highly dynamic nature caused by the substantial relative velocity between UAV and LEO-Sats, making LEO-Sats' features, e.g., elevation angle, remaining visible time, and the available bandwidth strongly time-dependent. Moreover, some LEO-Sats will appear/disappear from UAV visibility within short time intervals. An online learning approach will be proposed to efficiently address this highly dynamic problem, where it will be modeled as contextual multi-armed bandit (MAB) with dynamic arms. Through this modeling, the time varying LEO-Sats' features as well as their dynamic appearance/disappearance will be taken into account while selecting the best LEO-Sat. Through extensive numerical analysis, the performance of the proposed approach nearly matches that of the highly complex optimal performance based on exhaustive searching with negligible time complexity, and it outperforms conventional schemes that individually maximize one of the LEO-Sats' features.

A survey on contextual multi-armed bandits

As a powerful reinforcement learning framework, Contextual Multi-Armed Bandits have extensive applications in various domains. The models of Contextual Multi-Armed Bandits enable decision-makers to make intelligent choices in situations with uncertainty, and they find utility in fields such as online advertising, medical treatment optimization, resource allocation, and more. This paper reviews the evolution of algorithms for Contextual Multi-Armed Bandits, including traditional Bayesian approaches and the latest deep learning techniques. Successful case studies are summarized in different application domains, such as online ad click-through rate optimization and medical decision support. Furthermore, the author discusses future research directions, including more sophisticated context modeling, interpretability, fairness issues, and ethical considerations in the context of automated decision-making.

Survey of dynamic pricing based on Multi-Armed Bandit algorithms

Dynamic pricing seeks to determine the most optimal selling price for a product or service, taking into account factors like limited supply and uncertain demand. This study aims to provide a comprehensive exploration of dynamic pricing using the multi-armed bandit problem framework in various contexts. The investigation highlights the prevalence of Thompson sampling in dynamic pricing scenarios with a Bayesian backdrop, where the seller possesses prior knowledge of demand functions. On the other hand, in non-Bayesian situations, the Upper Confidence Bound (UCB) algorithm family gains traction due to their favorable regret bounds. As markets often exhibit temporal fluctuations, the domain of non-stationary multi-armed bandits within dynamic pricing emerges as crucial. Future research directions include enhancing traditional multi-armed bandit algorithms to suit online learning settings, especially those involving dynamic reward distributions. Additionally, merging prior insights into demand functions with contextual multi-armed bandit approaches holds promise for advancing dynamic pricing strategies. In conclusion, this study sheds light on dynamic pricing through the lens of multi-armed bandit problems, offering insights and pathways for further exploration.

Investigation of progress and application related to Multi-Armed Bandit algorithms

This paper discusses four Multi-armed Bandit algorithms: Explore-then-Commit (ETC), Epsilon-Greedy, Upper Confidence Bound (UCB), and Thompson Sampling algorithm. ETC algorithm aims to spend the majority of rounds on the best arm, but it can lead to a suboptimal outcome if the environment changes rapidly. The Epsilon-Greedy algorithm is designed to explore and exploit simultaneously, while it often tries sub-optimal arm even after the algorithm finds the best arm. Thus, the Epsilon-Greedy algorithm performs well when the environment continuously changes. UCB algorithm is one of the most used Multi-armed Bandit algorithms because it can rapidly narrow the potential optimal decisions in a wide range of scenarios; however, the algorithm can be influenced by some specific pattern of reward distribution or noise presenting in the environment. Thompson Sampling algorithm is also one of the most common algorithms in the Multi-armed Bandit algorithm due to its simplicity, effectiveness, and adaptability to various reward distributions. The Thompson Sampling algorithm performs well in multiple scenarios because it explores and exploits simultaneously, but its variance is greater than the three algorithms mentioned above. Today, Multi-armed bandit algorithms are widely used in advertisement, health care, and website and app optimization. Finally, the Multi-armed Bandit algorithms are rapidly replacing the traditional algorithms; in the future, the advanced Multi-armed Bandit algorithm, contextual Multi-armed Bandit algorithm, will gradually replace the old one.

Transfer learning for contextual multi-armed bandits

Transfer learning for contextual multi-armed bandits

Bandit algorithms: A comprehensive review and their dynamic selection from a portfolio for multicriteria top-k recommendation

This paper discusses the use of portfolio approaches based on bandit algorithms to optimize multicriteria decision-making in recommender systems (accuracy and diversity). While previous research has primarily focused on single-item recommendations, this study extends the research to consider the recommendation of several items per iteration. Two methods, Multiple-play Gorthaur and Budgeted-Gorthaur, are proposed to solve the algorithm selection problem and their performances on real-world datasets are compared. Both methods provide a generalization of the Gorthaur method, which enables it to operate with any Multi-Armed Bandit (MAB) and Contextual Multi-Armed Bandit (CMAB) algorithm as meta-algorithm in a multi-item recommendation scenario. For Multiple-play Gorthaur, an empirical evaluation shows that the use of Thompson Sampling for algorithm selection (Gorthaur-TS) yields better results than the original EXP3 method (Gorthaur-EXP3) and the exclusive use of the optimal algorithm in the portfolio in contextual recommendation problems. Additionally, the paper includes a theoretical regret analysis based on the TS sketch proof applied for this variant of the method. Concerning Budgeted-Gorthaur, experiments show that it allows more flexibility to achieve a suitable trade-off between criteria and a broader coverage of the Pareto set of solutions, overcoming a natural limit of “a-priori” methods. Finally, this paper provides a detailed review, including pseudocodes and theoretical bounds, for all the fundamental MAB and CMAB algorithms used in this study.

Contextual Multi-Armed Bandit With Costly Feature Observation in Non-Stationary Environments

Contextual Multi-Armed Bandit With Costly Feature Observation in Non-Stationary Environments

Artificial intelligence for smart bidding

Growth in industry digitisation in recent years has resulted in consumers, manufacturers and service providers succumbing to the allure of the online platform, leading to the erosion of the long-standing supremacy of traditional businesses. With increasing competition among manufacturers both at local and global scope, it becomes crucial for them to make intelligent digital marketing decisions which could help get ahead of their competitors. Artificial intelligence (AI) has proven to be an effective medium for supporting decision-making in digital marketing. This paper discusses the application of contextual multi-armed bandits to optimise the bidding strategy used by digital advertisers in main search platforms. The highly scalable algorithm, apart from suggesting a winning strategy in an advertising auction, also enables clients to improve return on investment (ROI) using digital advertising.

Cross-Frame Resource Allocation With Context-Aware QoE Estimation for 360° Video Streaming in Wireless Virtual Reality

Wireless virtual reality (VR), aiming to provide an untethered immersive experience through 360° videos, could be facilitated by viewport-guided streaming with the help of viewport prediction. Although many recent viewport predictors can output a series of predictions over upcoming frames, most existing work on video streaming does not fully utilize the capability of these predictors. In this paper, we investigate the problem of 360° video streaming by incorporating the complete series of viewport predictions for maximizing the quality of experience (QoE) through cross-frame resource allocation. To address the problem of viewport prediction errors that could result in erroneous estimation of QoE contribution of tiles in upcoming frames, we develop a novel approach based on contextual multi-armed bandit (CMAB) to “learn” <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">online</i> the viewing behavior of the user and the capability of the predictor such that resource can be preferentially allocated to tiles with significant QoE contribution. Further, to address the problem of transmission failures during wireless streaming, we formulate a constrained Markov decision process (CMDP) and apply model predictive control (MPC) to account for resource competition among reactive and proactive transmissions as well as retransmissions of tiles. The performance of the proposed streaming system is evaluated using a real-world VR dataset, state-of-the-art viewport predictors, and realistic mmWave channel models. An improvement of 10.2% in QoE and a reduction of 18.7% in resource waste are achieved across various videos, users, and predictors. Simulation results substantiate that the context-aware QoE learned by the proposed CMAB effectively addresses prediction errors for tiles with different temporal and spatial contexts, and the proposed CMDP can achieve the desired performance even under tight bandwidth constraint and severe channel condition.

Signal detection models as contextual bandits.

Signal detection theory (SDT) has been widely applied to identify the optimal discriminative decisions of receivers under uncertainty. However, the approach assumes that decision-makers immediately adopt the appropriate acceptance threshold, even though the optimal response must often be learned. Here we recast the classical normal-normal (and power-law) signal detection model as a contextual multi-armed bandit (CMAB). Thus, rather than starting with complete information, decision-makers must infer how the magnitude of a continuous cue is related to the probability that a signaller is desirable, while simultaneously seeking to exploit the information they acquire. We explain how various CMAB heuristics resolve the trade-off between better estimating the underlying relationship and exploiting it. Next, we determined how naive human volunteers resolve signal detection problems with a continuous cue. As anticipated, a model of choice (accept/reject) that assumed volunteers immediately adopted the SDT-predicted acceptance threshold did not predict volunteer behaviour well. The Softmax rule for solving CMABs, with choices based on a logistic function of the expected payoffs, best explained the decisions of our volunteers but a simple midpoint algorithm also predicted decisions well under some conditions. CMABs offer principled parametric solutions to solving many classical SDT problems when decision-makers start with incomplete information.

Dealing with expert bias in collective decision-making

Quite some real-world problems can be formulated as decision-making problems wherein one must repeatedly make an appropriate choice from a set of alternatives. Multiple expert judgments, whether human or artificial, can help in taking correct decisions, especially when exploration of alternative solutions is costly. As expert opinions might deviate, the problem of finding the right alternative can be approached as a collective decision making problem (CDM) via aggregation of independent judgments. Current state-of-the-art approaches focus on efficiently finding the optimal expert, and thus perform poorly if all experts are not qualified or if they display consistent biases, thereby potentially derailing the decision-making process. In this paper, we propose a new algorithmic approach based on contextual multi-armed bandit problems (CMAB) to identify and counteract such biased expertise. We explore homogeneous, heterogeneous and polarized expert groups and show that this approach is able to effectively exploit the collective expertise, outperforming state-of-the-art methods, especially when the quality of the provided expertise degrades. Our novel CMAB-inspired approach achieves a higher final performance and does so while converging more rapidly than previous adaptive algorithms.

NeuE: Automated Neural Network Ensembles for Edge Intelligence

Artificial Intelligence (AI) applications have been established in the mobile industry and are decisively determining the progress in entrepreneurial value creation. This paper explores the potential of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Edge Computing</i> to enhance the performance of AI applications. In particular, a DNN ensemble formation (DEF) problem is studied which judiciously recruits members for DNN ensembles considering the device heterogeneity, computing resource limitation, and service deadline of edge computing systems, in an attempt to optimize the performance of edge AI services. We design a novel algorithm called Neural Ensemble (NeuE) to solve the DEF problem. NeuE involves an online learning process that learns the in-practice performance of DNN ensembles and adaptively forms DNN ensembles according to the features of admitted tasks. It leverages the framework of contextual multi-armed bandit and follows the constraints of computing resource limitation and service deadline. We also show theoretically that NeuE provides asymptotic optimality. However, NeuE suffers from poor scalability due to exponentially-growing ensemble decision space. We then propose a variant of NeuE, called NeuE-S, to expedite NeuE. NeuE-S identifies representative ensemble decisions using similarities of ensemble decisions and carries out learning with a reduced decision space. We show via theoretical analysis that NeuE-S drastically reduces the computation complexity with negligible performance loss. We implement our method on an edge computing testbed. The results show that our method dramatically improves the performance of edge AI services.

Multi-armed bandit problem with online clustering as side information

We consider the sequential resource allocation problem under the multi-armed bandit model in the non-stationary stochastic environment. Motivated by many real applications, where information can naturally be grouped, we consider a variation of the contextual multi-armed bandit with online clustering representing side information. We assume a stochastic environment in which the reward of each action, conditioned on a cluster, follows a Bernoulli distribution with unknown parameters. Additionally, we assume that the nature of the problem changes over time and the clusters drift incrementally, making the reward process non-stationary. In this setting, we propose a new algorithm based on a two-stage approach. The first stage is a sequential modification of the traditional k-means clustering algorithm, in which the algorithm deals with the continuous data stream and acts on a subset of data rather than a single batch. In the second stage, we incorporate the current information about clusters into the Thompson Sampling policy with discounting mechanism to track changes in the underlying reward and account for a potential cluster misclassification.

Collaboration-Aware Relay Selection for AUV in Internet of Underwater Network: Evolving Contextual Bandit Learning Approach

In Internet of Underwater Things, data collection is assisted by autonomous underwater vehicle (AUV) to enhance the reliable transmission. AUV acts as a mobile collector and transmits the collected data to the station via relay nodes. However, the highly mobile nature of AUV needs an adaptive and efficient relay selection scheme for achieving good capacity performance. In this article, we propose a new contextual multiarmed bandit with evolving relay set (CMAB-ERS) learning framework, which successfully addresses crucial issues, including dynamic environment conditions and evolving relay set. To deal with the evolving relay set, CMAB-ERS incorporates collaborative effects into inference as well as learning processes, the new relays will acquire prior knowledge by having experienced nodes sharing observations, reducing the learning time significantly. To overcome the uncertainty of environmental information, we exploit the contextual environment factors to assist relay reward estimation and execute time-sensitive parameter update after every transmit–receive cycle, aiming for minimizing potential loss due to the time-varying channel. Correspondingly, the collaboration-aware online contextual bandit learning (COCBL) algorithm is designed that enables AUV to switch optimal relay adaptively and promises high-capacity transmission. Further, we rigorously prove the convergence of the COCBL algorithm by considering the evolving relay set and give its upper bound on the cumulative regret. Finally, extensive simulation results elucidate the effectiveness of the proposed COCBL.

Byzantine-Fault-Tolerant Consensus via Reinforcement Learning for Permissioned Blockchain-Empowered V2X Network

Blockchain has been forming the central piece of various types of vehicle-to-everything (V2X) network for trusted data exchange. Recently, permissioned blockchains garner particular attention thanks to their improved scalability and diverse needs from different organizations. One representative example of permissioned blockchain is Hyperledger Fabric (“Fabric”). Due to its unique execute-order procedure, there is a critical need for a client to select an optimal number of peers. The interesting problem that this paper targets to address is the tradeoff in the number of peers: a too large number will degrade scalability while a too small number will make the network vulnerable to faulty nodes. This optimization issue gets especially challenging in V2X networks due to mobility of nodes: a transaction must be executed, and the associated block must be committed before the vehicle leaves a network. To this end, this paper proposes a mechanism for selecting an optimal set of peers based on reinforcement learning (RL) to keep a Fabric-empowered V2X network impervious to dynamicity due to mobility. We model the RL as a contextual multi-armed bandit (MAB) problem. The results demonstrate the outperformance of the proposed scheme.