Combinatorial Multi-armed Bandit Research Articles

A survey of the application and technical improvement of the multi-armed bandit

In recent years, the multi-armed bandit (MAB) model has been widely used and has shown excellent performance. This article provides an overview of the applications and technical improvements of the multi-armed bandit machine problem. First, an overview of the multi-armed bandit problem is presented, including the explanation of a general modeling approach and several existing common algorithms, such as -greedy, ETC, UCB, and Thompson sampling. Then, the real-life applications of the multi-armed bandit model are explored, covering the fields of recommender systems, healthcare, and finance. Then, some improved algorithms and models are summarized by addressing the problems encountered in different application domains, including the multi-armed bandit considering multiple objectives, the mortal multi-armed bandits, the multi-armed bandit considering contextual side information, combinatorial multi-armed bandits. Finally, the characteristics, trends of changes among different algorithms, and applicable scenarios are summarized and discussed.

Auction-based client selection for online Federated Learning

Federated Learning (FL) has become a popular decentralized learning paradigm to train a machine learning model using distributed mobile devices without compromising user privacy. Despite its advantages, there are several technical challenges to achieving efficient FL. First, clients may have different amount of data with various quality, thus leading to different quality of model updates. In an online setting, it is challenging to select clients having high-quality models in advance or to adjust the selection on the fly. Second, it is an impractical assumption that the clients can selflessly engage in model training without incentives. Meanwhile, the training cost is often considered as private information which is not easily accessible. Under incomplete information, how we design an effective incentive mechanism to achieve the expected economic properties? Third, due to the lack of prior knowledge, managing long-run expenditures in real time is challenging. To this end, we propose a combinatorial multi-armed bandit- and auction-based client selection algorithm named CACS to achieve highly efficient model training. Specifically, CACS divides the client selection process into exploration and exploitation. In the former, CACS employs the upper confidence bound (UCB) quality to estimate the learning quality of clients. In the latter, CACS determines winners based on the UCB quality-bid-ratio and adopts the key payment as the payment. Theoretical analysis shows that CACS obtains the sub-linear regret and economic properties, and can be computationally efficient and ensure the convergence of global model. Extensive simulation experiments also confirm the practical advantages of CACS over state-of-the-art client selection approaches for FL.

CA-Live360: Crowd-assisted transcoding and delivery for live 360-degree video streaming

In the burgeoning field of digital media, the popularity of live 360-degree video has witnessed an upward trajectory, underpinned by its immersive experience. In live 360-degree video streaming services, transcoding enormous video tiles and providing low-latency services for global viewers require significantly more computing and communication resources than traditional live services. Inspired by the huge crowdsourcing computational resources available at crowd, this paper designs CA-Live360, a Crowd-Assisted Live 360-degree video streaming solution, in which transcoding and delivery are supported by crowdsourcing devices. Due to the dynamic availability of resources, it is challenging to achieve low-latency live services, while guaranteeing fair transcoding. Therefore, firstly an innovative fairness-guaranteed transcoding task assignment scheme is proposed. This scheme employs a novel Fair Bandit algorithm based on the combinatorial multi-armed bandit approach to achieve low-latency transcoding while considering time-varying available resources and fairness constraints. Secondly, to reduce delivery latency, we design a transcoding-aware delivery scheme, in which a provider–requester matching algorithm is introduced to realize effective requester scheduling. Real-world trace-driven experiments demonstrate the improved performance of the proposed CA-Live360 solution in terms of system delay and fairness.

Cache content placement in the presence of fictitious requests in mmWave 5G IAB networks

Caching popular content at the edge of wireless networks leads to backhaul congestion mitigation. To come up with an effective caching policy, content popularity distribution should be taken into account, which is not accurately known in most practical scenarios. Moreover, the mobile users’ (MU) request pattern may not always follow a well-defined distribution since some malicious MUs may deliberately issue their requests incompatible with the content popularity statistics. In this paper, we consider the problem of cache content placement in a 5G mmWave small cell network that relies on integrated access and backhaul (IAB) technology for pushing contents to MUs. We assume that the IAB node is equipped with a cache and has no prior knowledge about the content popularity profiles; instead, it only relies on the observation of the instantaneous demands to shape its caching policy. Also, malicious MUs may exist whose goals are to increase cache miss by issuing fictitious requests. The IAB node decides on which contents to cache and for how long, given that frequently replacing contents incurs administrative costs. We model the content placement problem as an ”adversarial combinatorial multi-armed bandit process with switching costs (ACMAB-SC)” and present an online learning algorithm for shaping the caching policy. We conduct extensive simulation experiments to evaluate the convergence property and assess the performance of our algorithm in terms of backhaul congestion, delay, and cache hit ratio. We also compare against two baseline online learning schemes, including a CMAB-based approach and a generic caching policy based on the ”Follow the Perturbed Leader (FPL)” algorithm.

Client selection for federated learning using combinatorial multi-armed bandit under long-term energy constraint

In a federated learning system, it is often the case that the more clients it involves, the less increment of the outcome it achieves. It is thus essential to design a client selection strategy to choose an appropriate subset of the clients to participate in federated learning. However, client selection is not easy due to the heterogeneity of clients and the long-term energy budget of each client. Moreover, long-term energy budgets intertwined with the short-term client selection often make the problem NP-hard. In this paper, we propose an online strategy Energy-Aware Client Selection for Federated Learning (EACS-FL) to address this problem. The problem is formulated with a joint energy and delay optimization objective, and the Combinatorial Multi-Armed Bandit (CMAB) is introduced to solve the problem in an online manner. We take advantage of Lyapunov optimization to manage energy consumption of clients, which enables us to deal with independent energy budgets through minimizing virtual energy deficit queues. Theoretical analysis shows that EACS-FL achieves sublinear regret and keeps all queues stable. Experiment results exhibit that the proposed approach outperforms the existing works and achieves close-to-optimal delay and energy consumption performance.

A Quality-Aware and Obfuscation-Based Data Collection Scheme for Cyber-Physical Metaverse Systems

In pursuit of an immersive virtual experience within the Cyber-Physical Metaverse Systems (CPMS), the construction of Avatars often requires a significant amount of real-world data. Mobile Crowd Sensing (MCS) has emerged as an efficient method for collecting data for CPMS. While progress has been made in protecting the privacy of workers, little attention has been given to safeguarding task privacy, potentially exposing the intentions of applications and posing risks to the development of the Metaverse. Additionally, existing privacy protection schemes hinder the exchange of information among entities, inadvertently compromising the quality of the collected data. To this end, we propose a Quality-aware and Obfuscation-based Task Privacy-Preserving (QOTPP) scheme, which protects task privacy and enhances data quality without third-party involvement. The QOTPP scheme initially employs the insight of “showing the fake, and hiding the real” by employing differential privacy techniques to create fake tasks and conceal genuine ones. Additionally, we introduce a two-tier truth discovery mechanism using Deep Matrix Factorization (DMF) to efficiently identify high-quality workers. Furthermore, we propose a Combinatorial Multi-Armed Bandit (CMAB)-based worker incentive and selection mechanism to improve the quality of data collection. Theoretical analysis confirms that our QOTPP scheme satisfies essential properties such as truthfulness, individual rationality, and ϵ-differential privacy. Extensive simulation experiments validate the state-of-the-art performance achieved by QOTPP.

Combinatorial Stochastic-Greedy Bandit

We propose a novel combinatorial stochastic-greedy bandit (SGB) algorithm for combinatorial multi-armed bandit problems when no extra information other than the joint reward of the selected set of n arms at each time step t in [T] is observed. SGB adopts an optimized stochastic-explore-then-commit approach and is specifically designed for scenarios with a large set of base arms. Unlike existing methods that explore the entire set of unselected base arms during each selection step, our SGB algorithm samples only an optimized proportion of unselected arms and selects actions from this subset. We prove that our algorithm achieves a (1-1/e)-regret bound of O(n^(1/3) k^(2/3) T^(2/3) log(T)^(2/3)) for monotone stochastic submodular rewards, which outperforms the state-of-the-art in terms of the cardinality constraint k. Furthermore, we empirically evaluate the performance of our algorithm in the context of online constrained social influence maximization. Our results demonstrate that our proposed approach consistently outperforms the other algorithms, increasing the performance gap as k grows.

Immersive Multimedia Service Caching in Edge Cloud with Renewable Energy

Immersive service caching, based on the intelligent edge cloud, can meet delay-sensitive service requirements. Although numerous service caching solutions for edge clouds have been designed, they have not been well explored. Moreover, to the best of our knowledge, there is no work to consider the immersive service caching scheme under the supply of renewable energy. In this article, we investigate the service caching problem under the renewable energy supply to minimize service latency while making full use of renewable energy. Specifically, we formulate the service caching and renewable energy harvesting problem, which considers the dynamic renewable energy, unknown service requests, and limited capacity of the edge cloud. To solve this problem, we propose an effective algorithm, called OSCRE. Our algorithm first uses Lyapunov optimization to convert the time-average problem into time-independence optimization and thus realizes optimal renewable energy harvesting. Then, it realizes the service caching scheme using data-driven combinatorial multi-armed bandit learning. The simulation results show that the OSCRE scheme can save service latency while making sufficient use of renewable energy.

Multi-hop relay selection for underwater acoustic sensor networks: A dynamic combinatorial multi-armed bandit learning approach

An efficient multi-hop relay selection method is the key of multi-hop relay technology to improve wireless communication reliability. Accordingly, this paper devotes a multi-hop relay selection problem for unknown time-varying underwater acoustic sensor networks, and a dynamic combinatorial multi-armed bandit (DCMAB) learning structure is proposed to achieve the multi-hop relay strategy with minimum propagation delay without any prior channel information. Compared with the strategy learning space of the single relay selection problem for static networks, the multi-hop relay learning space shows high-dimensional and dynamic characteristics. To cope with the high-dimensional characteristic of multi-hop relay strategy spaces, DCMAB develops a combinatorial bandit learning manner. It enables the player to learn the high-dimensional multi-hop relay strategy space by exploring the low-dimensional link sub-strategy space, thereby reducing the learning complexity. To cope with the dynamic characteristic of multi-hop relay strategy spaces, DCMAB makes newly-formed links able to employ the historical learning information of experienced links to reason their prior knowledge. Meanwhile, by adopting a probabilistic compensation manner, DCMAB intensifies the exploration for newly-formed links. It successfully overcomes learning inefficiency caused by the lack of learning information on newly-formed links. Besides, an energy-aware-based filtering mechanism is proposed to filter out potential long-delay relay links. It enables the player to focus on exploring and reasoning high-quality links, thereby enhancing the quick search ability of superior multi-hop relay strategies. Finally, the superiority of the proposed algorithm is demonstrated by extensive simulation results.

Investigation of selection and application of Multi-Armed Bandit algorithms in recommendation system

The Multi-Armed Bandit (MAB) algorithm holds significant prominence as a recommendation system technique, effectively presenting user-centric content preferences based on the analysis of collected data. However, the application of the basic MAB algorithm in real-world recommendation systems is not without challenges, including issues related to data volume and data processing accuracy. Therefore, the optimization algorithm based on the MAB algorithm is more widely used in the recommendation system. This paper briefly introduces the multi-armed bandit algorithm, that is, the use of MAB in the recommendation system and the problems of the basic MAB algorithm. Aiming at the problems of the basic MAB algorithm, it introduces the MAB-based optimization algorithm used in the recommendation system. At the same time, this paper also analyzes and summarizes such algorithms. This paper introduces two different MAB-based optimization algorithms, namely The Details of Dynamic clustering based contextual combinatorial multi-armed bandit (DC3MAB) and Binary Upper Confidence Bound (BiUCB). In addition, this paper also introduces the application of algorithm in recommended system. Finally, this paper summarizes the introduced algorithms and proposes the future prospects for MAB optimization algorithms.

BTV-CMAB: A Bi-Directional Trust Verification-Based Combinatorial Multiarmed Bandit Scheme for Mobile Crowdsourcing

BTV-CMAB: A Bi-Directional Trust Verification-Based Combinatorial Multiarmed Bandit Scheme for Mobile Crowdsourcing

Thompson Sampling-Based Partially Observable Online Change Detection for Exponential Families

This paper proposes a holistic sequential change detection framework for partially observable high-dimensional data streams with exponential-family distributions. The framework first proposes a general composite decomposition for exponential-family distributed data by projecting its natural parameter onto normal bases and abnormal bases, which enables efficient inference for sparse changes. Then, the inference results are used for detection scheme construction, and different types of test statistics can be compacted in our framework. Last, by further designing the test statistic as the reward function in the combinatorial multi-armed bandit problem, a Thompson sampling-based sensor allocation strategy is constructed to select the most anomalous variables. Theoretical properties of the detection framework are discussed. Finally, examples of Gaussian, Poisson, and binomial distributed data streams are given in numerical studies and case studies to evaluate the performance of our proposed method. History: Bianca Maria Colosimo served as the senior editor. Funding: C. Zhang is partially supported by the NSFC [Grants 71932006 and 72271138], the BNSF [Grant 9222014], and the ASFC [Grant 2020Z063058001]. H. Yan is partially supported by NIH [R21 AI157618] and NSF [CMMI 2316654]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijds.2022.00011 . Data Ethics & Reproducibility Note: No data ethics considerations are foreseen related to this paper. The code capsule is available on Code Ocean at https://codeocean.com/capsule/8794940/tree/v1 and in the e-Companion to this article (available at https://doi.org/10.1287/ijds.2022.00011 ).

Truthful and Dual-direction Combinatorial Multi-Armed Bandit Scheme to Maximize Profit for Mobile Crowd Sensing

Truthful and Dual-direction Combinatorial Multi-Armed Bandit Scheme to Maximize Profit for Mobile Crowd Sensing

A Differentially Private Approach for Budgeted Combinatorial Multi-armed Bandits

A Differentially Private Approach for Budgeted Combinatorial Multi-armed Bandits

Adaptive Label Propagation for Group Anomaly Detection in Large-Scale Networks

This paper concentrates on group anomalies in general large-scale networks. Existing algorithms on group anomalies mainly focus on homogeneous or bipartite networks, and thus are difficult to apply to heterogeneous networks directly. Moreover, these algorithms follow the non-overlapping hypothesis of groups implicitly, which is improper in many scenarios. In this paper, we introduce a novel algorithm called Adaptive Label Propagation (ALP) to solve these problems. ALP is designed based on label propagation (LP) frameworks, for the reason that LP-based frameworks are simple in thought and easy to scale. ALP is able to find overlapping groups by label propagation with belonging coefficients, and can be applied to heterogeneous networks for its design of adaptive neighbor weighting. Assigning different weights to neighbors in label propagation is a challenging task. Inspired by the combinatorial multi-armed bandit mechanism, ALP views the neighbors of each node as arms to be selected, and iteratively updates their weights by evaluating their expected rewards in following iterations. Experiments are conducted on four real-world networks. The results show that LP-based methods are effective for detecting group anomalies, and the comparison results with several state-of-the-art label propagation based community detection methods show the effectiveness of the proposed method

FedAB: Truthful Federated Learning With Auction-Based Combinatorial Multi-Armed Bandit

Federated Learning (FL) emerges as a new distributed machine learning (ML) paradigm that enables thousands of mobile devices to collaboratively train ML models using local data without compromising user privacy. However, the FL learning quality highly relies on the data contribution from the distributed mobile devices. Therefore, a well-designed incentive mechanism with effectiveness, fairness, and reciprocity is in urgent need to guarantee the stable participation of users. In this paper, we propose FedAB, an incentive and client selection strategy based on a novel multi-attribute reverse auction mechanism and a combinatorial multi-armed bandit (CMAB) algorithm. First, we develop a local contribution evaluation method based on importance sampling in the FL context. We then design a novel payment mechanism that is able to preserve individual rationality and incentive compatibility (truthfulness). At last, we design a UCB-based winner selection algorithm that is proven to achieve the server’s utility maximization with fairness and reciprocity. We have conducted extensive experiments on real datasets. The results demonstrate the superiority of FedAB, with a 10-50% improvement in total reward, final accuracy, and convergence speed compared to state-of-the-art solutions.

Spectrum Allocation and User Scheduling Based on Combinatorial Multi-Armed Bandit for 5G Massive MIMO.

As a key 5G technology, massive multiple-input multiple-output (MIMO) can effectively improve system capacity and reduce latency. This paper proposes a user scheduling and spectrum allocation method based on combinatorial multi-armed bandit (CMAB) for a massive MIMO system. Compared with traditional methods, the proposed CMAB-based method can avoid channel estimation for all users, significantly reduce pilot overhead, and improve spectral efficiency. Specifically, the proposed method is a two-stage method; in the first stage, we transform the user scheduling problem into a CMAB problem, with each user being referred to as a base arm and the energy of the channel being considered a reward. A linear upper confidence bound (UCB) arm selection algorithm is proposed. It is proved that the proposed user scheduling algorithm experiences logarithmic regret over time. In the second stage, by grouping the statistical channel state information (CSI), such that the statistical CSI of the users in the angular domain in different groups is approximately orthogonal, we are able to select one user in each group and allocate a subcarrier to the selected users, so that the channels of users on each subcarrier are approximately orthogonal, which can reduce the inter-user interference and improve the spectral efficiency. The simulation results validate that the proposed method has a high spectral efficiency.

Truth based three-tier Combinatorial Multi-Armed Bandit ecosystems for mobile crowdsensing

Many Multi-Armed Bandit (MAB) based workers selection schemes have been proposed to select high-quality workers to enhance the quality of tasks. However, in Mobile Crowd Sensing (MCS), a complex mutual effect exists among task requestors, the MCS platforms, and workers. Only considering the interaction of two sides doesn’t make MCS a balanced ecosystem. Therefore, it is urgent to establish a tripartite mutual incentive mechanism to make the MCS system a balanced ecosystem. In this paper, a truth based Three-tier Combinatorial Multi-Armed Bandit (TCMAB) incentive mechanism is proposed for selecting each other to maximize their revenues in MCS. In TCMAB, there exists a three-tier and two-way MAB-based incentive scheme. For the mutual interaction between the platform and the worker, a truth-based CMAB scheme is established for the platform to select high-quality workers, and also a CMAB scheme is proposed for workers to select a “good” platform to report data in order to maximize their revenue. Besides, for the mutual interaction between the task requestor and the platform, the platforms adopt the CMAB-based scheme to select a high-payment task requestor that gives high payment. And a platform selection scheme base on CMAB is also established for task requestors to select the platforms which have lower fees and higher quality. What’s more, we don’t adopt the assumption that platforms get the data quality as soon as they get data, but propose a data quality acquisition scheme based on the truth data discovery and cooperation frequency, which is the base to instruct the three-tier interaction, thus establish a kind of truth-based MCS interaction ecosystems. Simulation results show that the proposed TCMAB provides an effective solution for the problem of information elicitation without verification (IEWV) in MCS, and can improve the utilities, data quality, and applications quality for MCS, which is not achieved in the previous studies significantly.

LEO satellite assisted UAV distribution using combinatorial bandit with fairness and budget constraints.

In this paper, an integration between a low earth orbit satellite (LEO-Sat) and unmanned aerial vehicle (UAV) is proposed to assist users in post-disaster areas. In this scenario, multiple UAVs will be distributed to fully cover the victims and provide rescue services, while LEO-Sat provides backhaul links for UAVs to the ground base station (GBS). In this regard, we consider the problem of efficient UAVs distribution to maximize the total sum rate of the victims while assuring fairness in their coverage within the limited resources of UAVs batteries and LEO-Sat bandwidth. In this paper, UAV distribution problem is considered as a combinatorial multi-armed bandit (MAB) with arms' fairness and limited UAVs battery budget (CMAB-FB) constraints. Additionally, the utilization of LEO-Sat bandwidth resources is optimized based on the average traffic demands of the LEO-UAV links by means of gradient decent algorithm. The results of numerical analysis indicate that the proposed approach outperforms other naïve ben chmarks.

Federated Online Learning Aided Multi-Objective Proactive Caching in Heterogeneous Edge Networks

To address the drastic increase in multimedia traffic volume, mobile edge caching (MEC) has been exploited to reduce redundant data transmissions by equipping computation and storage capacity at the edge network. Previous works on learning-based caching problems often only concern pre-storing popular contents to satisfy users’ demands. In this work, we investigate the cache strategy design problem with two possibly conflicting objectives, namely, cache hit and cache profit, in heterogeneous multi-MEC server networks when content profiles are unknown. We then formulate this multi-objective caching problem as a Multi-agent Multi-objective Combinatorial Multi-Armed bandit (MMC-MAB) problem and propose a two-step caching framework that estimates content properties first and then optimizes cache placement. Specifically, to accommodate the system heterogeneity in estimation, we utilize an adaptive federated learning-based estimation to approach the unknown content popularity and profit profiles, which wisely use both local and external observations by adjusting the mixing factors. To address the multiple objective optimizations, we propose two effective methods, based on individual dominance and combinatorial dominance, to achieve adequate Pareto-optimal cache placement. Both theoretical results and comprehensive experiments clearly validate the effectiveness and efficiency of our proposed approaches.