Multi-agent Multi-armed Bandit Research Articles

Finite-Time Frequentist Regret Bounds of Multi-Agent Thompson Sampling on Sparse Hypergraphs

We study the multi-agent multi-armed bandit (MAMAB) problem, where agents are factored into overlapping groups. Each group represents a hyperedge, forming a hypergraph over the agents. At each round of interaction, the learner pulls a joint arm (composed of individual arms for each agent) and receives a reward according to the hypergraph structure. Specifically, we assume there is a local reward for each hyperedge, and the reward of the joint arm is the sum of these local rewards. Previous work introduced the multi-agent Thompson sampling (MATS) algorithm and derived a Bayesian regret bound. However, it remains an open problem how to derive a frequentist regret bound for Thompson sampling in this multi-agent setting. To address these issues, we propose an efficient variant of MATS, the epsilon-exploring Multi-Agent Thompson Sampling (eps-MATS) algorithm, which performs MATS exploration with probability epsilon while adopts a greedy policy otherwise. We prove that eps-MATS achieves a worst-case frequentist regret bound that is sublinear in both the time horizon and the local arm size. We also derive a lower bound for this setting, which implies our frequentist regret upper bound is optimal up to constant and logarithm terms, when the hypergraph is sufficiently sparse. Thorough experiments on standard MAMAB problems demonstrate the superior performance and the improved computational efficiency of eps-MATS compared with existing algorithms in the same setting.

Distributed Consensus Algorithm for Decision-Making in Multi-agent Multi-armed Bandit

Distributed Consensus Algorithm for Decision-Making in Multi-agent Multi-armed Bandit

Multi-Agent Multi-Armed Bandit Learning for Grant-Free Access in Ultra-Dense IoT Networks

Multi-Agent Multi-Armed Bandit Learning for Grant-Free Access in Ultra-Dense IoT Networks

An Efficient Algorithm for Fair Multi-Agent Multi-Armed Bandit with Low Regret

Recently a multi-agent variant of the classical multi-armed bandit was proposed to tackle fairness issues in online learning. Inspired by a long line of work in social choice and economics, the goal is to optimize the Nash social welfare instead of the total utility. Unfortunately previous algorithms either are not efficient or achieve sub-optimal regret in terms of the number of rounds. We propose a new efficient algorithm with lower regret than even previous inefficient ones. We also complement our efficient algorithm with an inefficient approach with regret that matches the lower bound for one agent. The experimental findings confirm the effectiveness of our efficient algorithm compared to the previous approaches.

Robust Multi-Agent Bandits Over Undirected Graphs

We consider a multi-agent multi-armed bandit setting in which n honest agents collaborate over a network to minimize regret but m malicious agents can disrupt learning arbitrarily. Assuming the network is the complete graph, existing algorithms incur O((m + K/n) łog (T) / Δ ) regret in this setting, where K is the number of arms and Δ is the arm gap. For m łl K, this improves over the single-agent baseline regret of O(Kłog(T)/Δ). In this work, we show the situation is murkier beyond the case of a complete graph. In particular, we prove that if the state-of-the-art algorithm is used on the undirected line graph, honest agents can suffer (nearly) linear regret until time is doubly exponential in K and n . In light of this negative result, we propose a new algorithm for which the i -th agent has regret O(( dmal (i) + K/n) łog(T)/Δ) on any connected and undirected graph, where dmal(i) is the number of i 's neighbors who are malicious. Thus, we generalize existing regret bounds beyond the complete graph (where dmal(i) = m), and show the effect of malicious agents is entirely local (in the sense that only the dmal (i) malicious agents directly connected to i affect its long-term regret).

Reinforcement learning for cost-effective IoT service caching at the edge

In the edge computing environment, Internet of Things (IoT) application service providers can rent resources from edge servers to cache their service items such as datasets and code libraries, and thus significantly reducing the service request latency and the core network traffic. Since IoT service providers need to pay for the rented edge computing resources, it is essential to find a dynamical service caching strategy to minimize the service cost while optimizing the performance objective such as service latency reduction. However, most of the existing studies either overlooked the problem of collaborative service caching or failed to consider the system's long-term service cost and latency. In this paper, to address such a problem, we coordinate multiple edge servers to cache service items and formulate the collaborative service caching problem using a multi-agent multi-armed bandit model. Furthermore, we propose a utility-aware collaborative service caching (UACSC) scheme based on a multi-agent reinforcement learning. The UACSC scheme can coordinate multiple edge servers to make a dynamic joint caching decision, aiming at maximizing the system's long-term utility. To evaluate the performance of our proposed scheme, we implement four representative baseline algorithms and compare them with six different performance metrics. In addition, a real-world case study is also presented to demonstrate the effectiveness of the UACSC scheme. Comprehensive experimental results show that the UACSC scheme can effectively coordinate multiple edge servers to cache service items, and achieve higher service latency reduction and lower service cost compared with other baseline algorithms.

Modelling Cournot Games as Multi-agent Multi-armed Bandits

We investigate the use of a multi-agent multi-armed bandit (MA-MAB) setting for modeling repeated Cournot oligopoly games, where the firms acting as agents choose from the set of arms representing production quantity (a discrete value). Agents interact with separate and independent bandit problems. In this formulation, each agent makes sequential choices among arms to maximize its own reward. Agents do not have any information about the environment; they can only see their own rewards after taking an action. However, the market demand is a stationary function of the total industry output, and random entry or exit from the market is not allowed. Given these assumptions, we found that an ε-greedy approach offers a more viable learning mechanism than other traditional MAB approaches, as it does not require any additional knowledge of the system to operate. We also propose two novel approaches that take advantage of the ordered action space: ε-greedy+HL and ε-greedy+EL. These new approaches help firms to focus on more profitable actions by eliminating less profitable choices and hence are designed to optimize the exploration. We use computer simulations to study the emergence of various equilibria in the outcomes and do the empirical analysis of joint cumulative regrets.

Multi-Connection Based Scalable Video Streaming in UDNs: A Multi-Agent Multi-Armed Bandit Approach

Scalable video coding (SVC) has received much attention for video transmission over wireless due to its flexibility. However, most previous work only considered SVC video streaming from a single base station (BS). At present, the densification of BSs enables a user equipment (UE) to connect to multiple BSs in ultra-dense networks (UDNs). In this paper, we consider the problem of SVC video streaming in a UDN, which allows different layers of a video block to be downloaded from different BSs. An optimization problem is formulated aiming to maximize the quality of experience (QoE) of users by selecting the optimal connection strategy and optimal number of video layers. Considering the complexity, to efficiently solve the problem in a distributed manner, the problem of choosing connection strategy is formulated as a multi-agent multi-armed bandit (MA-MAB) problem with only few information exchange. Each user can adapt its connection strategy in a distributed self-learning system. To obtain the optimal arm for the MA-MAB problem, we propose a multi-user arm decision algorithm. To avoid large computation and handover costs, we adopt the same connection strategy for the entire video sequence. Then for each video block, with the given connection strategy, the number of video layers is adjusted adaptively according to dynamic network conditions. Finally, based on the above designs, we provide the SVC-based video downloading scheme to obtain an approximate optimal solution to the original optimization problem. Extensive simulations and comparisons show the feasibility and superiority of the proposed scheme.

Low-Latency Federated Learning Over Wireless Channels With Differential Privacy

In federated learning (FL), model training is distributed over clients and local models are aggregated by a central server. The performance of uploaded models in such situations can vary widely due to imbalanced data distributions, potential demands on privacy protections, and quality of transmissions. In this paper, we aim to minimize FL training delay over wireless channels, constrained by overall training performance as well as each client’s differential privacy (DP) requirement. We solve this problem in a multi-agent multi-armed bandit (MAMAB) framework to deal with the situation where there are multiple clients confronting different unknown transmission environments, e.g., channel fading and interference. Specifically, we first transform long-term constraints on both training performance and each client’s DP into a virtual queue based on the Lyapunov drift technique. Then, we convert the MAMAB to a max-min bipartite matching problem at each communication round, by estimating rewards with the upper confidence bound (UCB) approach. More importantly, we propose two efficient solutions to this matching problem, i.e., a modified Hungarian algorithm and greedy matching with a better alternative (GMBA), of which the former can achieve the optimal solution with high complexity while the latter approaches a better trade-off by enabling verified low-complexity with little performance loss. In addition, we develop an upper bound on the expected regret of this MAMAB based FL framework, which shows a linear growth over the logarithm of communication rounds, justifying its theoretical feasibility. Extensive experimental results are conducted to validate the effectiveness of our proposed algorithms, and the impacts of various parameters on the FL performance over wireless edge networks are also discussed.

Multi-Agent Multi-Armed Bandit Learning for Online Management of Edge-Assisted Computing

By orchestrating resources of edge and core network, the delays of edge-assisted computing can decrease. Offloading scheduling is challenging though, especially in the presence of many edge devices with randomly varying link and computing conditions. This paper presents a new online learning-based approach to the offloading scheduling, where multi-agent multi-armed bandit (MA-MAB) learning is designed to exploit the randomly varying conditions and asymptotically minimize the computing delay. We first propose a combinatorial bandit upper confidence bound (CB-UCB) algorithm, where users collectively feed back the observed delays of all edge devices and links. The optimistic bound of the delay is derived to facilitate centralized offloading scheduling for all users. In addition, we put forth a distributed bandit upper confidence bound (DB-UCB) algorithm, where users take random turns to make conflict-free, distributed selections of edge devices. The optimistic confidence bound of each user is developed to allow the user’s selection only based on its own observations and decisions. Furthermore, we establish the asymptotic optimality of the proposed algorithms by proving the sublinearity of their regrets, and that the random turns the users take to make decisions do not compromise the asymptotic optimality of the DB-UCB algorithm, as corroborated by numerical simulations.

Bayesian Algorithms for Decentralized Stochastic Bandits

We study a decentralized cooperative multi-agent multi-armed bandit (MAB) problem with K arms and N agents connected over a network. In this model, each arm's reward distribution is the same for every agent, and rewards are drawn independently across agents and over time steps. At each iteration, agents independently choose an arm to play and exchange at most poly(K) real-valued messages with their neighbors. Existing lower bound on the average per-agent regret over the network shows that cooperation with other agents can achieve a reduction of O(\frac 1N) over when playing in isolation. Motivated by this, we study a message-passing algorithm that can be combined with existing Bayesian MAB algorithms. Using this, we propose a decentralized Thompson Sampling (TS) algorithm and a decentralized Bayes-UCB algorithm. Under decentralized TS for bounded rewards, we establish a problem-dependent upper bound on the average per-agent regret in terms of the number of agents in the network and the network topology. For Bernoulli rewards, our upper bound asymptotically matches the lower bound. We empirically show that the proposed decentralized TS algorithm incurs significantly lesser per-agent regret than previously proposed algorithms. Furthermore, we show that the proposed decentralized TS can be extended to general bandit problems, where posterior distribution cannot be computed in closed form. We combine our decentralized TS algorithm with Variational Inference to apply our proposed algorithm to complex realistic reward distributions in a computationally efficient manner. We implement our proposed decentralized TS under gossip protocol and over time-varying networks, where each communication link has a fixed probability of failure and show that it incurs logarithmic regret.

Distributed Online Caching for High-Definition Maps in Autonomous Driving Systems

This letter investigates online high-definition (HD) map caching for autonomous driving in vehicular networks when vehicles' requests and trajectories are unknown in advance. We first introduce a general HD map model that divides each HD map into different sub-maps to accommodate different driving functionalities. Then we introduce a service model for road side units (RSUs) by considering both the freshness of the dynamic sub-maps that are locally saved at each vehicle and the retrieving cost of sub-maps that are not cached at RSUs but must be needed for high-level driving control. After that, we propose a distributed multi-agent multi-armed bandit (MAMAB) algorithm for each RSU to learn its own cache strategy independently for maximizing the accumulated cache utility over a finite time horizon. Simulation results are provided to validate the effectiveness of our proposed algorithm.

Decentralized Multi-Agent Multi-Armed Bandit Learning With Calibration for Multi-Cell Caching

This paper investigates online decentralized cache strategy design in multi-cell networks without the knowledge of user preference. The goal is to minimize the cumulative transmission delay of multi-cell networks within a finite time interval. Each small base station (SBS) aims to decide on its own cache action autonomously based on its past observations and limited information transmitted from other SBSs without a central controller. To coordinate the cache actions of different SBSs in a decentralized manner, we first propose an ϵ-calibration learning algorithm for each SBS to predict the cache strategy of other SBSs in real-time, which can progressively improve the accuracy in cache action prediction. Then a decentralized multi-agent multi-armed bandit (MAMAB) algorithm is developed for each SBS to decide its own cache strategy based jointly on its past observations and estimated upcoming cache action of other SBSs. This decentralized MAMAB algorithm with ϵ-calibration enables multiple SBSs to converge to a reasonable joint cache action and realize a cooperative cache decision making in a decentralized manner with limited information exchange. Simulation results demonstrate that our proposed decentralized caching algorithm outperforms other decentralized caching algorithms and can rapidly approach towards the centralized caching solutions.

Multi-Agent Thompson Sampling for Bandit Applications with Sparse Neighbourhood Structures

Multi-agent coordination is prevalent in many real-world applications. However, such coordination is challenging due to its combinatorial nature. An important observation in this regard is that agents in the real world often only directly affect a limited set of neighbouring agents. Leveraging such loose couplings among agents is key to making coordination in multi-agent systems feasible. In this work, we focus on learning to coordinate. Specifically, we consider the multi-agent multi-armed bandit framework, in which fully cooperative loosely-coupled agents must learn to coordinate their decisions to optimize a common objective. We propose multi-agent Thompson sampling (MATS), a new Bayesian exploration-exploitation algorithm that leverages loose couplings. We provide a regret bound that is sublinear in time and low-order polynomial in the highest number of actions of a single agent for sparse coordination graphs. Additionally, we empirically show that MATS outperforms the state-of-the-art algorithm, MAUCE, on two synthetic benchmarks, and a novel benchmark with Poisson distributions. An example of a loosely-coupled multi-agent system is a wind farm. Coordination within the wind farm is necessary to maximize power production. As upstream wind turbines only affect nearby downstream turbines, we can use MATS to efficiently learn the optimal control mechanism for the farm. To demonstrate the benefits of our method toward applications we apply MATS to a realistic wind farm control task. In this task, wind turbines must coordinate their alignments with respect to the incoming wind vector in order to optimize power production. Our results show that MATS improves significantly upon state-of-the-art coordination methods in terms of performance, demonstrating the value of using MATS in practical applications with sparse neighbourhood structures.

Collaborative Multi-Agent Multi-Armed Bandit Learning for Small-Cell Caching

This paper investigates learning-based caching in small-cell networks (SCNs) when user preference is unknown. The goal is to optimize the cache placement in each small base station (SBS) for minimizing the system long-term transmission delay. We model this sequential multi-agent decision making problem in a multi-agent multi-armed bandit (MAMAB) perspective. Rather than estimating user preference first and then optimizing the cache strategy, we propose several MAMAB-based algorithms to directly learn the cache strategy online in both stationary and non-stationary environment. In the stationary environment, we first propose two high-complexity agent-based collaborative MAMAB algorithms with performance guarantee. Then we propose a low-complexity distributed MAMAB which ignores the SBS coordination. To achieve a better balance between SBS coordination gain and computational complexity, we develop an edge-based collaborative MAMAB with the coordination graph edge-based reward assignment method. In the non-stationary environment, we modify the MAMAB-based algorithms proposed in the stationary environment by proposing a practical initialization method and designing new perturbed terms to adapt to the dynamic environment. Simulation results are provided to validate the effectiveness of our proposed algorithms. The effects of different parameters on caching performance are also discussed.

Multi-Agent Reinforcement Learning for Efficient Content Caching in Mobile D2D Networks

To address the increase of multimedia traffic dominated by streaming videos, user equipment (UE) can collaboratively cache and share contents to alleviate the burden of base stations. Prior work on device-to-device (D2D) caching policies assumes perfect knowledge of the content popularity distribution. Since the content popularity distribution is usually unavailable in advance, a machine learning-based caching strategy that exploits the knowledge of content demand history would be highly promising. Thus, we design D2D caching strategies using multi-agent reinforcement learning in this paper. Specifically, we model the D2D caching problem as a multi-agent multi-armed bandit problem and use Q-learning to learn how to coordinate the caching decisions. The UEs can be independent learners (ILs) if they learn the Q-values of their own actions, and joint action learners (JALs) if they learn the Q-values of their own actions in conjunction with those of the other UEs. As the action space is very vast leading to high computational complexity, a modified combinatorial upper confidence bound algorithm is proposed to reduce the action space for both IL and JAL. The simulation results show that the proposed JAL-based caching scheme outperforms the IL-based caching scheme and other popular caching schemes in terms of average downloading latency and cache hit rate.

Multi-Agent Reinforcement Learning Based Cooperative Content Caching for Mobile Edge Networks

To address the drastic growth of data traffic dominated by streaming of video-on-demand files, mobile edge caching/computing (MEC) can be exploited to develop intelligent content caching at mobile network edges to alleviate redundant traffic and improve content delivery efficiency. Under the MEC architecture, content providers (CPs) can deploy popular video files at MEC servers to improve users' quality of experience (QoE). Designing an efficient content caching policy is crucial for CPs due to the content dynamics, unknown spatial-temporal traffic demands, and limited service capacity. The knowledge of users' preference is very useful and important for efficient content caching, yet often unavailable in advance. Under this circumstance, machine learning can be used to learn the users' preference based on historical demand information and decide the video files to be cached at the MEC servers. In this paper, we propose a multi-agent reinforcement learning (MARL)-based cooperative content caching policy for the MEC architecture when the users' preference is unknown and only the historical content demands can be observed. We formulate the cooperative content caching problem as a multi-agent multi-armed bandit problem and propose a MARL-based algorithm to solve the problem. The simulation experiments are conducted based on a real dataset from MovieLens and the numerical results show that the proposed MARL-based cooperative content caching scheme can significantly reduce content downloading latency and improve content cache hit rate when compared with other popular caching schemes.