Multi-player Multi-armed Bandit Research Articles

Distributed Learning and Coordination in Cognitive Infrastructureless Networks of Unknown Size

The cognitive infrastructureless network has received significant attention to reduce signaling load in next-generation networks, which are expected to be ultradense with very high peak rate but relatively lower expected traffic per user. Research in this area has evolved from the distributed algorithms requiring prior knowledge of the number of secondary users (SUs) $U$ to the existing algorithms, which can estimate $U$ independently by counting the number of collisions. The major drawback of these algorithms is the large number of collisions leading to wastage of power and bring down the effective life of battery operated SUs. In this paper, we develop algorithms that learn faster and incurs fewer collisions. We assume unknown $U$ with two types of networks: fixed $U$ (i.e., static networks) and time-varying $U$ (i.e., dynamic networks). Proposed algorithms are based on the multiplayer multi-armed bandit (MAB) approach. We show that the proposed algorithms offer constant regret (i.e., throughput loss) with high probability and sub-linear regret in static and dynamic networks, respectively. Theoretical analysis, simulation, and experimental results demonstrate the efficacy of the proposed algorithms in terms of vacant spectrum utilization, regret, and the number of collisions. Fewer collisions significantly increase the operational life of SUs.

Joint resource allocation in underwater acoustic communication networks: A game-based hierarchical adversarial multiplayer multiarmed bandit algorithm

In this study, an adversarial multiplayer multiarmed bandit (MAB) game is employed to model the problem of joint channel and power allocation in multiuser underwater acoustic communication networks (UACNs). Moreover, this study presents a distributed hierarchical learning algorithm that does not require any prior environmental information and direct information exchange among users. This algorithm has a two-tier learning approach that effectively improves user learning ability and decreases learning time. In upper learning, each user formulates a strategy by learning the actual played reward. Outdated virtual information, which can be obtained as the reward of a past-played strategy, is learned in lower learning. The dynamic lower learning mechanism is proposed to prevent falling into an inadequate local extreme value. The algorithm has high tolerance for delay and noncomplete information because of its unique learning behavior. Simulation results showed that the proposed algorithm achieves a high level of performance.

On Regret-Optimal Learning in Decentralized Multiplayer Multiarmed Bandits

We consider the problem of learning in single-player and multiplayer multiarmed bandit models. Bandit problems are classes of online learning problems that capture exploration versus exploitation tradeoffs. In a multiarmed bandit model, players can pick among many arms, and each play of an arm generates an i.i.d. reward from an unknown distribution. The objective is to design a policy that maximizes the expected reward over a time horizon for a single-player setting and the sum of expected rewards for the multiplayer setting. In the multiplayer setting, arms may give different rewards to different players. There is no separate channel for coordination among the players. Any attempt at communication is costly and adds to regret. We propose two decentralizable policies, $\tt E^3$ ( $\tt E$ - $\tt cubed$ ) and $\tt E^3$ - $\tt TS$ , that can be used in both single-player and multiplayer settings. These policies are shown to yield expected regret that grows, at most, as $O(\log ^{1+\delta } T)$ (and $O(\log T)$ under some assumption). It is well known that $O(\log T)$ is the lower bound on the rate of growth of regret even in a centralized case. The proposed algorithms improve on prior work where regret grew at $O(\log ^2 T)$ . More fundamentally, these policies address the question of additional cost incurred in decentralized online learning, suggesting that there is, at most, an $\delta$ -factor cost in terms of the order of regret. This solves a problem of relevance in many domains and had been open for a while.

Joint Channel Selection and Power Control in Infrastructureless Wireless Networks: A Multiplayer Multiarmed Bandit Framework

This paper deals with the problem of efficient resource allocation in dynamic infrastructureless wireless networks. Assuming a reactive interference-limited scenario, each transmitter is allowed to select one frequency channel (from a common pool) together with a power level at each transmission trial; hence, for all transmitters, not only the fading gain, but also the number of interfering transmissions and their transmit powers are varying over time. Due to the absence of a central controller and time-varying network characteristics, it is highly inefficient for transmitters to acquire global channel and network knowledge. Therefore a reasonable assumption is that transmitters have no knowledge of fading gains, interference, and network topology. Each transmitting node selfishly aims at maximizing its average reward (or minimizing its average cost), which is a function of the action of that specific transmitter as well as those of all other transmitters. This scenario is modeled as a multi-player multi-armed adversarial bandit game, in which multiple players receive an a priori unknown reward with an arbitrarily time-varying distribution by sequentially pulling an arm, selected from a known and finite set of arms. Since players do not know the arm with the highest average reward in advance, they attempt to minimize their so-called regret, determined by the set of players' actions, while attempting to achieve equilibrium in some sense. To this end, we design in this paper two joint power level and channel selection strategies. We prove that the gap between the average reward achieved by our approaches and that based on the best fixed strategy converges to zero asymptotically. Moreover, the empirical joint frequencies of the game converge to the set of correlated equilibria. We further characterize this set for two special cases of our designed game.

Channel Selection for Network-Assisted D2D Communication via No-Regret Bandit Learning With Calibrated Forecasting

We consider the distributed channel selection problem in the context of device-to-device (D2D) communication as an underlay to a cellular network. Underlaid D2D users communicate directly by utilizing the cellular spectrum but their decisions are not governed by any centralized controller. Selfish D2D users that compete for access to the resources construct a distributed system, where the transmission performance depends on channel availability and quality. This information, however, is difficult to acquire. Moreover, the adverse effects of D2D users on cellular transmissions should be minimized. In order to overcome these limitations, we propose a network-assisted distributed channel selection approach in which D2D users are only allowed to use vacant cellular channels. This scenario is modeled as a multi-player multi-armed bandit game with side information, for which a distributed algorithmic solution is proposed. The solution is a combination of no-regret learning and calibrated forecasting, and can be applied to a broad class of multi-player stochastic learning problems, in addition to the formulated channel selection problem. Analytically, it is established that this approach not only yields vanishing regret (in comparison to the global optimal solution), but also guarantees that the empirical joint frequencies of the game converge to the set of correlated equilibria.

Decentralized Learning for Multiplayer Multiarmed Bandits

We consider the problem of distributed online learning with multiple players in multiarmed bandit (MAB) models. Each player can pick among multiple arms. When a player picks an arm, it gets a reward. We consider both independent identically distributed (i.i.d.) reward model and Markovian reward model. In the i.i.d. model, each arm is modeled as an i.i.d. process with an unknown distribution with an unknown mean. In the Markovian model, each arm is modeled as a finite, irreducible, aperiodic and reversible Markov chain with an unknown probability transition matrix and stationary distribution. The arms give different rewards to different players. If two players pick the same arm, there is a collision, and neither of them get any reward. There is no dedicated control channel for coordination or communication among the players. Any other communication between the users is costly and will add to the regret. We propose an online index-based distributed learning policy called dUCB4 algorithm that trades off exploration versus exploitation in the right way, and achieves expected regret that grows at most as near- O(log2T). The motivation comes from opportunistic spectrum access by multiple secondary users in cognitive radio networks wherein they must pick among various wireless channels that look different to different users. This is the first distributed learning algorithm for multiplayer MABs with heterogeneous players (that have player-dependent rewards) to the best of our knowledge.