No-regret Learning Algorithm Research Articles

No-regret learning for repeated non-cooperative games with lossy bandits

This paper considers no-regret learning for repeated continuous-kernel games with lossy bandit feedback. Since it is difficult to give an explicit model of the utility functions in dynamic environments, the players’ actions can only be learned with bandit feedback. Moreover, due to unreliable communication channels or privacy protection, the bandit feedback may be lost or dropped at random. Therefore, we study the asynchronous online learning strategy of the players to adaptively adjust the next actions for minimizing the long-term regret loss. The paper provides a novel no-regret learning algorithm, called Online Gradient Descent with lossy bandits (OGD-lb). We first give the regret analysis for concave games with differentiable and Lipschitz utilities. Then we show that the action profile converges to a Nash equilibrium with probability 1 when the game is also strictly monotone. We further provide the mean-squared convergence rate ONpi−2k−1/3 when the game is β-strongly monotone, where N denotes the number of players and pi is the update probability. In addition, we extend the algorithm to the case when the loss probability of the bandit feedback is unknown, and prove its almost sure convergence to Nash equilibrium for strictly monotone games. Finally, we take the resource management in fog computing as an application example, and carry out numerical experiments to empirically demonstrate the algorithm performance.

Learning in auctions: Regret is hard, envy is easy

A large line of recent work studies the welfare guarantees of simple and prevalent combinatorial auction formats, such as selling m items via simultaneous second price auctions (SiSPAs). These guarantees hold even when the auctions are repeatedly executed and the players use no-regret learning algorithms. Unfortunately, off-the-shelf no-regret algorithms for these auctions are computationally inefficient. We show that this obstacle is insurmountable: there are no polynomial-time no-regret algorithms for SiSPAs, unless RP⊇NP, even when bidders are unit-demand. Our lower bound raises the question of how good outcomes polynomially-bounded bidders may discover in such auctions. We propose a novel concept of learning in auctions, termed “no-envy learning”, and show that it is both efficiently implementable and results in approximately optimal welfare, even when the bidders have valuations from the broad class of fractionally subadditive valuations, assuming demand oracle access to the valuations, or coverage valuations, even without demand oracles.

Distributed Learning-Based Resource Allocation for Self-Organizing C-V2X Communication in Cellular Networks

In this paper, we investigate a resource allocation problem for a Cellular Vehicle to Everything (C-V2X) network to improve energy efficiency of the system. To address this problem, self-organizing mechanisms are proposed for joint and disjoint subcarrier and power allocation procedures which are performed in a fully distributed manner. A multi-agent Q-learning algorithm is proposed for the joint power and subcarrier allocation. In addition, for the sake of simplicity, it is decoupled into two sub-problems: a subcarrier allocation sub-problem and a power allocation sub-problem. First, to allocate the subcarrier among users, a distributed Q-learning method is proposed. Then, given the optimal subcarriers, a dynamic power allocation mechanism is proposed where the problem is modeled as a non-cooperative game. To solve the problem, a no-regret learning algorithm is utilized. To evaluate the performance of the proposed approaches, other learning mechanisms are used which are presented in fig.8. Simulation results show the multi-agent joint Q-learning algorithm yields significant performance gains of up to about 11% and 18% in terms of energy efficiency compared to proposed disjoint mechanism and the third disjoint Q-learning mechanism for allocating the power and subcarrier to each user; however, the multi-agent joint Q-learning algorithm uses more memory than disjoint methods.

No-Regret Matching Game Algorithm for NOMA Based UAV-Assisted NB-IoT Systems

NB-IoT is the standardized technology for machine type communication (MTC) in Long Term Evolution (LTE). NB-IoT can achieve IoT requirements nevertheless, it suffers a low rate and capacity. On the other hand, Unmanned aerial vehicles (UAV) and Non-Orthogonal Multiple Access (NOMA) are promising technology used to enhance the throughput, capacity, and coverage of wireless communication networks. In this paper, we propose a heterogeneous network scenario where a UAV small Base Station (UBS) is used to assist the LTE Macro Base Station (MBS) with the help of the Non-Orthogonal Multiple Access technique to solve the NB-IoT throughput and capacity issues. Matching game based no-regret learning algorithm is proposed to optimize the NB-IoT device association and using NOMA pairing at each base station to provide the maximum system total rate and capacity. Simulation results show that our proposed scheme increases the total rate of the system by 60% and the system capacity by at least 80%, compared to NOMA without UAV and the total rate and capacity of the system by 200% and 85% respectively, with OMA scheme.

Optimal No-Regret Learning in Strongly Monotone Games with Bandit Feedback

We consider online no-regret learning in unknown games with bandit feedback, where each agent only observes its reward at each time -- determined by all players' current joint action -- rather than its gradient. We focus on the class of smooth and strongly monotone games and study optimal no-regret learning therein. Leveraging self-concordant barrier functions, we first construct an online bandit convex optimization algorithm and show that it achieves the single-agent optimal regret of $\tilde{\Theta}(\sqrt{T})$ under smooth and strongly-concave payoff functions. We then show that if each agent applies this no-regret learning algorithm in strongly monotone games, the joint action converges in \textit{last iterate} to the unique Nash equilibrium at a rate of $\tilde{\Theta}(1/\sqrt{T})$. Prior to our work, the best-know convergence rate in the same class of games is $O(1/T^{1/3})$ (achieved by a different algorithm), thus leaving open the problem of optimal no-regret learning algorithms (since the known lower bound is $\Omega(1/\sqrt{T})$). Our results thus settle this open problem and contribute to the broad landscape of bandit game-theoretical learning by identifying the first doubly optimal bandit learning algorithm, in that it achieves (up to log factors) both optimal regret in the single-agent learning and optimal last-iterate convergence rate in the multi-agent learning. We also present results on several simulation studies -- Cournot competition, Kelly auctions, and distributed regularized logistic regression -- to demonstrate the efficacy of our algorithm.

Robustness enhancement of complex networks via No-Regret learning

Abstract Optimizing complex networks to be resilient against various attack models has been an important problem that is actively studied in the academia. In the proposed optimization method, individual node degrees are balanced iteratively based on the No-Regret learning algorithm, resulting in a robust network topology with increased resilience against outside attacks. Through simulation results, we show that the proposed robustness enhanced networks perform well under targeted attacks compared to the conventional optimized networks.

Distributed Learning for Energy-Efficient Resource Management in Self-Organizing Heterogeneous Networks

In heterogeneous networks, a dense deployment of base stations (BSs) leads to increased total energy consumption, and, consequently, increased cochannel interference (CCI). In this paper, to deal with this problem, self-organizing mechanisms are proposed, for joint channel and power allocation procedures, which are performed in a fully distributed manner. A dynamic channel allocation mechanism is proposed, in which the problem is modeled as a noncooperative game, and a no-regret learning algorithm is applied for solving the game. In order to improve the accuracy and reduce the effect of shadowing, we propose another channel allocation algorithm executed at each user equipment (UE). In this algorithm, each UE reports the channel with minimum CCI to its associated BS. Then, the BS selects its channel based on these received reports. To combat the energy consumption problem, BSs choose their transmission power by employing an on – off switching scheme. Simulation results show that the proposed mechanism, which is based on the second proposed channel allocation algorithm and combined with the on–off switching scheme, balances load among BSs. Furthermore, it yields significant performance gains up to about $40.3\%$ , $44.8\%$ , and $70.6\%$ in terms of average energy consumption, UE's rate, and BS's load, respectively, compared to a benchmark based on an interference-aware dynamic channel allocation algorithm.

Jamming-Resistant Learning in Wireless Networks

We consider capacity maximization in wireless networks under adversarial interference conditions. There are $n$ links, i.e., sender-receiver pairs, which repeatedly try to perform a successful transmission. In each time step, the success of attempted transmissions depends on interference conditions, which are captured by an interference model (e.g., the SINR model). Additionally, an adversarial jammer can render a $(1-\delta)$ -fraction of time steps in a time window unsuccessful. For this scenario, we analyze a framework for distributed no-regret learning algorithms to get provable approximation guarantees. We obtain an $O(1/\delta)$ -approximation for the problem of maximizing the number of successful transmissions. Our approach provides even a constant-factor approximation when the jammer exactly blocks a $(1-\delta)$ -fraction of time steps. In addition, we consider the parameters of the jammer being partially unknown to the algorithm, and we also consider a stochastic jammer, for which we obtain a constant-factor approximation after a polynomial number of time steps. We extend our results to more general settings, in which links arrive and depart dynamically, and where each sender tries to reach multiple receivers. Our algorithms perform favorably in simulations.

Opponent Modeling by Expectation–Maximization and Sequence Prediction in Simplified Poker

We consider the problem of learning an effective strategy online in a hidden information game against an opponent with a changing strategy. We want to model and exploit the opponent and make three proposals to do this; first, to infer its hidden information using an expectation–maximization (EM) algorithm; second, to predict its actions using a sequence prediction method; and third, to simulate games between our agent and our opponent model in-between games against the opponent. Our approach does not require knowledge outside the rules of the game, and does not assume that the opponent’s strategy is stationary. Experiments in simplified poker games show that it increases the average payoff per game of a state-of-the-art no-regret learning algorithm.

Algorithmic game theory and econometrics

The traditional econometrics approach for inferring properties of strategic interactions that are not fully observable in the data, heavily relies on the assumption that the observed strategic behavior has settled at an equilibrium. This assumption is not robust in complex economic environments such as online markets where players are typically unaware of all the parameters of the game in which they are participating, but rather only learn their utility after taking an action. Behavioral models from online learning theory have recently emerged as an attractive alternative to the equilibrium assumption and have been extensively analyzed from a theoretical standpoint in the algorithmic game theory literature over the past decade. In this letter we survey two recent works, [Nekipelov et al. 2015, Hoy et al. 2015], in which we take a learning agent approach to econometrics, i.e. infer properties of the game, such as private valuations or efficiency of observed allocation, by only assuming that the observed repeated behavior is the outcome of a no-regret learning algorithm, rather than a static equilibrium. In both works we apply our methods to datasets from Microsoft's sponsored search auction system.

Distributed Channel Selection for Interference Mitigation in Dynamic Environment: A Game-Theoretic Stochastic Learning Solution

In this paper, we investigate the problem of distributed channel selection for interference mitigation in a canonical communication network. The channel is assumed time-varying, and the active user set is considered dynamically variable due to the specific service requirement. This problem is formulated as an exact potential game, and the optimality property of the solution to this problem is first analyzed. Then, we design a low-complexity fully distributed no-regret learning algorithm for channel adaptation in a dynamic environment, where each active player can independently and automatically update its action with no information exchange. The proposed algorithm is proven to converge to a set of correlated equilibria with a probability of 1. Finally, we conduct simulations to demonstrate that the proposed algorithm achieves near-optimal performance for interference mitigation in dynamic environments.

Near-optimal no-regret algorithms for zero-sum games

We propose a new no-regret learning algorithm. When used against an adversary, our algorithm achieves average regret that scales optimally as O(1T) with the number T of rounds. However, when our algorithm is used by both players of a zero-sum game, their average regret scales as O(ln⁡TT), guaranteeing a near-linear rate of convergence to the value of the game. This represents an almost-quadratic improvement on the rate of convergence to the value of a zero-sum game known to be achievable by any no-regret learning algorithm. Moreover, it is essentially optimal as we also show a lower bound of Ω(1T) for all distributed dynamics, as long as the players do not know their payoff matrices in the beginning of the dynamics. (If they did, they could privately compute minimax strategies and play them ad infinitum.)

Distributed resource allocation in cognitive radio networks with a game learning approach to improve aggregate system capacity

This paper presents a game theoretic solution for joint channel allocation and power control in cognitive radio networks analyzed under the physical interference model. The objective is to find a distributed solution that maximizes the network utility, defined with different criteria, with limited information. The problem is addressed through a non-cooperative game based on local information. Although the existence of a pure Nash Equilibrium cannot be assured for this game, simulation results show that it exists with high probability and with a performance similar to that of a potential game, where each player requires overall network information. The obtained results are compared with a centralized heuristic genetic algorithm to show the correctness of the proposals. From this point, utility functions for the local game are modified to restrict the transmitted power to drive the solution to a more cooperative approach. To overcome the convergence limitations of the local game, no-regret learning algorithms are used to perform the joint channel and power allocation. These algorithms provide stable mixed strategies in any scenario with even better global performance. This opens an interesting perspective to develop realistic protocols based on the modeled interactions and increases the adaptability to perform efficient opportunistic spectrum access.

Sensor Networks for High-Confidence Cyber-Physical Systems

Technical advances in ubiquitous sensing, embedded computing, and wireless communication are leading to a new generation of engineered systems called cyber-physical systems (CPS). This field is attracting more and more attention from researchers, practitioners, as well as the governments. Technically, cyber-physical systems are integrations of computation, networking, and physical dynamics, in which embedded devices are massively networked to sense, monitor and control the physical world. CPS has been regarded as the next computing revolution. This revolution will be featured by the envisioned transform that CPS will make on how we interact with the physical world. To facilitate unprecedented interactions between human beings and the physical world, sensor networks will become a crucial ingredient of CPS due to the need for coupling geographically distributed computing devices with physical elements. The proliferation of affordable sensor network technologies has significantly contributed to recent progress in CPS. On the other hand, the unique features of CPS (e.g. cyber-physical coupling driven by new demands and applications) give rise to a lot of open challenges for design and deployment of sensor networks in such systems. In particular, high-confidence CPS requires the employed sensor networks to support real-time, dependable, safe, secure, and efficient operations. In this issue we are focused on latest research results in wireless sensor networks (WSNs) that address key issues related to high-confidence cyber-physical systems and applications. In response to our call-for-papers, we have received 36 submissions, out of which 15 papers are finally accepted as a result of thorough review process by international experts in respective areas. The selection provides a glimpse of the state-of-the-art research in the field. In the paper Game Theoretic Approach for Interuser Interference Reduction in Body Sensor Networks, Wu et al present a decentralized inter-user interference reduction scheme with non-cooperative game for body sensor networks (BSNs). A no-regret learning algorithm for reducing the effect of the inter-user interference with low power consumption is proposed. The correctness and effectiveness of the proposed scheme are theoretically proved, and experimental results demonstrate that the effect of inter-user interference can be reduced effectively with low power consumption.

A Game Theoretic Approach for Interuser Interference Reduction in Body Sensor Networks

As a kind of small-scale cyber-physical systems (CPSs), body sensor networks (BSNs) can provide the pervasive, long-term, and real-time health monitoring. A high degree of quality-of-service (QoS) for BSN is extremely required to meet some critical services. Interuser interference between different BSNs can cause unreliable critical data transmission and high bite error rate. In this paper, a game theoretic decentralized interuser interference reduction scheme for BSN is proposed. The selection of the channel and transmission power is modeled as a noncooperative game between different BSNs congregating in the same area. Each BSN measures the interference from other BSNs and then can adaptively select the suitable channel and transmission power by utilizing no-regret learning algorithm. The correctness and effectiveness of our proposed scheme are theoretically proved, and the extensive experimental results demonstrate that the effect of inter-user interference can be reduced effectively with low power consumption.

A non-cooperative game-theoretic approach to channel assignment in multi-channel multi-radio wireless networks

Channel assignment in multi-channel multi-radio wireless networks poses a significant challenge due to scarcity of number of channels available in the wireless spectrum. Further, additional care has to be taken to consider the interference characteristics of the nodes in the network especially when nodes are in different collision domains. This work views the problem of channel assignment in multi-channel multi-radio networks with multiple collision domains as a non-cooperative game where the objective of the players is to maximize their individual utility by minimizing its interference. Necessary and sufficient conditions are derived for the channel assignment to be a Nash Equilibrium (NE) and efficiency of the NE is analyzed by deriving the lower bound of the price of anarchy of this game. A new fairness measure in multiple collision domain context is proposed and necessary and sufficient conditions for NE outcomes to be fair are derived. The equilibrium conditions are then applied to solve the channel assignment problem by proposing three algorithms, based on perfect/imperfect information, which rely on explicit communication between the players for arriving at an NE. A no-regret learning algorithm known as Freund and Schapire Informed algorithm, which has an additional advantage of low overhead in terms of information exchange, is proposed and its convergence to the stabilizing outcomes is studied. New performance metrics are proposed and extensive simulations are done using Matlab to obtain a thorough understanding of the performance of these algorithms on various topologies with respect to these metrics. It was observed that the algorithms proposed were able to achieve good convergence to NE resulting in efficient channel assignment strategies.

Editor's introduction

I am very happy to introduce Issue 9.1 of SIGecom Exchanges. All the contributions in this issue concern markets of various kinds. In "The Pathologies of Online Display Advertising Marketplaces," Edelman discusses various challenges faced by marketplaces for banner ads on websites (in contrast to ads displayed next to search results). In "Designing Aggregation Mechanisms for Reputation Systems in Online Marketplaces," Aperjis and Johari discuss mechanisms for aggregating ratings that a seller has received; they consider mechanisms that average over a fixed window of past transactions (and discuss how to optimize the window size), as well as a broader class of mechanisms. In "Matching, Cardinal Utility, and Social Welfare," Anshelevich and Das discuss matching markets where the participants' preferences are modeled cardinally (whereas they are often modeled ordinally in this type of market). In "Competitive Equilibria in Matching Markets with Budgets," N. Chen, Deng, and Ghosh consider the Shapley-Shubik assignment model (with general utility functions) and extend it with budget constraints; they then study how to compute a competitive equilibrium (if one still exists). In "Connections Between Markets and Learning," Y. Chen and Vaughan discuss the mathematical connections between market maker mechanisms for prediction markets, and no-regret learning, showing that any cost-function-based prediction market with bounded loss can be interpreted as a no-regret learning algorithm, and studying what the resulting no-regret learning algorithms look like. In "Competition in Mechanisms," Pai considers settings with multiple sellers that compete with each other by announcing mechanisms, and discusses some of the key issues as well as recent results. Finally, in "Approximability of Combinatorial Problems with Multi-agent Submodular Cost Functions," Goel, Karande, Tripathi, and Wang consider computational problems that come up in winner determination in combinatorial procurement auctions: a feasible set of elements must be selected, and there are multiple agents that can provide subsets of these elements. They consider the case where each agent has a submodular cost function, and establish upper and lower bounds on the approximability of these problems.