Regret Matching Research Articles

User-Aware Game Theoretic Approach for Demand Management

Demand-management programs intend to maintain supply-demand balance and reduce the total energy cost. In this paper, we propose a user-aware demand-management approach that manages residential loads while taking into consideration user preferences. Maximizing users' savings and comfort can be two contradicting objectives. We identify a trade-off between these two objectives and propose an energy consumption optimization model, as well as a game theoretic approach to take this trade-off into account. User comfort is modeled in a simple yet effective way that considers waiting time, type of appliance, as well as a weight factor to prioritize comfort over savings. The proposed game is based on a modified regret matching procedure and borrows advantages of both centralized and decentralized schemes. Through simulations, we show that the proposed approach is scalable, converges in acceptable times, introduces a very limited amount of overhead in the system, achieves very high cost savings, and preserves users' preferences. Extensive simulations are used to evaluate the performance of the optimization model and the proposed approach.

Dynamic channel assignment for wireless sensor networks: A regret matching based approach

Multiple channels in Wireless Sensor Networks (WSNs) are often exploited to support parallel transmission and to reduce interference. However, the extra overhead posed by the multi-channel usage coordination dramatically challenges the energy-constrained WSNs. In this paper, we propose a Regret Matching based Channel Assignment algorithm (RMCA) to address this challenge, in which each sensor node updates its choice of channels according to the historical record of these channels’ performance to reduce interference. The advantage of RMCA is that it is highly distributed and requires very limited information exchange among sensor nodes. It is proved that RMCA converges almost surely to the set of correlated equilibrium. Moreover, RMCA can adapt the channel assignment among sensor nodes to the time-variant flows and network topology. Simulations show that RMCA achieves better network performance in terms of both delivery ratio and packet latency than CONTROL <citerefgrp xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><citeref refid="ref1"> </citeref></citerefgrp> , MMSN <citerefgrp xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><citeref refid="ref2"/></citerefgrp> and randomized CSMA. In addition, real hardware experiments are conducted to demonstrate that RMCA is easy to be implemented and performs better.

ALTERNATIVE SELECTION FUNCTIONS FOR INFORMATION SET MONTE CARLO TREE SEARCH

We evaluate the performance of various selection methods for the Monte Carlo Tree Search algorithm in two-player zero-sum extensive-form games with imperfect information. We compare the standard Upper Confident Bounds applied to Trees (UCT) along with the less common Exponential Weights for Exploration and Exploitation (Exp3) and novel Regret matching (RM) selection in two distinct imperfect information games: Imperfect Information Goofspiel and Phantom Tic-Tac-Toe. We show that UCT after initial fast convergence towards a Nash equilibrium computes increasingly worse strategies after some point in time. This is not the case with Exp3 and RM, which also show superior performance in head-to-head matches.

Regret Transfer and Parameter Optimization

Regret matching is a widely-used algorithm for learning how to act. We begin by proving that regrets on actions in one setting (game) can be transferred to warm start the regrets for solving a different setting with same structure but different payoffs that can be written as a function of parameters. We prove how this can be done by carefully discounting the prior regrets. This provides, to our knowledge, the first principled warm-starting method for no-regret learning. It also extends to warm-starting the widely-adopted counterfactual regret minimization (CFR) algorithm for large incomplete-information games; we show this experimentally as well. We then study optimizing a parameter vector for a player in a two-player zero-sum game (e.g., optimizing bet sizes to use in poker). We propose a custom gradient descent algorithm that provably finds a locally optimal parameter vector while leveraging our warm-start theory to significantly save regret-matching iterations at each step. It optimizes the parameter vector while simultaneously finding an equilibrium. We present experiments in no-limit Leduc Hold'em and no-limit Texas Hold'em to optimize bet sizing. This amounts to the first action abstraction algorithm (algorithm for selecting a small number of discrete actions to use from a continuum of actions---a key preprocessing step for solving large games using current equilibrium-finding algorithms) with convergence guarantees for extensive-form games.

Impacts of unreliable communication and modified regret matching based anti-jamming approach in smart microgrid

Demand response management (DRM) is one of the main features of smart grid, which is realized via bidirectional communications between the power provider and the consumers. Due to the vulnerabilities of communication channels, especially the wireless networks, communication is not perfect in practice and will be threatened by kinds of attacks, among which jamming attack is deemed as the primary one. In this paper, we consider jamming attack in the wireless communication of a smart microgrid. Firstly, the DRM performance degradation induced by jamming attack is fully studied and explicitly presented, by considering both packet loss ratio and load estimation error. Then, a modified regret matching based anti-jamming algorithm is proposed to enhance the quality of communication, and the performance of DRM. Finally, numerical results are presented to illustrate the impacts of unreliable communication on DRM, as well as the performance of the proposed anti-jamming algorithm.

Distributed Regret Matching Algorithm for Dynamic Congestion Games with Information Provision

The advances in adaptive learning dynamics to pure Nash equilibria in game theory provide promising results for the modeling of selfish agents with limited information in congestion games. In this study, a distributed game-theoretical learning algorithm with real-time information provision for dynamic congestion games is proposed. The learning algorithm is based on the regret matching process by considering a user's previously realised payoffs and real-time information. The numerical studies show that the proposed algorithm can converge to a non-cooperative Nash equilibrium in both static and dynamic congestion networks. Moreover, the proposed algorithm leads to a plausible real-time route choice modeling framework based on a user's perception being updated by incorporating the user's past experience, real-time information and behaviour inertia.

Cooperative Strategies for Energy-Aware Ad Hoc Networks: A Correlated-Equilibrium Game-Theoretical Approach

Energy efficiency is a crucial requirement for energy-aware ad hoc networks. Cooperative communications can be applied to significantly reduce and balance energy consumption. However, this advantage mainly depends on efficient cooperative strategies, i.e., the joint issue of relay node selection and transmit power control. In this paper, we propose a cooperative behavior control scheme, which determines cooperative strategies by using the correlated equilibrium (CE). Specifically, we first model a cooperative behavior control game where the individual utility function is derived from the energy efficiency in terms of the global max-min fairness with the outage performance constraint. Since the CE can achieve better performance by helping the noncooperative players to coordinate their strategies, we employ the CE to analyze the proposed game. Next, we derive the condition under which the CE is Pareto optimal and employ linear programming duality to show its closed-form expression. Furthermore, we design a linear programming method and a distributed learning algorithm based on the regret matching procedure to achieve the CE, respectively. Simulation results demonstrate that our scheme achieves good convergence, Pareto optimality, and max-min fairness.

Multiagent Reinforcement Learning with Regret Matching for Robot Soccer

This paper proposes a novel multiagent reinforcement learning (MARL) algorithm Nash- learning with regret matching, in which regret matching is used to speed up the well-known MARL algorithm Nash- learning. It is critical that choosing a suitable strategy for action selection to harmonize the relation between exploration and exploitation to enhance the ability of online learning for Nash- learning. In Markov Game the joint action of agents adopting regret matching algorithm can converge to a group of points of no-regret that can be viewed as coarse correlated equilibrium which includes Nash equilibrium in essence. It is can be inferred that regret matching can guide exploration of the state-action space so that the rate of convergence of Nash- learning algorithm can be increased. Simulation results on robot soccer validate that compared to original Nash- learning algorithm, the use of regret matching during the learning phase of Nash- learning has excellent ability of online learning and results in significant performance in terms of scores, average reward and policy convergence.

Convergent Learning Algorithms for Unknown Reward Games

In this paper, we address the problem of convergence to Nash equilibria in games with rewards that are initially unknown and must be estimated over time from noisy observations. These games arise in many real-world applications, whenever rewards for actions cannot be prespecified and must be learned online, but standard results in game theory do not consider such settings. For this problem, we derive a multiagent version of $\mathcal{Q}$-learning to estimate the reward functions using novel forms of the $\epsilon$-greedy learning policy. Using these $\mathcal{Q}$-learning schemes to estimate reward functions, we then provide conditions guaranteeing the convergence of adaptive play and the better-reply processes to Nash equilibria in potential games and games with more general forms of acyclicity, and of regret matching to the set of correlated equilibria in generic games. A secondary result is that we prove the strong ergoditicity of stochastic adaptive play and stochastic better-reply processes in the ca...

Energy-efficient resource allocation for uplink orthogonal frequency division multiple access systems using correlated equilibrium

Owing to the evolution of green communications, energy efficiency is treated as an important performance metric of a uplink orthogonal frequency division multiple access (OFDMA) system. In this study, the authors propose an energy-efficient resource allocation scheme by using the correlated equilibrium (CE). At first, the authors construct an energy-efficient resource allocation game, where each subcarrier is viewed as a player to choose the most satisfying user, and the objective is to balance the tradeoff between the total energy efficiency and the fairness. Since the CE can achieve better performance by helping the non-cooperative players coordinate their strategies, the authors employ the CE to analyse the proposed game. Next, the authors derive the condition under which the CE is Pareto optimal and employ linear programming duality to show its closed-form expressions. Furthermore, the authors present a linear programming method and a distributed algorithm based on the regret matching procedure to implement the CE, respectively, which can help us determine the desired resource allocation. Simulation results demonstrate that our scheme is able to achieve good convergence, Pareto optimality and fairness.

Cooperation Policy Selection for Energy-Constrained Ad Hoc Networks Using Correlated Equilibrium

In this letter, we propose a distributed cooperation policy selection scheme for energy-constrained ad hoc networks. In particular, we model a correlated equilibrium (CE)-based cooperation policy selection game where the individual utility function is designed from the global energy efficiency perspective. We derive the condition under which the CE is Pareto optimal, and propose a decentralized algorithm based on the regret matching procedure that converges to the CE. The resulting CE can help us select proper cooperation policies. Simulation results demonstrate that our scheme achieves good convergence, Pareto optimality, and max-min fairness.

Learning equilibria with partial information in decentralized wireless networks

In this article, a survey of several important equilibrium concepts for decentralized networks is presented. The term decentralized is used here to refer to scenarios where decisions (e.g., choosing a power allocation policy) are taken autonomously by devices interacting with each other (e.g., through mutual interference). The iterative long-term interaction is characterized by stable points of the wireless network called equilibria. The interest in these equilibria stems from the relevance of network stability and the fact that they can be achieved by letting radio devices to repeatedly interact over time. To achieve these equilibria, several learning techniques - the best response dynamics, fictitious play, smoothed fictitious play, reinforcement learning algorithms, and regret matching - are discussed in terms of information requirements and convergence properties. Most of the notions introduced here, for both equilibria and learning schemes, are illustrated by a simple case study, an interference channel with two transmitter-receiver pairs.

Regret Matching with Finite Memory

We consider the regret matching process with finite memory. For general games in normal form, it is shown that any recurrent class of the dynamics must be such that the action profiles that appear in it constitute a closed set under the “same or better reply” correspondence (CUSOBR set) that does not contain a smaller product set that is closed under “same or better replies,” i.e., a smaller PCUSOBR set. Two characterizations of the recurrent classes are offered. First, for the class of weakly acyclic games under better replies, each recurrent class is monomorphic and corresponds to each pure Nash equilibrium. Second, for a modified process with random sampling, if the sample size is sufficiently small with respect to the memory bound, the recurrent classes consist of action profiles that are minimal PCUSOBR sets. Our results are used in a robust example that shows that the limiting empirical distribution of play can be arbitrarily far from correlated equilibria for any large but finite choice of the memory bound.

Regret Matching with Finite Memory

We consider the regret matching process with finite memory. For general games in normal form, it is shown that any recurrent class of the dynamics must be such that the action profiles that appear in it constitute a closed set under the “same or better reply” correspondence (CUSOBR set) that does not contain a smaller product set that is closed under “same or better replies,” i.e., a smaller PCUSOBR set. Two characterizations of the recurrent classes are offered. First, for the class of weakly acyclic games under better replies, each recurrent class is monomorphic and corresponds to each pure Nash equilibrium. Second, for a modified process with random sampling, if the sample size is sufficiently small with respect to the memory bound, the recurrent classes consist of action profiles that are minimal PCUSOBR sets. Our results are used in a robust example that shows that the limiting empirical distribution of play can be arbitrarily far from correlated equilibria for any large but finite choice of the memory bound.

A game-theoretic approach to non-cooperative target assignment

In this paper, we have adopted a game-theoretic approach to consider the optimal assignment of targets to vehicles. We have taken optimality to mean the minimum sum of costs which are proportional to the Euclidean distance between a vehicle and its assigned target; this occurs when the set of targets is assigned to their nearest vehicles. In other words, each vehicle should assign itself to a target located within the Voronoi region associated with the vehicle. Standard schemes require communication between vehicles at least to rank the targets according to the nearest-neighbour rule. In our proposed model, there is no need for communication among the vehicles, and only limited communication between the targets and the vehicles is required. Specifically, we have introduced an appropriate utility function which depends on the distance between the vehicles and targets and the number of vehicles engaging a particular target. The vehicles negotiate their choice of targets via regret matching. We present simulations which demonstrate that vehicles select targets that fall within their Voronoi region. Vehicles that do not have any targets within their Voronoi region select their nearest unassigned target. We also present a communication model illustrating how the model can be implemented in a practical scenario.

Parameterized Complexity and Local Search Approaches for the Stable Marriage Problem with Ties

We consider the variant of the classical Stable Marriage problem where preference lists can be incomplete and may contain ties. In such a setting, finding a stable matching of maximum size is NP-hard. We study the parameterized complexity of this problem, where the parameter can be the number of ties, the maximum or the overall length of ties. We also investigate the applicability of a local search algorithm for the problem. Finally, we examine the possibilities for giving an FPT algorithm or an FPT approximation algorithm for finding an egalitarian or a minimum regret matching.

Modeling of path-choice behavior based on the regret minimization criteria givena full travel information

This article formulates the drivers' behaviors of path-choice when travel information for a set of paths is given to each driver after finishing his or her trip.While an underpinning theory of the behavioral model stems from a regret matching theory, the basic framework is adjusted to capture the characteristic of congested networks. The notable feature of the model is that each driver is treatedas an individual decision maker as in game theory. In addition, uncertainty thata driver faces is endogenously generated as the results of mutually dependent path-choice behavior. Drivers are modeled to achieve no-regret criteria and to learn uncertain environment with stochastic approximation.Dynamic of updating the propensity of path-choice is characterized by the ODE approach, and is shownto converge to an internally chain recurrent set. However, this is not the case for the congested network with monotone cost functions. If we admit monotone cost function, then the regret matching approach with stochastic approximation has very nice feature that allows a generic travel cost function such as asymmetric Jacobian, piece-wise linear and so on.

Online calibrated forecasts: Memory efficiency versus universality for learning in games

We provide a simple learning process that enables an agent to forecast a sequence of outcomes. Our forecasting scheme, termed tracking forecast, is based on tracking the past observations while emphasizing recent outcomes. As opposed to other forecasting schemes, we sacrifice universality in favor of a significantly reduced memory requirements. We show that if the sequence of outcomes has certain properties--it has some internal (hidden) state that does not change too rapidly--then the tracking forecast is weakly calibrated so that the forecast appears to be correct most of the time. For binary outcomes, this result holds without any internal state assumptions. We consider learning in a repeated strategic game where each player attempts to compute some forecast of the opponent actions and play a best response to it. We show that if one of the players uses a tracking forecast, while the other player uses a standard learning algorithm (such as exponential regret matching or smooth fictitious play), then the player using the tracking forecast obtains the best response to the actual play of the other players. We further show that if both players use tracking forecast, then under certain conditions on the game matrix, convergence to a Nash equilibrium is possible with positive probability for a larger class of games than the class of games for which smooth fictitious play converges to a Nash equilibrium.