Regret Matching Research Articles

Learning Not to Regret

The literature on game-theoretic equilibrium finding predominantly focuses on single games or their repeated play. Nevertheless, numerous real-world scenarios feature playing a game sampled from a distribution of similar, but not identical games, such as playing poker with different public cards or trading correlated assets on the stock market. As these similar games feature similar equilibra, we investigate a way to accelerate equilibrium finding on such a distribution. We present a novel ``learning not to regret'' framework, enabling us to meta-learn a regret minimizer tailored to a specific distribution. Our key contribution, Neural Predictive Regret Matching, is uniquely meta-learned to converge rapidly for the chosen distribution of games, while having regret minimization guarantees on any game. We validated our algorithms' faster convergence on a distribution of river poker games. Our experiments show that the meta-learned algorithms outpace their non-meta-learned counterparts, achieving more than tenfold improvements.

A channel selection game for multi-operator LoRaWAN deployments

With over 75 billion Internet of Things (IoT) devices expected worldwide by the year 2025, inaugural MAC layer solutions for long-range IoT deployments no longer suffice. LoRaWAN, the principal technology for comprehensive IoT deployments, enables low power and long range communications. However, synchronous transmissions on the same spreading factors and within the same frequency channel, augmented by the necessary clustering of IoT devices, will cause collisions and drops in efficiency. The performance is further impacted by the shortage of radio resources and by multiple operators utilizing the same unlicensed frequency bands. In this paper, we propose a game theoretic based channel selection algorithm for LoRaWAN in a multi-operator deployment scenario . We begin by proposing an optimal formulation for the selection process with the objective of maximizing the total normalized throughput per spreading factor, per channel. Then, we propose centralized optimal approaches, as well as distributed algorithms , based on reinforcement learning and regret matching dynamics, to finding both the Nash and Correlated equilibria of the proposed game. We simulate our proposals and compare them to the legacy approach of randomized channel access, stressing their efficiency in improving the total normalized throughput as well as the packet delivery ratios .

Distributed Channel and Power Level Selection in VANET Based on SINR using Game Model

This paper proposes a scheme of channel selection and transmission power adjustment in Vehicular Ad hoc Network (VANET) using game theoretic approach. The paradigm of VANET enables groups of vehicles to establish a mesh-like communication network. However, the mobility of vehicle, highly dynamic network environment, and the shared-spectrum concept used in VANET pose some challenges such as interference that can decrease the quality of signal. Channel selection and transmit power adjustment are aimed to obtain the higher signal to interference and noise ratio (SINR). In this paper, game theory is implemented to model the channel and power level selection in VANET. Each vehicle represents the player and the combination of channel and power level represents the strategy used by the player to obtain the utility i.e. the SINR. Strategy selection is arranged distributively to each player using Regret Matching Learning (RML) algorithm. Each vehicle evaluates current utility obtained by selecting a strategy to define the probability of that strategy to be selected in the next time. However, RML has a shortcoming for using assumption that hard to be implemented in real VANET environment. Therefore modification of RML devised for this application is also proposed. The simulation model of channel and power level selection is build to evaluate the performance of the proposed scheme. The results of simulation display the improvement of VANET performance in term of SINR and throughput from the proposed scheme.

A New Cooperative Game—Theoretic Approach for Customer-Owned Energy Storage

The increasing demand for energy storage systems (ESSs) alongside the continuous enhancements to storage technology have been of great positive impact on the electric grid. Their unceasing development has been driven by the need to accommodate increased penetration of renewable energy resources and defer capital investments, among other benefits. Moreover, ESSs have played a key role in the grid’s ability to cope with its ever-shifting load profiles, resulting in large economic gain for ESS owners. For this reason, this prospective study was designed to investigate privately-owned energy storage hubs (ESHs) and their interactions with potential customers as well as with the electric grid. This research examined two contrasting interaction approaches for customer-owned stationary energy storage hubs: a cooperative and a non-cooperative game-theoretic approach. The goal of the cooperative technique is to conduce to a correlated equilibrium increasing the social welfare of all players involved using a regret matching algorithm. On the other hand, in the non-cooperative approach, modeled as an ascending price-clinching auction, each player acts greedily, maximizing only their individual welfare. Implementing both case studies resulted in important insights into ESH players’ interactions and provided contrasting methods of modeling their behaviors. Finally, depending on the application at hand, the choice of one approach may be more realistic than the other.

Finding nash equilibrium for imperfect information games via fictitious play based on local regret minimization

Finding Nash equilibrium in the domain of imperfect information games as a challenging problem has received much attention. Neural Fictitious Self-Play (NFSP) is a popular model-free machine learning algorithm and has computed approximate Nash equilibrium on such games. However, the deep reinforcement learning method used to approximate the best response in NFSP requires reaching a fully observable Markov state, while the states in imperfect information games are partially observable and non-Markovian, which results in a poor approximation of the best response. Thus, NFSP needs more iterations to converge. In this study, we present a new reinforcement learning method that is inspired by counterfactual regret minimization to relax the Markov requirement by iteratively updating policy according to the regret matching process. Combining this new reinforcement learning algorithm with fictitious play, we further present a novel algorithm to find approximate Nash equilibrium in zero-sum imperfect information games. Experimental results in three benchmark games show that this new algorithm can find approximate Nash equilibrium effectively and converge much faster compared with baseline.

Faster Game Solving via Predictive Blackwell Approachability: Connecting Regret Matching and Mirror Descent

Blackwell approachability is a framework for reasoning about repeated games with vector-valued payoffs. We introduce predictive Blackwell approachability, where an estimate of the next payoff vector is given, and the decision maker tries to achieve better performance based on the accuracy of that estimator. In order to derive algorithms that achieve predictive Blackwell approachability, we start by showing a powerful connection between four well-known algorithms. Follow-the-regularized-leader (FTRL) and online mirror descent (OMD) are the most prevalent regret minimizers in online convex optimization. In spite of this prevalence, the regret matching (RM) and regret matching+ (RM+) algorithms have been preferred in the practice of solving large-scale games (as the local regret minimizers within the counterfactual regret minimization framework). We show that RM and RM+ are the algorithms that result from running FTRL and OMD, respectively, to select the halfspace to force at all times in the underlying Blackwell approachability game. By applying the predictive variants of FTRL or OMD to this connection, we obtain predictive Blackwell approachability algorithms, as well as predictive variants of RM and RM+. In experiments across 18 common zero-sum extensive-form benchmark games, we show that predictive RM+ coupled with counterfactual regret minimization converges vastly faster than the fastest prior algorithms (CFR+, DCFR, LCFR) across all games but two of the poker games, sometimes by two or more orders of magnitude.

On using reinforcement learning for network slice admission control in 5G: Offline vs. online

SummaryAchieving a fair usage of network resources is of vital importance in Slice‐ready 5G network. The dilemma of which network slice to accept or to reject is very challenging for the Infrastructure Provider (InfProv). On one hand, InfProv aims to maximize the network resources usage by accepting as many network slices as possible; on the other hand, the network resources are limited, and the network slice requirements regarding Quality of Service (QoS) need to be fulfilled. In this paper, we devise three admission control mechanisms based on Reinforcement Learning, namely, Q‐Learning, Deep Q‐Learning, and Regret Matching, which allow deriving admission control decisions (policy) to be applied by InfProv to admit or reject network slice requests. We evaluated the three algorithms using computer simulation, showing results on each mechanism's performance in terms of maximizing the InfProv revenue and their ability to learn offline or online.

A regret-based behavioral model for shared water resources management: Application of the correlated equilibrium concept

The competition over water use in shared water resources systems may lead to conflict. Conflict can lead to strategic behaviors with the consequence of “Tragedy of Common” in water resources. In this paper, a novel approach is proposed for the quantity and quality management of shared water resources using the Correlated Equilibrium (CE) concept. For the first time in water resources management studies, a Reinforcement Learning (RL)-based method, namely Regret Matching (RM), is proposed to simulate agents' behaviors. In the proposed methodology, an agent, which is responsible for water allocations, tries to reduce illegal water withdrawal from resources, using some non-mandatory and mandatory suggestions. This agent's objectives are leading the system towards social optimality (SO) and reaching the environmental sustainability goal. A modified RM algorithm is also developed for behavioral simulation in urban areas. The proposed methodology's applicability and efficiency are evaluated considering some criteria such as the concentration of the nitrate pollutant in groundwater, the groundwater table fluctuations, the rate of illegal water extraction from the groundwater, and the stakeholders' general satisfaction. The results of applying the methodology to the western part of the Tehran metropolitan area show its ability to deal with the water and treated wastewater allocation problems in urban areas and increase in the learning and cooperation among agents. According to the results, a meaningful decrease in nitrate concentration in the aquifer and an increase in groundwater table levels are observed. The results also indicate that the model could teach the stakeholders to act more responsibly towards protecting the environment and conserving shared water resources.

Geometrical regret matching: A new dynamics to Nash equilibrium

We argue that the existing regret matchings for Nash equilibrium approximation conduct “jumpy” strategy updating when the probabilities of future plays are set to be proportional to positive regret measures. We propose a geometrical regret matching that features “smooth” strategy updating. Our approach is simple, intuitive, and natural. The analytical and numerical results show that “smoothly” suppressing “unprofitable” pure strategies is sufficient for the game to evolve toward Nash equilibrium, suggesting that, in reality, the tendency for equilibrium could be pervasive and irresistible. Technically, iterative regret matching gives rise to a sequence of adjusted mixed strategies for us to examine its approximation to the true equilibrium point. The sequence can be studied in the metric space and visualized nicely as a clear path toward an equilibrium point. Our theory has limitations in optimizing the approximation accuracy.

Solving Imperfect-Information Games via Discounted Regret Minimization

Counterfactual regret minimization (CFR) is a family of iterative algorithms that are the most popular and, in practice, fastest approach to approximately solving large imperfectinformation games. In this paper we introduce novel CFR variants that 1) discount regrets from earlier iterations in various ways (in some cases differently for positive and negative regrets), 2) reweight iterations in various ways to obtain the output strategies, 3) use a non-standard regret minimizer and/or 4) leverage “optimistic regret matching”. They lead to dramatically improved performance in many settings. For one, we introduce a variant that outperforms CFR+, the prior state-of-the-art algorithm, in every game tested, including large-scale realistic settings. CFR+ is a formidable benchmark: no other algorithm has been able to outperform it. Finally, we show that, unlike CFR+, many of the important new variants are compatible with modern imperfect-informationgame pruning techniques and one is also compatible with sampling in the game tree.

Agent-Based Decentralized Optimal Charging Strategy for Plug-in Electric Vehicles

This paper presents a game-theoretic decentralized electric vehicle charging schedule for minimizing the customers’ payments, maximizing the grid efficiency, and providing the maximum potential capacity for ancillary services. Most of the available methods for electric vehicle charging assume that the customers are rational, there is a low-latency perfect two-way communication infrastructure without communication/computation limitation between the distribution company and all the customers, and they have perfect knowledge about the system parameters. To avoid these strong assumptions and preserve the customers’ privacy, we take advantage of the regret matching and the Nash Folk theorems. In the considered game, the players (customers) interact and communicate locally with only their neighbors. We propose a mechanism for this game, which results in a full Nash Folk theorem. We demonstrate and prove that the on-off charging strategy provides the maximum regulation capacity. However, our mechanism is quite general, takes into account the battery characteristics and degradation costs of the vehicles, provides a real-time dynamic pricing model, and supports the vehicle-to-grid and modulated charging protocols. Moreover, the developed mechanism is robust to the data disruptions and takes into account the long/short-term uncertainties.

Target Selection for Tracking in Multifunction Radar Networks: Nash and Correlated Equilibria

We consider a target selection problem for multitarget tracking in a multifunction radar network from a game-theoretic perspective. The problem is formulated as a noncooperative game. The radars are considered to be players in this game with utilities modeled using a proper tracking accuracy criterion and their strategies are the observed targets whose number is known. Initially, for the problem of coordination, the Nash equilibria are characterized and, in order to find equilibria points, a distributed algorithm based on the best response dynamics is proposed. Afterward, the analysis is extended to the case of partial target observability and radar connectivity and heterogeneous interests among radars. The solution concept of correlated equilibria is employed and a distributed algorithm based on the regret matching is proposed. The proposed algorithms are shown to perform well compared to the centralized approach of significantly higher complexity.

Sampled fictitious play is Hannan consistent

Fictitious play is a simple and widely studied adaptive heuristic for playing repeated games. It is well known that fictitious play fails to be Hannan consistent. Several variants of fictitious play including regret matching, generalized regret matching and smooth fictitious play, are known to be Hannan consistent. In this note, we consider sampled fictitious play: at each round, the player samples past times and plays the best response to previous moves of other players at the sampled time points. We show that sampled fictitious play, using Bernoulli sampling, is Hannan consistent. Unlike several existing Hannan consistency proofs that rely on concentration of measure results, ours instead uses anti-concentration results from Littlewood–Offord theory.

Wireless Networks under a Backoff Attack: A Game Theoretical Perspective.

We study a wireless sensor network using CSMA/CA in the MAC layer under a backoff attack: some of the sensors of the network are malicious and deviate from the defined contention mechanism. We use Bianchi’s network model to study the impact of the malicious sensors on the total network throughput, showing that it causes the throughput to be unfairly distributed among sensors. We model this conflict using game theory tools, where each sensor is a player. We obtain analytical solutions and propose an algorithm, based on Regret Matching, to learn the equilibrium of the game with an arbitrary number of players. Our approach is validated via simulations, showing that our theoretical predictions adjust to reality.

Region-based prescriptive route guidance for travelers of multiple classes

The performance and complicated interactions of different classes of travelers on regional urban networks are presented and analyzed. A new multi-class extension of a regional dynamic traffic model, the Network Transmission Model is proposed. The classes in question correspond to travelers using autonomous vehicles, conventional vehicles, equipped with Route Guidance and Information Systems, and unequipped vehicles. Each class is represented by a different routing method. Incremental Route Planning, an innovative predictive simulation-based routing method, Proxy Regret Matching, a non-predictive strategic learning-based method and Multinomial Logit-based Routing for 1st, 2nd and 3rd class respectively. All routing methods include a Public Transit Diversion mechanism and are assumed to provide prescriptive route guidance, with pre-trip information dissemination for every departing vehicle. We consider the possibility of non-compliance for conventional vehicles equipped with Route Guidance and Information Systems. We also consider 2 possible scenarios for autonomous vehicles that affect their travel time prediction accuracy. We simulate regional traffic dynamics for simultaneous application of all aforementioned routing methods, employing a market penetration scheme for each class of travelers. We analyze results regarding the overall network performance for various combinations of traveler class market penetration rates and non-compliance rates. We come to the conclusion that autonomous vehicles will not only provide benefits for 1st class travelers, but for all traveler classes on the network.

Power Demand-supply Adjustment in Multi-area Power Grid Based on Negawatt Trading via Regret Matching and VCG Mechanism Design

Power Demand-supply Adjustment in Multi-area Power Grid Based on Negawatt Trading via Regret Matching and VCG Mechanism Design

Direct incentive-compatible mechanism for innovator-investor bargain in an innovation investment system

The functioning of the system of innovation investment, where an objective evaluation of an investment project containing new knowledge is impossible both for the investor and the innovator, currently is a matter of great interest. While for traditional industries, the use of statistical data is a satisfactory approach to the problem of uncertainty, for innovation projects this method isn’t applicable due to the natural absence of a valid statistical base. The practice shows that the empirical mechanisms used for innovation projects are rarely used for investing in traditional industries and vice versa. In this connection, there arises the problem of development an effective mechanism for financing innovation projects. We developed such a mechanism for the innovator-investor system in the form of a Bayesian non-cooperative repeating game with recalculated payments. The equilibrium parameters for any period of the given game are obtained. It is shown that, depending on the a priori estimates of the type of project, four different equilibria are possible in any particular period of that game. Therefore, using the strategy of adjusting a priori estimates of the investor and innovator on the basis of the Regret Matching rule, equilibrium for this finite Bayesian game is obtained.

Distributed Channel and Power Level Selection in VANET Based on SINR using Game Model

This paper proposes a scheme of channel selection and transmission power adjustment in Vehicular Ad hoc Network (VANET) using game theoretic approach. The paradigm of VANET enables groups of vehicles to establish a mesh-like communication network. However, the mobility of vehicle, highly dynamic network environment, and the shared-spectrum concept used in VANET pose some challenges such as interference that can decrease the quality of signal. Channel selection and transmit power adjustment are aimed to obtain the higher signal to interference and noise ratio (SINR). In this paper, game theory is implemented to model the channel and power level selection in VANET. Each vehicle represents the player and the combination of channel and power level represents the strategy used by the player to obtain the utility i.e. the SINR. Strategy selection is arranged distributively to each player using Regret Matching Learning (RML) algorithm. Each vehicle evaluates current utility obtained by selecting a strategy to define the probability of that strategy to be selected in the next time. However, RML has a shortcoming for using assumption that hard to be implemented in real VANET environment. Therefore modification of RML devised for this application is also proposed. The simulation model of channel and power level selection is build to evaluate the performance of the proposed scheme. The results of simulation display the improvement of VANET performance in term of SINR and throughput from the proposed scheme.

Dynamic Thresholding and Pruning for Regret Minimization

Regret minimization is widely used in determining strategies for imperfect-information games and in online learning. In large games, computing the regrets associated with a single iteration can be slow. For this reason, pruning — in which parts of the decision tree are not traversed in every iteration — has emerged as an essential method for speeding up iterations in large games. The ability to prune is a primary reason why the Counterfactual Regret Minimization (CFR) algorithm using regret matching has emerged as the most popular iterative algorithm for imperfect-information games, despite its relatively poor convergence bound. In this paper, we introduce dynamic thresholding, in which a threshold is set at every iteration such that any action in the decision tree with probability below the threshold is set to zero probability. This enables pruning for the first time in a wide range of algorithms. We prove that dynamic thresholding can be applied to Hedge while increasing its convergence bound by only a constant factor in terms of number of iterations. Experiments demonstrate a substantial improvement in performance for Hedge as well as the excessive gap technique.

Two “little treasure games” driven by unconditional regret

For Traveler’s Dilemma and Minimal Effort Coordination games, the unconditional regret matching (URM) procedure predicts outcomes close to the experimental ones. This supports a claim that the URM procedure can be well suited to predict the behavior of experimental subjects in repeated games.