Regret Minimization Research Articles

Randregret: A command for fitting random regret minimization models using Stata

In this article, we describe the randregret command, which implements a variety of random regret minimization (RRM) models. The command allows the user to apply the classic RRM model introduced in Chorus (2010, European Journal of Transport and Infrastructure Research 10: 181–196), the generalized RRM model introduced in Chorus (2014, Transportation Research, Part B 68: 224–238), and also the µRRM and pure RRM models, both introduced in van Cranenburgh, Guevara, and Chorus (2015, Transportation Research, Part A 74: 91–109). We illustrate the use of the randregret command by using stated choice data on route preferences. The command offers robust and cluster standarderror correction using analytical expressions of the score functions. It also offers likelihood-ratio tests that can be used to assess the relevance of a given model specification. Finally, users can obtain the predicted probabilities from each model by using the randregretpred command.

Scalable sub-game solving for imperfect-information games

Counterfactual regret minimization (CFR) is a popular and effective method for solving a game with imperfect information. The effect of CFR is limited by the size of the game state space. With the increase in the number of game participants, the game state space will increase rapidly. Although the vanilla CFR is suitable for two-player imperfect-information games, it does not work well in imperfect-information games with three or more players. In this paper, we design a framework for imperfect-information games, which can not only deal with two-player imperfect-information games but also can efficiently solve three-player imperfect-information games. Compared with traditional solving methods, in this framework we propose real-time hand abstraction (RTHA), which can reduce the error caused by the abstraction. We also propose a warm-start online solution of sub-game (WSOS-SG) method, which can improve the accuracy of the action estimation and solve the sub-game in real time. Experimental results show that the agent based on our method achieve better performances than traditional methods. The agent based on our method took part in the 2018 AAAI-ACPC poker competition and won third place in heads-up no-limit Texas hold’em.

Designing experiments informed by observational studies

AbstractThe increasing availability of passively observed data has yielded a growing interest in “data fusion” methods, which involve merging data from observational and experimental sources to draw causal conclusions. Such methods often require a precarious tradeoff between the unknown bias in the observational dataset and the often-large variance in the experimental dataset. We propose an alternative approach, which avoids this tradeoff: rather than using observational data for inference, we use it to design a more efficient experiment. We consider the case of a stratified experiment with a binary outcome and suppose pilot estimates for the stratum potential outcome variances can be obtained from the observational study. We extend existing results to generate confidence sets for these variances, while accounting for the possibility of unmeasured confounding. Then, we pose the experimental design problem as a regret minimization problem subject to the constraints imposed by our confidence sets. We show that this problem can be converted into a concave maximization and solved using conventional methods. Finally, we demonstrate the practical utility of our methods using data from the Women’s Health Initiative.

Values of Travel Time for Recreational Trips under Different Behavioural Rules

In this study, discrete choice models that combine different behavioural rules are estimated to study the visitors’ preferences in relation to their travel mode choices to access a national park. Using a revealed preference survey conducted on visitors of Teide National Park (Tenerife, Spain), we present a hybrid model specification—with random parameters—in which we assume that some attributes are evaluated by the individuals under conventional random utility maximization (RUM) rules, whereas others are evaluated under random regret minimization (RRM) rules. We then compare the results obtained using exclusively a conventional RUM approach to those obtained using both RUM and RRM approaches, derive monetary valuations of the different components of travel time and calculate direct elasticity measures. Our results provide useful instruments to evaluate policies that promote the use of more sustainable modes of transport in natural sites. Such policies should be considered as priorities in many national parks, where negative transport externalities such as traffic congestion, pollution, noise and accidents are causing problems that jeopardize not only the sustainability of the sites, but also the quality of the visit.

The trade-off between safety and time in the red light running behaviors of pedestrians: A random regret minimization approach

Pedestrian safety has been a major concern in Hong Kong, where walking is an important access mean to urban transportation services and pedestrian-vehicle conflicts are prevalent. Red light running violation of pedestrians is a leading cause of pedestrian-vehicle crashes at the signal intersections. It is necessary to examine the possible factors including personal characteristics and road environments that affect the propensities of red light running violation of pedestrians. Therefore, effective traffic control and enforcement measures can be implemented to deter against the red light running behaviors of pedestrians. This study attempts to examine the roles of trade-off between safety and time, as well as situational features and personality traits, in the red light running behaviors of pedestrians using a stated preference survey method. Then, a regret-based panel mixed multinomial logit model is established for the association measure between propensities of red light running violation and possible factors, with which the effects of unobserved heterogeneity and correlation in the choices between different scenarios of the same person are considered. Results indicate that the choice decision of pedestrians are more sensitive to a reduction in time loss, as compared to the equivalent increase in safety risk. In addition, the safety versus time trade-off may vary between pedestrian groups. Furthermore, presence and type of another violator also significantly affect the propensities of red light running violation. Such findings are indicative to effective policy interventions that can deter against the red light running behaviors of vulnerable pedestrian groups. Therefore, overall pedestrian safety level can be improved in the long term.

An accumulation of preference: Two alternative dynamic models for understanding transport choices

Interest in behavioural realism has gradually led to the introduction of alternatives to random utility models (RUMs) as a paradigm for representing choice behaviour, with notable interest, for example, in random regret minimisation (RRM). These more general models continue to rely on a framework where a single value function is calculated for each alternative in each choice setting, and the choice probabilities are calculated by comparing these value functions across alternatives. By contrast, research in mathematical psychology has used a more dynamic approach, where the preference value of each alternative updates over time in a given situation while the decision maker is deliberating about the choice to make. These accumulator models are well suited to accommodating a variety of context effects, and have been shown to give good performance for data collected in laboratory-based settings. The present paper considers two such accumulator models, namely decision field theory (DFT) and the multi-attribute linear ballistic accumulator (MLBA), and addresses limitations that have prevented their use in travel behaviour research. The methodological additions include the ability to capture the influence of socio-demographics, the presence of underlying preferences for specific alternatives, and/or the representation of attributes that have opposite effects on choice probabilities. We develop what we believe to be the first in-depth simultaneous comparison of DFT and MLBA with typical discrete choice models, and test both DFT and MLBA on a revealed preference dataset. We find that each model outperforms typical RUM and RRM implementations for both in-sample estimation and out-of-sample prediction, including in a large scale simulation experiment.

Model-Free Online Learning in Unknown Sequential Decision Making Problems and Games

Regret minimization has proved to be a versatile tool for tree-form sequential decision making and extensive-form games. In large two-player zero-sum imperfect-information games, modern extensions of counterfactual regret minimization (CFR) are currently the practical state of the art for computing a Nash equilibrium. Most regret-minimization algorithms for tree-form sequential decision making, including CFR, require (i) an exact model of the player’s decision nodes, observation nodes, and how they are linked, and (ii) full knowledge, at all times t, about the payoffs—even in parts of the decision space that are not encountered at time t. Recently, there has been growing interest towards relaxing some of those restrictions and making regret minimization applicable to settings for which reinforcement learning methods have traditionally been used—for example, those in which only black-box access to the environment is available. We give the first, to our knowledge, regret-minimization algorithm that guarantees sublinear regret with high probability even when requirement (i)—and thus also (ii)—is dropped. We formalize an online learning setting in which the strategy space is not known to the agent and gets revealed incrementally whenever the agent encounters new decision points. We give an efficient algorithm that achieves O(T^3/4) regret with high probability for that setting, even when the agent faces an adversarial environment. Our experiments show it significantly outperforms the prior algorithms for the problem, which do not have such guarantees. It can be used in any application for which regret minimization is useful: approximating Nash equilibrium or quantal response equilibrium, approximating coarse correlated equilibrium in multi-player games, learning a best response, learning safe opponent exploitation, and online play against an unknown opponent/environment.

Faster Game Solving via Predictive Blackwell Approachability: Connecting Regret Matching and Mirror Descent

Blackwell approachability is a framework for reasoning about repeated games with vector-valued payoffs. We introduce predictive Blackwell approachability, where an estimate of the next payoff vector is given, and the decision maker tries to achieve better performance based on the accuracy of that estimator. In order to derive algorithms that achieve predictive Blackwell approachability, we start by showing a powerful connection between four well-known algorithms. Follow-the-regularized-leader (FTRL) and online mirror descent (OMD) are the most prevalent regret minimizers in online convex optimization. In spite of this prevalence, the regret matching (RM) and regret matching+ (RM+) algorithms have been preferred in the practice of solving large-scale games (as the local regret minimizers within the counterfactual regret minimization framework). We show that RM and RM+ are the algorithms that result from running FTRL and OMD, respectively, to select the halfspace to force at all times in the underlying Blackwell approachability game. By applying the predictive variants of FTRL or OMD to this connection, we obtain predictive Blackwell approachability algorithms, as well as predictive variants of RM and RM+. In experiments across 18 common zero-sum extensive-form benchmark games, we show that predictive RM+ coupled with counterfactual regret minimization converges vastly faster than the fastest prior algorithms (CFR+, DCFR, LCFR) across all games but two of the poker games, sometimes by two or more orders of magnitude.

A Heuristic Evaluation Function for Hand Strength Estimation in Gin Rummy

This paper describes a fast hand strength estimation mod-el for the game of Gin Rummy. The algorithm is computationally inexpensive, and it incorporates not only cards in the player’s hand but also cards known to be in the opponent’s hand, cards in the discard pile, and the current game stage. This algorithm is used in conjunction with counterfactual regret (CFR) minimization to develop a gin rummy bot. CFR strategies were developed for the knocking strategies. The hand strength estimation algorithm was used to select a discard that balances the goals of maximizing the utility of the player’s hand and minimizing the likelihood that a card will be useful to the opponent. A study of the parameterization of this estimation algorithm demonstrates the soundness of approach as well as good performance under a wide range of parameter values.

Policy Optimization as Online Learning with Mediator Feedback

Policy Optimization (PO) is a widely used approach to address continuous control tasks. In this paper, we introduce the notion of mediator feedback that frames PO as an online learning problem over the policy space. The additional available information, compared to the standard bandit feedback, allows reusing samples generated by one policy to estimate the performance of other policies. Based on this observation, we propose an algorithm, RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST), for regret minimization in PO, that employs a randomized exploration strategy, differently from the existing optimistic approaches. When the policy space is finite, we show that under certain circumstances, it is possible to achieve constant regret, while always enjoying logarithmic regret. We also derive problem-dependent regret lower bounds. Then, we extend RANDOMIST to compact policy spaces. Finally, we provide numerical simulations on finite and compact policy spaces, in comparison with PO and bandit baselines.

Hindsight and Sequential Rationality of Correlated Play

Driven by recent successes in two-player, zero-sum game solving and playing, artificial intelligence work on games has increasingly focused on algorithms that produce equilibrium-based strategies. However, this approach has been less effective at producing competent players in general-sum games or those with more than two players than in two-player, zero-sum games. An appealing alternative is to consider adaptive algorithms that ensure strong performance in hindsight relative to what could have been achieved with modified behavior. This approach also leads to a game-theoretic analysis, but in the correlated play that arises from joint learning dynamics rather than factored agent behavior at equilibrium. We develop and advocate for this hindsight rationality framing of learning in general sequential decision-making settings. To this end, we re-examine mediated equilibrium and deviation types in extensive-form games, thereby gaining a more complete understanding and resolving past misconceptions. We present a set of examples illustrating the distinct strengths and weaknesses of each type of equilibrium in the literature, and prove that no tractable concept subsumes all others. This line of inquiry culminates in the definition of the deviation and equilibrium classes that correspond to algorithms in the counterfactual regret minimization (CFR) family, relating them to all others in the literature. Examining CFR in greater detail further leads to a new recursive definition of rationality in correlated play that extends sequential rationality in a way that naturally applies to hindsight evaluation.

Complexity and Algorithms for Exploiting Quantal Opponents in Large Two-Player Games

Solution concepts of traditional game theory assume entirely rational players; therefore, their ability to exploit subrational opponents is limited. One type of subrationality that describes human behavior well is the quantal response. While there exist algorithms for computing solutions against quantal opponents, they either do not scale or may provide strategies that are even worse than the entirely-rational Nash strategies. This paper aims to analyze and propose scalable algorithms for computing effective and robust strategies against a quantal opponent in normal-form and extensive-form games. Our contributions are: (1) we define two different solution concepts related to exploiting quantal opponents and analyze their properties; (2) we prove that computing these solutions is computationally hard; (3) therefore, we evaluate several heuristic approximations based on scalable counterfactual regret minimization (CFR); and (4) we identify a CFR variant that exploits the bounded opponents better than the previously used variants while being less exploitable by the worst-case perfectly-rational opponent.

Evolution Strategies for Approximate Solution of Bayesian Games

We address the problem of solving complex Bayesian games, characterized by high-dimensional type and action spaces, many (> 2) players, and general-sum payoffs. Our approach applies to symmetric one-shot Bayesian games, with no given analytic structure. We represent agent strategies in parametric form as neural networks, and apply natural evolution strategies (NES) [wierstra2014natural] for deep model optimization. For pure equilibrium computation, we formulate the problem as bi-level optimization, and employ NES in an iterative algorithm to implement both inner-loop best response optimization and outer-loop regret minimization. In simple games including first- and second-price auctions, it is capable of recovering known analytic solutions. For mixed equilibrium computation, we adopt an incremental strategy generation framework, with NES as strategy generator producing a finite sequence of approximate best-response strategies. We then calculate equilibria over this finite strategy set via a model-based optimization process. Both our pure and mixed equilibrium computation methods employ NES to efficiently search for strategies over the functional space, given only black-box simulation access to noisy payoff samples. We experimentally demonstrate the efficacy of all methods on two simultaneous sealed-bid auction games with distinct type distributions, and observe that the solutions exhibit qualitatively different behavior in these two environments.

Extracting Learned Discard and Knocking Strategies from a Gin Rummy Bot

Various Gin Rummy strategy guides provide heuristics for human players to improve their gameplay. Often these heuristics are either conflicting or contain ambiguity that limits their applicability, especially for discard and end-of-game decisions. This paper describes an approach to analyzing the machine learning capabilities of a Gin Rummy agent to help resolve these conflicts and ambiguities. There are three main decision points in the game: when to draw from the discard pile, which card to discard from the player's hand, and when to knock. The agent us-es a learning approach to estimate the expected utility for discards. An analysis of these utility values provides in-sight into resolving ambiguities in tips for discard decisions in human play. The agent’s end-of-game, or knocking, strategy was derived using Monte Carlo Counterfactual regret minimization (MCCFR). This approach was applied to estimate Nash equilibrium knocking strategies under different rules of the game. The analysis suggests that conflicts in the end-of-game playing tips are due in part to different rules used in common Gin Rummy variants.

Estimating Card Fitness for Discard in Gin Rummy

Due to the computation time and resources required, there is no known optimal strategy for the game of Gin Rummy. Previous work in extensive games, such as Texas Hold ’em Poker, has found that hand fitness and information sets about the state of the game can be used to determine an improved strategy. These information sets, combined with algorithms for Counterfactual Regret Minimization, can arrive at a Nash Equilibrium strategy for smaller abstractions of extensive games. This paper builds on previous research by extending the premise of hand fitness to card fitness in the discard decision point of Gin Rummy. We argue that a card can be ranked based on whether it meets four specific characteristics at that stage in the game. These characteristics include its effect on deadwood points after one more turn, its utility to the opponent, and if it can contribute to a meld. An optimal discard choice can then be picked from the highest-ranked card by using a simplified Counterfactual regret minimization strategy that can be trained in less time due to its limited information set. While this does not look at every potential characteristic of card fitness, it outperformed other bots when evaluated in a large number of games. These bots did not consider card fitness as a whole, but rather considered characteristics separately. We argue that the characteristics defined are a part of the total information set that can determine the discard fitness of a card within a hand in the game of Gin Rummy.

Exploring the impact of COVID-19 on individual's travel mode choice in China.

Travel activities and travel behaviors have been greatly affected by the outbreak of Covid-19. Facing the change of individuals' travel choices, policymakers have to make an appropriate response to mitigate negative consequences. This paper aims to explore how the COVID-19 would impact travel mode choice and the intention of car purchase. The data was collected from a large-scale survey conducted in June 2020 after the highest point. Random utility maximization (RUM), random regret minimization (RRM) and generalized regret minimization (GRRM) are employed to examine the effects of various factors on mode choice behaviors. The estimation results reveal that regret aversion psychology doesn't have a dominant proportion of decision choices, even if the congested condition of the mass mobility plays a significant role in the consideration of decision-making. Combined with the statistical results from the official departments, we concluded that public transport displays a great propensity on the long trip, and meanwhile, the industry of ride-hailing services has shocked sharply. In terms of the intention of traffic tool purchase, carless people prefer to buy electric two-wheel vehicles rather than automobiles. The research findings and the contribution to policy implications give assistance to authority in understanding citizens' travel mode preferences under the impact of COVID-19.

The application of imperfect information game theory in social games

The research on game AI has always been one of the focuses of artificial intelligence research. More complex real-world problems can be solved by the study of game AI. Since our cognition and information acquisition of the world are imperfect, we proposed single deep counterfactual regret minimization method with advantage baselines, a deep reinforcement learning algorithm based on imperfect information game. The new approach combines the single deep counterfactual regret minimization approach with the baseline network to achieve a better performance than the benchmark in poker. At last, this paper applies the new algorithm to two Chinese poker games with imperfect information game, and obtains better performance. Meanwhile, it also provides the possibility for the extension to real life.

The Value Function with Regret Minimization Algorithm for Solving the Nash Equilibrium of Multi-Agent Stochastic Game

The Value Function with Regret Minimization Algorithm for Solving the Nash Equilibrium of Multi-Agent Stochastic Game

A competition-based pricing strategy in Cloud markets using regret minimisation techniques

A competition-based pricing strategy in Cloud markets using regret minimisation techniques

A competition-based pricing strategy in Cloud markets using regret minimisation techniques

A competition-based pricing strategy in Cloud markets using regret minimisation techniques