Imperfect Information Game Research Articles

Automatically designing counterfactual regret minimization algorithms for solving imperfect-information games

Strategic decision-making in imperfect-information games is an important problem in artificial intelligence. Counterfactual regret minimization (CFR), a family of iterative algorithms, has been the workhorse for solving these types of games since its inception. In recent years, a series of novel CFR variants have been proposed, significantly improving the convergence rate of vanilla CFR. However, most of these new variants are hand-designed by researchers through trial and error, often based on different motivations, which generally requires a tremendous amount of effort and insight. This work proposes AutoCFR, a systematic framework that meta-learns novel CFR algorithms through evolution, easing the burden of manual algorithm design. We first design a search language that is rich enough to represent various CFR variants. We then exploit a scalable regularized evolution algorithm with a set of acceleration techniques to efficiently search over the combinatorial space of algorithms defined by this language. The learned novel CFR algorithm can generalize to new imperfect-information games not seen during training and performs on par with or better than existing state-of-the-art CFR variants. In addition to superior empirical performance, we also theoretically show that the learned algorithm converges to an approximate Nash equilibrium. Extensive experiments across diverse imperfect-information games highlight the scalability, extensibility, and generalizability of AutoCFR, establishing it as a general-purpose framework for solving imperfect-information games.

Different strokes in randomised strategies: Revisiting Kuhn's theorem under finite-memory assumptions

Two-player (antagonistic) games on (possibly stochastic) graphs are a prevalent model in theoretical computer science, notably as a framework for reactive synthesis.Optimal strategies may require randomisation when dealing with inherently probabilistic goals, balancing multiple objectives, or in contexts of partial information. There is no unique way to define randomised strategies. For instance, one can use so-called mixed strategies or behavioural ones. In the most general setting, these two classes do not share the same expressiveness. A seminal result in game theory — Kuhn's theorem — asserts their equivalence in games of perfect recall.This result crucially relies on the possibility for strategies to use infinite memory, i.e., unlimited knowledge of all past observations. However, computer systems are finite in practice. Hence it is pertinent to restrict our attention to finite-memory strategies, defined as automata with outputs. Randomisation can be implemented in these in different ways: the initialisation, outputs or transitions can be randomised or deterministic respectively. Depending on which aspects are randomised, the expressiveness of the corresponding class of finite-memory strategies differs.In this work, we study two-player concurrent stochastic games and provide a complete taxonomy of the classes of finite-memory strategies obtained by varying which of the three aforementioned components are randomised. Our taxonomy holds in games of perfect and imperfect information with perfect recall, and in games with more than two players. We also provide an adapted taxonomy for games with imperfect recall.

OpenHoldem: A Benchmark for Large-Scale Imperfect-Information Game Research.

Owing to the unremitting efforts from a few institutes, researchers have recently made significant progress in designing superhuman artificial intelligence (AI) in no-limit Texas hold'em (NLTH), the primary testbed for large-scale imperfect-information game research. However, it remains challenging for new researchers to study this problem since there are no standard benchmarks for comparing with existing methods, which hinders further developments in this research area. This work presents OpenHoldem, an integrated benchmark for large-scale imperfect-information game research using NLTH. OpenHoldem makes three main contributions to this research direction: 1) a standardized evaluation protocol for thoroughly evaluating different NLTH AIs; 2) four publicly available strong baselines for NLTH AI; and 3) an online testing platform with easy-to-use APIs for public NLTH AI evaluation. We will publicly release OpenHoldem and hope it facilitates further studies on the unsolved theoretical and computational issues in this area and cultivates crucial research problems like opponent modeling and human-computer interactive learning.

GameTable Working Group 1 meeting report on search, planning, learning, and explainability

The inaugural in-person meeting for the “GameTable” COST Action’s Working Group 1 (WG1) on Search, Planning, Learning, and Explainability took place at the Leiden Institute of Advanced Computer Science (LIACS) on January 31st, 2024. The primary aims of this meeting were to facilitate talks and discussions on, and connect researchers interested in, three core research goals: (1) human-like game-playing AI, (2) imperfect-information games within a general game playing context, and (3) explainable search and reinforcement learning in games. This report provides a summary of the discussions and talks that took place during the meeting.

Proposal and generation of endgame puzzles for an imperfect information game Geister

Geister is a two-player imperfect information game played with two kinds of pieces, blue and red, where each player cannot observe the colors of the opponent’s pieces on the board. In this paper, we propose Geister endgame puzzles for players to enjoy the game in another form or to practice figuring out moves that are proven to win, similar to chess mating problems. In Geister endgame puzzles, the goal is to find the shortest winning moves under the assumption of the worst cases. We propose not-revealed and partly-revealed puzzles according to how the opponent’s piece colors are revealed to the player. We also propose two kinds of special puzzles, capture-win and red-wall, that utilize specific victory conditions of Geister. We generate Geister endgame puzzles by randomly placing pieces on the board and then using a solver to check whether winning moves exist. The generation success rates for not-revealed and partly-revealed puzzles are approximately 20% to 30%. The experiments also show that puzzles with more moves to win are less frequently generated. In addition, we conduct preliminary experiments that invite beginners to evaluate generated puzzles, which shows that longer-win and special puzzles tend to be more difficult and interesting, respectively.

Evolutionary reinforcement learning with action sequence search for imperfect information games

Deep Reinforcement Learning (DRL) has achieved remarkable success in perfect information games. However, when applied to imperfect information games like Contract Bridge, DRL faces challenges not only from unobservable partial information but also from the lack of efficient exploration. Although several Evolutionary Reinforcement Learning algorithms (ERLs) have harnessed Evolutionary Algorithms (EAs) to enhance exploration capabilities, the practical performance of EAs is limited either by the high-dimensional parameter space or possibly inaccurate guidance from value networks. In this paper, we introduce a novel ERL algorithm that employs the Particle Swarm Optimization (PSO) algorithm to search for superior action sequences, thereby aiding agents in exploring uncharted territory. We conduct the search in action space and evaluate action sequences through interactions with the environment to avoid the limitations of parameter space and value networks, respectively. The diverse experiences collected in the search and evaluation can boost the learning of the DRL agent. In addition, the action sequence search is executed only when the agent converges to local optima, which can reduce the overall cost of action evaluation and avoid influencing the optimization process of DRL. Through experimental comparisons conducted on Contract Bridge, our method demonstrates superior performance when compared with several state-of-the-art DRL and ERL algorithms. Furthermore, we utilize our method in the Bridge Competition of AAMAS 2023 Imperfect Information Card Games Competition and rank the first.

MDou: Accelerating DouDiZhu Self-Play Learning Using Monte-Carlo Method With Minimum Split Pruning and a Single Q-Network

Artificial Intelligence (AI) has demonstrated outstanding performance in some perfect- and imperfect-information games, such as Go, Atari, and Texas Hold'em. Even though AI is successful in these games with small action spaces, it does not play well in large-scale multi-player, imperfect-information games like DouDiZhu. DouZero, a DouDizhu AI system, has recently been proposed and beaten all the existing DouDizhu AI programs. This paper introduces Minimum Split Pruning (MSP) and a single Q-network to accelerate the training of DouZero, called MDou. Our experiments show that MDou improved through self-play using limited computational ability (only a 4-core CPU and 1 GPU) and less learning time (30 days) while achieving comparable performance to DouZero.

Two-Stage Games of Complete but Imperfect Information: Application in International Trade with Tariff

Trade relationships are very complicated especially given the battles between Keynesian and Neo-liberal economic views on the issue. Game theory and its application in international trade can provide a simplistic but important method for quantifying the benefits of trading. In this paper, we intend to use two-stage games of complete but imperfect information in the trading context among two and three countries and compare it with the non-trading country. The purpose of this work is to apply a two-level game model of complete but incomplete information in game theory and apply it to international economics in the context of customs trading. The basic literature for this work is “Game Theory for Applied Economists” by Robert Gibbons. Glenn W. Harrison and E. E. Rustrom on Trade War, Trade Negotiation, and Applied Game Theory. We provide mathematical models and eventually provide the interpretation in a broader context.

Research of artificial intelligence in imperfect information card games

Artificial intelligence (AI) in games has advanced significantly, notably in perfect information games such as Go and Chess. Imperfect information games, in which participants do not have complete information about the game state, create more difficulties. They incorporate both public and private observations, where strategies must be improved to achieve a Nash equilibrium. This study investigates artificial intelligence and reinforcement learning approaches, in which agents learn to maximize future rewards through interactions with their surroundings. The paper then focuses on card game research platforms such as RLCard and OpenAI Gym. It gives a comprehensive summary of research in No Limit Texas Hold'em, a difficult two-player poker game with a large decision space. DeepStack and Libratus are successful systems that have attained expert-level and superhuman play, respectively. Pluribus, a superhuman artificial intelligence for six-player poker, and DouZero, a pure reinforcement learning technique for the multiplayer card game, DouDiZhu, are both investigated. Overall, this paper provides background information on reinforcement learning and imperfect information games, analyzes commonly used research platforms, evaluates the effectiveness of AI algorithms in various card games, and offers future research areas and directions.

Student of Games: A unified learning algorithm for both perfect and imperfect information games.

Games have a long history as benchmarks for progress in artificial intelligence. Approaches using search and learning produced strong performance across many perfect information games, and approaches using game-theoretic reasoning and learning demonstrated strong performance for specific imperfect information poker variants. We introduce Student of Games, a general-purpose algorithm that unifies previous approaches, combining guided search, self-play learning, and game-theoretic reasoning. Student of Games achieves strong empirical performance in large perfect and imperfect information games-an important step toward truly general algorithms for arbitrary environments. We prove that Student of Games is sound, converging to perfect play as available computation and approximation capacity increases. Student of Games reaches strong performance in chess and Go, beats the strongest openly available agent in heads-up no-limit Texas hold'em poker, and defeats the state-of-the-art agent in Scotland Yard, an imperfect information game that illustrates the value of guided search, learning, and game-theoretic reasoning.

Model-Free Neural Counterfactual Regret Minimization With Bootstrap Learning

Counterfactual Regret Minimization (CFR) has achieved many fascinating results in solving large-scale Imperfect Information Games (IIGs). Neural network approximation CFR (neural CFR) is one of the promising techniques that can reduce computation and memory consumption by generalizing decision information between similar states. Current neural CFR algorithms have to approximate cumulative regrets. However, efficient and accurate approximation in a large-scale IIG is still a tough challenge. In this paper, a new CFR variant, Recursive CFR (ReCFR), is proposed. In ReCFR, Recursive Substitute Values (RSVs) are learned and used to replace cumulative regrets. It is proven that ReCFR can converge to a Nash equilibrium at a rate of $O({1}/{\sqrt{T}})$. Based on ReCFR, a new model-free neural CFR with bootstrap learning, Neural ReCFR-B, is proposed. Due to the recursive and non-cumulative nature of RSVs, Neural ReCFR-B has lower-variance training targets than other neural CFRs. Experimental results show that Neural ReCFR-B is competitive with the state-of-the-art neural CFR algorithms at a much lower training cost.

Alternating (In)Dependence-Friendly Logic

Hintikka and Sandu originally proposed Independence Friendly Logic (▪) as a first-order logic of imperfect information to describe game-theoretic phenomena underlying the semantics of natural language. The logic allows for expressing independence constraints among quantified variables, in a similar vein to Henkin quantifiers, and has a nice game-theoretic semantics in terms of imperfect information games. However, the ▪ semantics exhibits some limitations, at least from a purely logical perspective. It treats the players asymmetrically, considering only one of the two players as having imperfect information when evaluating truth, resp., falsity, of a sentence. In addition, truth and falsity of sentences coincide with the existence of a uniform winning strategy for one of the two players in the semantic imperfect information game. As a consequence, ▪ does admit undetermined sentences, which are neither true nor false, thus failing the law of excluded middle. These idiosyncrasies limit its expressive power to the existential fragment of Second Order Logic (▪). In this paper, we investigate an extension of ▪, called Alternating Dependence/Independence Friendly Logic (▪), tailored to overcome these limitations. To this end, we introduce a novel compositional semantics, generalising the one based on trumps proposed by Hodges for ▪. The new semantics (i) allows for meaningfully restricting both players at the same time, (ii) enjoys the property of game-theoretic determinacy, (iii) recovers the law of excluded middle for sentences, and (iv) grants ▪ the full descriptive power of ▪. We also provide an equivalent Herbrand-Skolem semantics and a game-theoretic semantics for the prenex fragment of ▪, the latter being defined in terms of a determined infinite-duration game that precisely captures the other two semantics on finite structures.

RM-FSP: Regret minimization optimizes neural fictitious self-play

To compute the optimal strategy in competitive games, algorithms have been developed to achieve the Nash equilibrium. Current deep learning algorithms have succeeded in many games; however, optimizing the algorithms to approach the Nash equilibrium in imperfect-information games like StarCraft and Poker remains challenging.Neural Fictitious Self-Play (NFSP) is an effective end-to-end algorithm to learn an approximate Nash equilibrium in imperfect-information games. However, because a player in NFSP trains its best response according to its opponents’ past strategies, a discrepancy exists between the optimal strategy and the learned best response after the player updates its strategies. We call this discrepancy the optimality gap. During training, the optimality gap does not decay monotonically, which causes suboptimal results or unstable convergence of NFSP. We improve the performance of NFSP by allowing the optimality gap to decay monotonically. In this study, we propose Regret Minimization Fictitious Self-Play (RM-FSP), which applies a regret minimization method to compute NFSP’s best response. The regret minimization method effectively converges the optimality gap monotonically and faster than in NFSP. We prove there will be a better learning bound than the original NFSP after applying regret minimization methods to NFSP. Experiments on three typical environments in OpenSpiel demonstrate that RM-FSP outperforms NFSP in both exploitability (discrepancy between the learned policy profile and the Nash equilibrium) and time efficiency.

NFSP-PLT: Solving Games with a Weighted NFSP-PER-Based Method

Nash equilibrium strategy is a typical goal when solving two-player imperfect-information games (IIGs). Neural fictitious self-play (NFSP) is a popular method to find the Nash equilibrium in IIGs, which is the first end-to-end method used to compute the Nash equilibrium strategy. However, the training of NFSP requires a large number of sample data and the interactive cost of obtaining such data is often very high. Realizing the efficient training of network under limited samples is an urgent problem. In this paper, we first proposed a new NFSP-based method, NFSP with prioritized experience replay (NFSP-PER), to improve the sample training efficiency. Then, a weighted NFSP-PER with learning time (NFSP-PLT) was proposed to control the utilization degree of priority-weighted samples. Furthermore, based on the NFSP-PLT, an adaptive upper-confidence-bound applied to tree (UCT) is used to solve the optimal response strategy, which makes the solving strategy more accurate. Extensive experimental results show that the proposed NFSP-PLT effectively improves the sample learning efficiency compared with the existing works.

AlphaZe∗∗: AlphaZero-like baselines for imperfect information games are surprisingly strong.

In recent years, deep neural networks for strategy games have made significant progress. AlphaZero-like frameworks which combine Monte-Carlo tree search with reinforcement learning have been successfully applied to numerous games with perfect information. However, they have not been developed for domains where uncertainty and unknowns abound, and are therefore often considered unsuitable due to imperfect observations. Here, we challenge this view and argue that they are a viable alternative for games with imperfect information-a domain currently dominated by heuristic approaches or methods explicitly designed for hidden information, such as oracle-based techniques. To this end, we introduce a novel algorithm based solely on reinforcement learning, called AlphaZe∗∗, which is an AlphaZero-based framework for games with imperfect information. We examine its learning convergence on the games Stratego and DarkHex and show that it is a surprisingly strong baseline, while using a model-based approach: it achieves similar win rates against other Stratego bots like Pipeline Policy Space Response Oracle (P2SRO), while not winning in direct comparison against P2SRO or reaching the much stronger numbers of DeepNash. Compared to heuristics and oracle-based approaches, AlphaZe∗∗ can easily deal with rule changes, e.g., when more information than usual is given, and drastically outperforms other approaches in this respect.

A Survey on Population-Based Deep Reinforcement Learning

Many real-world applications can be described as large-scale games of imperfect information, which require extensive prior domain knowledge, especially in competitive or human–AI cooperation settings. Population-based training methods have become a popular solution to learn robust policies without any prior knowledge, which can generalize to policies of other players or humans. In this survey, we shed light on population-based deep reinforcement learning (PB-DRL) algorithms, their applications, and general frameworks. We introduce several independent subject areas, including naive self-play, fictitious self-play, population-play, evolution-based training methods, and the policy-space response oracle family. These methods provide a variety of approaches to solving multi-agent problems and are useful in designing robust multi-agent reinforcement learning algorithms that can handle complex real-life situations. Finally, we discuss challenges and hot topics in PB-DRL algorithms. We hope that this brief survey can provide guidance and insights for researchers interested in PB-DRL algorithms.

Ensemble strategy learning for imperfect information games

Algorithms with several paradigms (such as rule-based methods, game theory and reinforcement learning) have achieved great success in solving imperfect information games (IIGs). However, agents based on a single paradigm tend to be brittle in certain aspects due to the paradigm’s weaknesses. In this paper, we first present three base-solvers with diversified paradigms for IIGs, and then combine them to design three ensemble-solvers (including an attention ensemble-solver, a gradient ensemble-solver and an evolution ensemble-solver) to learn ensemble strategies given base-solvers’ strengths. We evaluate our methods on Leduc poker with nonstationary opponents and limited games. The results show that our ensemble strategy learning method can effectively integrate the advantages of various advanced individual algorithms and significantly outperform them.

Kdb-D2CFR: Solving Multiplayer imperfect-information games with knowledge distillation-based DeepCFR

Counterfactual regret minimization (CFR) is a popular method for finding approximate Nash equilibrium in imperfect-information games (IIG). However, CFR based methods for the IIG are either only designed for two-player IIGs or require much expert knowledge. In this paper, towards to solve the multiplayer IIG problem without utilizing much expert knowledge, we proposed a practical knowledge distillation based framework, which aims to transferring the knowledge from model of two-player IIG into the multiplayer one. By this framework, both of the training efficiency and performance is improved. To eliminate the requirement of expert knowledge, here we introduced a deep learning based CFR in the framework, by which the counterfactual value of CFR can be estimated in an end-to-end way without any expert knowledge and abstraction. We further propose kdb-DeepCFR and kdb-D2CFR based on DeepCFR and D2CFR respectively, which can effectively solve the strategy of multiplayer large-scale game problems. The extensive experiments conducted on 3-8 players poker games suggest that our method outperforms other baselines in the game performance.

Revealing perceived individuals’ self-interest

In non-cooperative game-theory models, the Nash equilibrium concept is used to assess the outcome of rational decision-makers’ strategic involvement. This method allows researchers to look into certain outcomes while keeping in mind people’s self-interest restrictions. Participants, on the other hand, are not bound to provide accurate information when acting in their own self-interest. In this work, we investigate noncooperative behavior in a repeating game and suggest a new method in game theory for designing an observer. An average repeated non-cooperative Markov game with imperfect information is used to determine the categories of participants in this situation. A sequential technique, in which information is revealed a finite number of times, can approximate all of Nash’s equilibria. By computing the derivative of the player’s equilibrium with regard to his or her present type, we explain how to instrument the observer’s design. The dynamic observer creates the imperfect information game. We study a Markov model extension that incorporates a new variable that reflects the product of the observer design and the distribution vector to solve the problem. The strategies and the observation kernel make up the observer design. In this approach, the approaches are first seen as mappings from private and reported states to lotteries over alternatives. We build equations to retrieve the variables of interest using the available type: observer, strategies, observation kernels, and distribution vectors. We provide a numerical example of an industrial organization in which enterprises compete on the market and decide on the amount of production they will produce simultaneously and independently of one another.

Radar and Jammer Intelligent Game under Jamming Power Dynamic Allocation

In modern electronic warfare, the intelligence of the jammer greatly worsens the anti-jamming performance of traditional passive suppression methods. How to actively design anti-jamming strategies to deal with intelligent jammers is crucial to the radar system. In the existing research on radar anti-jamming strategies’ design, the assumption of jammers is too ideal. To establish a model that is closer to real electronic warfare, this paper explores the intelligent game between a subpulse-level frequency-agile (FA) radar and a transmit/receive time-sharing jammer under jamming power dynamic allocation. Firstly, the discrete allocation model of jamming power is established, and the multiple-round sequential interaction between the radar and the jammer is described based on an extensive-form game. A detection probability calculation method based on the signal-to-interference-pulse-noise ratio (SINR) accumulation gain criterion (SAGC) is proposed to evaluate the game results. Secondly, considering that the competition between the radar and the jammer has the feature of imperfect information, we utilized neural fictitious self-play (NFSP), an end-to-end deep reinforcement learning (DRL) algorithm, to find the Nash equilibrium (NE) of the game. Finally, the simulation results showed that the game between the radar and the jammer can converge to an approximate NE under the established model. The approximate NE strategies are better than the elementary strategies from the perspective of detection probability. In addition, comparing NFSP and the deep Q-network (DQN) illustrates the effectiveness of NFSP in solving the NE of imperfect information games.