Opponent Model Research Articles

OpenHoldem: A Benchmark for Large-Scale Imperfect-Information Game Research.

Owing to the unremitting efforts from a few institutes, researchers have recently made significant progress in designing superhuman artificial intelligence (AI) in no-limit Texas hold'em (NLTH), the primary testbed for large-scale imperfect-information game research. However, it remains challenging for new researchers to study this problem since there are no standard benchmarks for comparing with existing methods, which hinders further developments in this research area. This work presents OpenHoldem, an integrated benchmark for large-scale imperfect-information game research using NLTH. OpenHoldem makes three main contributions to this research direction: 1) a standardized evaluation protocol for thoroughly evaluating different NLTH AIs; 2) four publicly available strong baselines for NLTH AI; and 3) an online testing platform with easy-to-use APIs for public NLTH AI evaluation. We will publicly release OpenHoldem and hope it facilitates further studies on the unsolved theoretical and computational issues in this area and cultivates crucial research problems like opponent modeling and human-computer interactive learning.

Full DouZero+: Improving DouDizhu AI by Opponent Modeling, Coach-Guided Training and Bidding Learning

Full DouZero+: Improving <i>DouDizhu</i> AI by Opponent Modeling, Coach-Guided Training and Bidding Learning

COMB: Scalable Concession-Driven Opponent Models Using Bayesian Learning for Preference Learning in Bilateral Multi-Issue Automated Negotiation

COMB: Scalable Concession-Driven Opponent Models Using Bayesian Learning for Preference Learning in Bilateral Multi-Issue Automated Negotiation

Modeling Rationality: Toward Better Performance Against Unknown Agents in Sequential Games.

Opponent modeling is necessary for autonomous agents to capture the intents of others during strategic interactions. Most previous works assume that they can access enough interaction history to build the model. However, it may not be realistic. To solve this problem, we present a novel rationality-consistent opponent modeling (ROM) method for games with imperfect information. In our approach, a game-theoretical concept of consistence about rationality is proposed to take advantage of the characteristic of imperfect information sequential games that rational behavior at disjoint information sets is correlated through anticipated opponent's behavior. With the correlation between different information sets, agents could infer the opponents' strategies at information sets correlated to observed behavior. To exploit the correlation, ROM attempts to conduct reasoning from the opponent's perspective and rationalize its past behavior. In this way, ROM acquires the ability to better adapt to different opponents and achieves a more accurate opponent model with insufficient observation history, which is verified by experiments in different settings. A heuristic adaptation approach is also applied in ROM, which updates the opponent model in an online manner and significantly reduces the computation cost. We evaluate ROM in both a grid world game and a poker game. Compared with other opponent modeling methods, ROM shows better performance and has more accurate predictions in both games against different types of opponents with limited action interactions. Experimental results also show that ROM's time cost is significantly reduced through heuristic adaptation.

Dynamic Jamming Power Allocation With Incomplete Sensing Information: Improving by GAN and Opponent Modeling

Dynamic Jamming Power Allocation With Incomplete Sensing Information: Improving by GAN and Opponent Modeling

Improved learning efficiency of deep Monte-Carlo for complex imperfect-information card games

Deep Reinforcement Learning (DRL) has achieved considerable success in games involving perfect and imperfect information, such as Go, Texas Hold’em, Stratego, and DouDiZhu. Nevertheless, training a state-of-the-art model for complex imperfect-information card games like DouDiZhu and Big2 remains resource and time-intensive. To address this challenge, this paper introduces two innovative methods: the Opponent Model and Optimized Deep Monte-Carlo (ODMC). These methods are designed to improve the training efficiency of Deep Monte-Carlo (DMC) for imperfect-information card games. The Opponent Model predicts hidden information, enhancing the agent’s learning speed in DMC compared to the original training that only utilizes observed information as input features. In ODMC, the Minimum Combination Search (MCS) is a heuristic search algorithm based on dynamic programming. It calculates the minimum combination of actions in the current state, and ODMC uses MCS to filter suboptimal actions in each state. This reduces the action space considered by DMC, resulting in faster training that focuses on evaluating the most promising actions. The effectiveness of the proposed approach is evaluated by examining two complex card games with imperfect information: DouDiZhu and Big2. Ablation experiments are conducted to evaluate both the Opponent Model (D+OM and B+OM) and ODMC (D+ODMC and B+ODMC), along with their combined variants (D+OMODMC and B+OMODMC). Furthermore, D+OMODMC and B+OMODMC are compared with state-of-the-art DouDiZhu and Big2 artificial intelligence (AI) programs, respectively. The experimental results demonstrate that the proposed methods achieve comparable performance to the original DMC, but with only 25.5% of the training time on the same device. These findings are valuable for mitigating both the equipment requirements and training time in complex imperfect-information card games.

Opponent-Model Search in Games with Incomplete Information

Games with incomplete information are games that model situations where players do not have common knowledge about the game they play, e.g. card games such as poker or bridge. Opponent models can be of crucial importance for decision-making in such games. We propose algorithms for computing optimal and/or robust strategies in games with incomplete information, given various types of knowledge about opponent models. As an application, we describe a framework for reasoning about an opponent's reasoning in such games, where opponent models arise naturally.

Decision-Making With Speculative Opponent Models.

Opponent modeling has proven effective in enhancing the decision-making of the controlled agent by constructing models of opponent agents. However, existing methods often rely on access to the observations and actions of opponents, a requirement that is infeasible when such information is either unobservable or challenging to obtain. To address this issue, we introduce distributional opponent-aided multiagent actor-critic (DOMAC), the first speculative opponent modeling algorithm that relies solely on local information (i.e., the controlled agent's observations, actions, and rewards). Specifically, the actor maintains a speculated belief about the opponents using the tailored speculative opponent models that predict the opponents' actions using only local information. Moreover, DOMAC features distributional critic models that estimate the return distribution of the actor's policy, yielding a more fine-grained assessment of the actor's quality. This thus more effectively guides the training of the speculative opponent models that the actor depends upon. Furthermore, we formally derive a policy gradient theorem with the proposed opponent models. Extensive experiments under eight different challenging multiagent benchmark tasks within the MPE, Pommerman, and starcraft multiagent challenge (SMAC) demonstrate that our DOMAC successfully models opponents' behaviors and delivers superior performance against state-of-the-art (SOTA) methods with a faster convergence speed.

Conflict-based negotiation strategy for human-agent negotiation

Day by day, human-agent negotiation becomes more and more vital to reach a socially beneficial agreement when stakeholders need to make a joint decision together. Developing agents who understand not only human preferences but also attitudes is a significant prerequisite for this kind of interaction. Studies on opponent modeling are predominantly based on automated negotiation and may yield good predictions after exchanging hundreds of offers. However, this is not the case in human-agent negotiation in which the total number of rounds does not usually exceed tens. For this reason, an opponent model technique is needed to extract the maximum information gained with limited interaction. This study presents a conflict-based opponent modeling technique and compares its prediction performance with the well-known approaches in human-agent and automated negotiation experimental settings. According to the results of human-agent studies, the proposed model outpr erforms them despite the diversity of participants’ negotiation behaviors. Besides, the conflict-based opponent model estimates the entire bid space much more successfully than its competitors in automated negotiation sessions when a small portion of the outcome space was explored. This study may contribute to developing agents that can perceive their human counterparts’ preferences and behaviors more accurately, acting cooperatively and reaching an admissible settlement for joint interests.

An opponent model for agent-based shared decision-making via a genetic algorithm.

Shared decision-making (SDM) has received a great deal of attention as an effective way to achieve patient-centered medical care. SDM aims to bring doctors and patients together to develop treatment plans through negotiation. However, time pressure and subjective factors such as medical illiteracy and inadequate communication skills prevent doctors and patients from accurately expressing and obtaining their opponent's preferences. This problem leads to SDM being in an incomplete information environment, which significantly reduces the efficiency of the negotiation and even leads to failure. In this study, we integrated a negotiation strategy that predicts opponent preference using a genetic algorithm with an SDM auto-negotiation model constructed based on fuzzy constraints, thereby enhancing the effectiveness of SDM by addressing the problems posed by incomplete information environments and rapidly generating treatment plans with high mutual satisfaction. A variety of negotiation scenarios are simulated in experiments and the proposed model is compared with other excellent negotiation models. The results indicated that the proposed model better adapts to multivariate scenarios and maintains higher mutual satisfaction. The agent negotiation framework supports SDM participants in accessing treatment plans that fit individual preferences, thereby increasing treatment satisfaction. Adding GA opponent preference prediction to the SDM negotiation framework can effectively improve negotiation performance in incomplete information environments.

Improving Agent Decision Payoffs via a New Framework of Opponent Modeling

The payoff of an agent depends on both the environment and the actions of other agents. Thus, the ability to model and predict the strategies and behaviors of other agents in an interactive decision-making scenario is one of the core functionalities in intelligent systems. State-of-the-art methods for opponent modeling mainly use an explicit model of opponents’ actions, preferences, targets, etc., that the primary agent uses to make decisions. It is more important for an agent to increase its payoff than to accurately predict opponents’ behavior. Therefore, we propose a framework synchronizing the opponent modeling and decision making of the primary agent by incorporating opponent modeling into reinforcement learning. For interactive decisions, the payoff depends not only on the behavioral characteristics of the opponent but also the current state. However, confounding the two obscures the effects of state and action, which then cannot be accurately encoded. To this end, state evaluation is separated from action evaluation in our model. The experimental results from two game environments, a simulated soccer game and a real game called quiz bowl, show that the introduction of opponent modeling can effectively improve decision payoffs. In addition, the proposed framework for opponent modeling outperforms benchmark models.

Effects of altered-reality training on interocular disinhibition in amblyopia.

Training of viewing an altered-reality environment dichoptically has been found to reactivate human adult ocular dominance plasticity, allowing improvement of vision for amblyopia. One suspected mechanism for this training effect is ocular dominance rebalancing through interocular disinhibition. Here, we investigated whether the training modulated the neural responses reflecting interocular inhibition. Thirteen patients with amblyopia and 11 healthy controls participated in this study. Before and after six daily altered-reality training sessions, participants watched flickering video stimuli with their steady-state visually evoked potential (SSVEP) signals recorded simultaneously. We assessed the amplitude of SSVEP response at intermodulation frequencies, which was a potential neural indicator of interocular suppression. The results showed that training weakened the intermodulation response only in the amblyopic group, which was in agreement with the hypothesis that the training reduced interocular suppression specific to amblyopia. Moreover, even one month after the training ended, we could still observe this neural training effect. These findings provide preliminary neural evidence in support of the disinhibition account for treating amblyopia. We also explain these results with the ocular opponency model, which, to our knowledge, is the first time for this binocular rivalry model to be used in explaining long-term ocular dominance plasticity.

Know Thy Enemy: An Opponent Modeling-Based Anti-Intelligent Jamming Strategy Beyond Equilibrium Solutions

We investigate the problem of dynamic spectrum anti-jamming access against intelligent jammer using game theory and opponent modeling. Previous work has formulated the interaction between the user and the intelligent jammer as an adversarial game, and aimed to find the Nash Equilibrium (NE). However, sticking to NE will lead to overcautious behaviors and can’t achieve the best performance while the jammer is sub-optimal. Thus, this letter tries to exploit the adaptive jammer and find the Best Response (BR) rather than NE. We propose the minimax deep Q network (DQN) to approximate anti-jamming utility while applying imitation learning to reason about the jammer’s policy. Based on the utility and imitation jamming policy, the user is able to find the policy beyond equilibrium solutions and enhance anti-jamming performance. Numerical results demonstrate that our scheme achieves a 30% improvement in success access rate over the NE-based and single-user DRL.

Dynamic Hypergames for Synthesis of Deceptive Strategies With Temporal Logic Objectives

In this paper, we study the use of deception for strategic planning in adversarial environments. We model the interaction between the agent (player 1) and the adversary (player 2) as a two-player concurrent stochastic game in which the adversary has incomplete information about the agent’s task specification given as a temporal logic formula. During the interaction, the adversary can infer the agent’s intention from observations and adapt its strategy so as to prevent the agent from satisfying the objective. To plan against such an adaptive opponent, the agent must leverage its knowledge about the adversary’s incomplete information to influence the behavior of the opponent, and thereby be deceptive. To synthesize a deceptive strategy, we introduce a class of hypergame models that capture the interaction between the agent and its adversary given asymmetric, incomplete information. We develop a solution concept for this class of hypergames and show that the subjectively rationalizable strategy for the agent is deceptive and maximizes the probability of satisfying the task in temporal logic. Such a deceptive strategy is obtained by modeling the opponent’s evolving perception of the agent’s objective and integrating it into planning. This allows the agent to manipulate the opponent’s perception so as to induce the opponent into taking actions that benefit the agent. We demonstrate the effectiveness of our deceptive planning algorithm using robot motion planning examples with temporal logic objectives and design a detection mechanism to notify the agent of potential errors in modeling the adversary’s behavior. Note to Practitioners—Many security and defense applications employ deception mechanisms for strategic advantages. This work presents a game-theoretic framework for planning deceptive strategies in stochastic environments and shows that the opponent modeling plays a key role in the design of effective deception mechanisms. For applications to cyber-physical security, the practitioners can employ temporal logic for specifying security properties in the system and analyze defense with deception using the proposed methods.

Athlete doping confrontation efficacy and confrontation likelihood

Evidence supports the role of coach doping confrontation efficacy (DCE; Sullivan et al., 2015) as a deterrent against athletes’ doping cognitions (Boardley et al., 2019; Sullivan & Razavi, 2017), but the role of the athlete has largely been ignored. Current anti-doping campaigns encourage athletes to report doping misconduct (i.e., whistleblowing), but some athletes would prefer to confront the athlete directly (Erickson et al., 2017). Thus, it is important to consider what may contribute to athletes’ likelihood to confront a doping teammate or opponent. The purpose of this study was to determine whether DCE could predict an athlete’s likelihood to confront a doping teammate or opponent. Additionally, doping moral disengagement (MD) was included as a possible moderator of this relationship. Surveys were completed by 155 college athletes (nmale = 145) to measure their perceived DCE, doping MD, likelihood to confront a teammate, and likelihood to confront an opponent. Separate linear regression analyses were run for the two targets of confrontation. In the teammate model, both DCE and doping MD were significant predictors of confrontation likelihood. DCE was the only significant predictor in the opponent model. Neither model presented with a significant interaction, suggesting no moderation effect. Results suggest perceived DCE is associated with a greater likelihood to confront a doping athlete, regardless of whether they are a teammate or opponent; however, moral disengagement plays a greater role if the athlete is a teammate. These findings imply that confrontation may be the first line of defense against doping before whistleblowing action is taken. Research should continue to explore antecedents and consequences of athlete doping confrontation, providing greater insight into the whistleblowing process.

Multi-agent actor-critic with time dynamical opponent model

In multi-agent reinforcement learning, multiple agents learn simultaneously while interacting with a common environment and each other. Since the agents adapt their policies during learning, not only the behavior of a single agent becomes non-stationary, but also the environment as perceived by the agent. This renders it particularly challenging to perform policy improvement. In this paper, we propose to exploit the fact that the agents seek to improve their expected cumulative reward and introduce a novel Time Dynamical Opponent Model (TDOM) to encode the knowledge that the opponent policies tend to improve over time. We motivate TDOM theoretically by deriving a lower bound of the log objective of an individual agent and further propose Multi-Agent Actor-Critic with Time Dynamical Opponent Model (TDOM-AC). We evaluate the proposed TDOM-AC on a differential game and the Multi-agent Particle Environment. We show empirically that TDOM achieves superior opponent behavior prediction during test time. The proposed TDOM-AC methodology outperforms state-of-the-art Actor-Critic methods on the performed tasks in cooperative and especially in mixed cooperative-competitive environments. TDOM-AC results in a more stable training and a faster convergence. Our code is available at https://github.com/Yuantian013/TDOM-AC.

Looking more masculine among females: Spatial context modulates gender perception of face and biological motion.

Perception of visual information highly depends on spatial context. For instance, perception of a low-level visual feature, such as orientation, can be shifted away from its surrounding context, exhibiting a simultaneous contrast effect. Although previous studies have demonstrated the adaptation aftereffect of gender, a high-level visual feature, it remains largely unknown whether gender perception can also be shaped by a simultaneously presented context. In the present study, we found that the gender perception of a central face or a point-light walker was repelled away from the gender of its surrounding faces or walkers. A norm-based opponent model of lateral inhibition, which accounts for the adaptation aftereffect of high-level features, can also excellently fit the simultaneous contrast effect. But different from the reported contextual effect of low-level features, the simultaneous contrast effect of gender cannot be observed when the centre and the surrounding stimuli are from different categories, or when the surrounding stimuli are suppressed from awareness. These findings on one hand reveal a resemblance between the simultaneous contrast effect and the adaptation aftereffect of high-level features, on the other hand highlight different biological mechanisms underlying the contextual effects of low- and high-level visual features.

Entropy regularized actor-critic based multi-agent deep reinforcement learning for stochastic games

Multi-agent reinforcement learning (MARL) is an abstract framework modeling a dynamic environment that involves multiple learning and decision-making agents, each of which tries to maximize her cumulative reward. In MARL, each agent discovers a strategy alongside others and adapts her policy in response to the behavioural changes of others. A fundamental difficulty faced by MARL is that every agent is dynamically learning and changing to improve her reward, making the whole system unstable and agents’ policies difficult to converge. In this paper, we introduce the entropy regularizer into the Bellman equation and utilize Lagrange approach to optimize the entropy regularizer. We then propose a MARL algorithm based on the maximum entropy principle and the actor-critic method. This algorithm follows the policy gradient approach and uses a policy network and a value network. We call it Multi-Agent Deep Soft Policy Gradient (MADSPG). Then by using the Lagrange approach and dynamic minimax optimization, we propose the AUTO-MADSPG algorithm with an automatically adjusted entropy regularizer. These algorithms make multi-agent learning more stable while sufficient exploration is guaranteed. Finally, we also incorporate MADSPG with the recently proposed opponent modeling component into an integrated framework. This framework outperforms many state-of-the-art MARL algorithms in conventional cooperative and competitive game settings.

Deep multiagent reinforcement learning: challenges and directions

This paper surveys the field of deep multiagent reinforcement learning (RL). The combination of deep neural networks with RL has gained increased traction in recent years and is slowly shifting the focus from single-agent to multiagent environments. Dealing with multiple agents is inherently more complex as (a) the future rewards depend on multiple players’ joint actions and (b) the computational complexity increases. We present the most common multiagent problem representations and their main challenges, and identify five research areas that address one or more of these challenges: centralised training and decentralised execution, opponent modelling, communication, efficient coordination, and reward shaping. We find that many computational studies rely on unrealistic assumptions or are not generalisable to other settings; they struggle to overcome the curse of dimensionality or nonstationarity. Approaches from psychology and sociology capture promising relevant behaviours, such as communication and coordination, to help agents achieve better performance in multiagent settings. We suggest that, for multiagent RL to be successful, future research should address these challenges with an interdisciplinary approach to open up new possibilities in multiagent RL.

OM-TCN: A dynamic and agile opponent modeling approach for competitive games

The non-stationarity of the environment is a crucial challenge for competitive Multi-Agent Reinforcement Learning (MARL) due to the constantly changing opponent policy. Existing schemes are challenging to make the protagonist agent that agilely responds to the opponent’s changes and the resulting non-stationarity, which may inevitably limit their applicability. To address the dynamic opponent policy and adapt to the non-stationary environment continuously, we propose a Temporal Convolutional Network (TCN) model for modeling and predicting opponent behaviors called OM-TCN, and apply it to the widely-used Multi-Agent Deep Deterministic Policies Gradient (MADDPG) algorithm of competitive MARL. In this work, we collect the opponent’s behavior data observed by the protagonist agent and serialize it in granularity of episodes. Then we input the time-series data into OM-TCN for sequence modeling. The OM-TCN learns the historical behaviors of the opponent instead of overfitting to a specific opponent policy, and can make predictions about the opponent’s future actions. Finally, we use predictions of opponent actions in place of the history sampled from the playback buffer, and apply the OM-TCN model to the MADDPG framework for decentralized training. We use the competitive scenario of Multi-agent Particle Environment (MPE) to evaluate the proposed method. Simulation results show that the protagonist agent is able to learn more efficient and stable policy and converge easier than other baselines.