Learning Opponent Research Articles

Strategic Teaching and Learning in Games

We show there is no uncoupled learning heuristic leading to Nash equilibrium in all finite games that a player has an incentive to adopt, that would be evolutionary stable, or that could “learn itself.” Rather, a player has an incentive to strategically teach a learning opponent to secure at least the Stackelberg leader payoff. This observation holds even when we restrict to generic games, two-player games, potential games, games with strategic complements, or 2 × 2 games, in which learning is known to be “nice.” It also applies to uncoupled learning heuristics leading to correlated equilibria, rationalizability, iterated admissibility, or minimal CURB sets. (JEL C73, D83)

Learning to Model Opponent Learning (Student Abstract)

Multi-Agent Reinforcement Learning (MARL) considers settings in which a set of coexisting agents interact with one another and their environment. The adaptation and learning of other agents induces non-stationarity in the environment dynamics. This poses a great challenge for value function-based algorithms whose convergence usually relies on the assumption of a stationary environment. Policy search algorithms also struggle in multi-agent settings as the partial observability resulting from an opponent's actions not being known introduces high variance to policy training. Modelling an agent's opponent(s) is often pursued as a means of resolving the issues arising from the coexistence of learning opponents. An opponent model provides an agent with some ability to reason about other agents to aid its own decision making. Most prior works learn an opponent model by assuming the opponent is employing a stationary policy or switching between a set of stationary policies. Such an approach can reduce the variance of training signals for policy search algorithms. However, in the multi-agent setting, agents have an incentive to continually adapt and learn. This means that the assumptions concerning opponent stationarity are unrealistic. In this work, we develop a novel approach to modelling an opponent's learning dynamics which we term Learning to Model Opponent Learning (LeMOL). We show our structured opponent model is more accurate and stable than naive behaviour cloning baselines. We further show that opponent modelling can improve the performance of algorithmic agents in multi-agent settings.

Real-time opponent learning in automated negotiation using recursive Bayesian filtering

Automated negotiation is a toolset to model human interactions during a negotiation process with the aim of improving the efficiency and quality of decision-making using advanced information analytics. During the negotiation, the participants share their viewpoints and concerns about the negotiation issues. However, in reality, they usually do not reveal the details of their preferences to one another. Therefore, modeling and learning opponents’ behavior is a crucial component of automated negotiation. In this paper, we propose an estimation technique based on recursive Bayesian filtering to facilitate opponent-modeling and -learning in the context of multi-participant, multi-issue negotiations. In the proposed technique, opponents’ preference profiles are modeled using fuzzy functions, which are very close to the way humans evaluate alternatives. As the negotiation progresses, the agents can recursively learn the parameters of these models in real time. The only required information for this learning process includes the feedback and the arguments the participants may provide in support of their decisions. At each round, a probabilistic graphical model is also implemented that utilizes the learned preference limits of the participants to offer a new proposal with a high probability of satisfying the participants and reaching an agreement. The proposed methodology is examined in two different negotiation contexts: energy-system development and real estate service. The experiments show that the proposed opponent modeling/learning approach increases the efficiency of the negotiation up to 85% and facilitates reaching an agreement in fewer rounds of negotiation without requiring any prior understanding of the negotiation participants.

An Evolutionary Approach for Learning Opponent's Deadline and Reserve Points in Multi-Issue Negotiation

The efficiency of automated multi-issue negotiation depends on the available information about the opponent. In a competitive negotiation environment, agents do not reveal their parameters to their opponents in order to avoid exploitation. Several researchers have argued that an agent's optimal strategy can be determined using the opponent's deadline and reserve points. In this paper, we propose a new learning agent, so-called Evolutionary Learning Agent (ELA), able to estimate its opponent's deadline and reserve points in bilateral multi-issue negotiation based on opponent's counter-offers (without any additional extra information). ELA reduces the learning problem to a system of non-linear equations and uses an evolutionary algorithm based on the elitism aspect to solve it. Experimental study shows that our learning agent outperforms others agents by improving its outcome in term of average and joint utility

Strategic Teaching and Learning in Games

It is known that there are uncoupled learning heuristics leading to Nash equilibrium in all finite games. Why should players use such learning heuristics and where could they come from? We show that there is no uncoupled learning heuristic leading to Nash equilibrium in all finite games that a player has an incentive to adopt, that would be evolutionary stable or that could learn itself. Rather, a player has an incentive to teach such a learning opponent in order secure at least the Stackelberg leader payoff. The impossibility result remains intact when restricted to the classes of generic games, two-player games, potential games, games with strategic complements or 2 x 2 games, in which learning is known to be nice. More generally, it also applies to uncoupled learning heuristics leading to correlated equilibria, rationalizable outcomes, iterated admissible outcomes, or minimal curb sets. A possibility result restricted to strategically trivial games fails if some generic games outside this class are considered as well.

Learning opponent’s preferences for effective negotiation: an approach based on concept learning

We consider automated negotiation as a process carried out by software agents to reach a consensus. To automate negotiation, we expect agents to understand their user's preferences, generate offers that will satisfy their user, and decide whether counter offers are satisfactory. For this purpose, a crucial aspect is the treatment of preferences. An agent not only needs to understand its own user's preferences, but also its opponent's preferences so that agreements can be reached. Accordingly, this paper proposes a learning algorithm that can be used by a producer during negotiation to understand consumer's needs and to offer services that respect consumer's preferences. Our proposed algorithm is based on inductive learning but also incorporates the idea of revision. Thus, as the negotiation proceeds, a producer can revise its idea of the consumer's preferences. The learning is enhanced with the use of ontologies so that similar service requests can be identified and treated similarly. Further, the algorithm is targeted to learning both conjunctive as well as disjunctive preferences. Hence, even if the consumer's preferences are specified in complex ways, our algorithm can learn and guide the producer to create well-targeted offers. Further, our algorithm can detect whether some preferences cannot be satisfied early and thus consensus cannot be reached. Our experimental results show that the producer using our learning algorithm negotiates faster and more successfully with customers compared to several other algorithms.

Learning opponent’s beliefs via fuzzy constraint-directed approach to make effective agent negotiation

This work presents a general framework of agent negotiation with opponent learning via fuzzy constraint-directed approach. The fuzzy constraint-directed approach involves the fuzzy probability constraint and the fuzzy instance reasoning. The proposed approach via fuzzy probability constraint can not only cluster the opponent's information in negotiation process as proximate regularities to improve the convergence of behavior patterns, but also eliminate the noisy hypotheses or beliefs to enhance the effectiveness on beliefs learning. By using fuzzy instance method, our approach can reuse the prior opponent knowledge to speed up the problem-solving, and reason the proximate regularities to acquire desirable results on predicting opponent behavior. In addition, the proposed interaction method enables the agent to make a concession dynamically based on expected objectives. Moreover, experimental results suggest that the proposed framework allows an agent to achieve a higher reward, a fairer deal, or a smaller cost of negotiation.

TOWARDS OPTIMIZING ENTERTAINMENT IN COMPUTER GAMES

Mainly motivated by the current lack of a qualitative and quantitative entertainment formulation of computer games and the procedures to generate it, this article covers the following issues: It presents the features—extracted primarily from the opponent behavior—that make a predator/prey game appealing; provides the qualitative and quantitative means for measuring player entertainment in real time, and introduces a successful methodology for obtaining games of high satisfaction. This methodology is based on online (during play) learning opponents who demonstrate cooperative action. By testing the game against humans, we confirm our hypothesis that the proposed entertainment measure is consistent with the judgment of human players. As far as learning in real time against human players is concerned, results suggest that longer games are required for humans to notice some sort of change in their entertainment.

A Bayesian classifier for learning opponents’ preferences in multi-object automated negotiation

We present a classification method for learning an opponent’s preferences during a bilateral multi-issue negotiation. Similar candidate preference relations over the set of offers are grouped into classes, and a Bayesian technique is used to determine, for each class, the likelihood that the opponent’s true preference relation lies in that class. Evidence used for classification decision-making is obtained by observing the opponent’s sequence of offers, and applying the concession assumption, which states that negotiators usually decrease their offer utilities as time passes in order to find a deal. Simple experiments show that the technique can find the correct class after very few offers and can select a preference relation that is likely to match closely with the opponent’s true preferences.