Iterated Prisoner's Dilemma Game Research Articles

On the Role of Emotions in Games

Numerous experiments have shown that players cooperate in a prisoner's dilemma (PD) game. I contribute to this literature by showing that emotions might be the key to overcoming the coordination failure. I model emotions by generally distinguishing between two types of emotions: First, expected emotions play a central role. These are experienced not during the decision-making, but are rather ex post, i. e. after the outcomes are realized. In this regard, standard theory accounts for expected emotions by allowing for phenomenons such as fairness considerations, inequality aversion and temptation in iterated PD games. I describe how expected emotions - in particular satisfaction, anger, remorse and disappointment - may have an impact on the perceived payoff in a PD game. Second, immediate emotions seem to be equally important in my model. Immediate emotions, just as expected emotions, arise from considering the consequences of one's decision. The crucial difference, however, is that they are experienced during the decision-making.In introducing immediate emotions I apply the projection bias model developed by Loewenstein et al. (2003) and embed the perceived payoff into the projection bias model. I find that expected emotions might lead to the emergence of a new Pareto-efficient Nash equilibrium in which all players cooperate. However, immediate emotions might destroy this equilibrium instantaneously.

Evolution of Strategies With Different Representation Schemes in a Spatial Iterated Prisoner's Dilemma Game

The iterated prisoner's dilemma (IPD) game has been frequently used to examine the evolution of cooperative behavior among agents in the field of evolutionary computation. It has been demonstrated that various factors are related to the evolution of cooperative behavior. One well-known factor is spatial relations among agents. The IPD game is often played in a 2-D grid world. Such a spatial IPD game has a neighborhood structure, which is used to choose opponents for the IPD game and parents for genetic operations. Another important factor is the choice of a representation scheme to encode the strategy of each agent. Different representation schemes often lead to different results. Whereas the choice of a representation scheme is known to be important, a mixture of different representation schemes has not been examined for the spatial IPD game in the literature. That is, a population of homogeneous agents with the same representation scheme has been usually assumed in the literature. In this paper, we introduce the use of different representation schemes in a single population to the spatial IPD game in order to examine the evolution of cooperative behavior under more general assumptions. With the use of different representation schemes, we can examine the evolution of cooperative behavior in various settings such as partial interaction through the IPD game, partial interaction through crossover, full interaction through the IPD game and crossover, and no interaction between different subpopulations of agents.

Spiking Neural Networks with Different Reinforcement Learning (RL) Schemes in a Multiagent Setting

This paper investigates the effectiveness of spiking agents when trained with reinforcement learning (RL) in a challenging multiagent task. In particular, it explores learning through reward-modulated spike-timing dependent plasticity (STDP) and compares it to reinforcement of stochastic synaptic transmission in the general-sum game of the Iterated Prisoner's Dilemma (IPD). More specifically, a computational model is developed where we implement two spiking neural networks as two "selfish" agents learning simultaneously but independently, competing in the IPD game. The purpose of our system (or collective) is to maximise its accumulated reward in the presence of reward-driven competing agents within the collective. This can only be achieved when the agents engage in a behaviour of mutual cooperation during the IPD. Previously, we successfully applied reinforcement of stochastic synaptic transmission to the IPD game. The current study utilises reward-modulated STDP with eligibility trace and results show that the system managed to exhibit the desired behaviour by establishing mutual cooperation between the agents. It is noted that the cooperative outcome was attained after a relatively short learning period which enhanced the accumulation of reward by the system. As in our previous implementation, the successful application of the learning algorithm to the IPD becomes possible only after we extended it with additional global reinforcement signals in order to enhance competition at the neuronal level. Moreover it is also shown that learning is enhanced (as indicated by an increased IPD cooperative outcome) through: (i) strong memory for each agent (regulated by a high eligibility trace time constant) and (ii) firing irregularity produced by equipping the agents' LIF neurons with a partial somatic reset mechanism.

Cognitive and Motivational Requirements for the Emergence of Cooperation in a Rat Social Game

BackgroundGame theory and the Prisoner's Dilemma (PD) game in particular, which captures the paradox of cooperative interactions that lead to benefits but entail costs to the interacting individuals, have constituted a powerful tool in the study of the mechanisms of reciprocity. However, in non-human animals most tests of reciprocity in PD games have resulted in sustained defection strategies. As a consequence, it has been suggested that under such stringent conditions as the PD game humans alone have evolved the necessary cognitive abilities to engage in reciprocity, namely, numerical discrimination, memory and control of temporal discounting.Methodology/Principal FindingsWe use an iterated PD game to test rats (Rattus norvegicus) for the presence of such cognitive abilities by manipulating the strategy of the opponent, Tit-for-Tat and Pseudo-Random, or the relative size of the temptation to defect. We found that rats shape their behaviour according to the opponent's strategy and the relative outcome resulting from cooperative or defective moves. Finally, we show that the behaviour of rats is contingent upon their motivational state (hungry versus sated).Conclusions/SignificanceHere we show that rats understand the payoff matrix of the PD game and the strategy of the opponent. Importantly, our findings reveal that rats possess the necessary cognitive capacities for reciprocity-based cooperation to emerge in the context of a prisoner's dilemma. Finally, the validation of the rat as a model to study reciprocity-based cooperation during the PD game opens new avenues of research in experimental neuroscience.

Evolution of cooperative behavior among heterogeneous agents with different strategy representations in an iterated prisoner’s dilemma game

The iterated prisoner’s dilemma (IPD) game has frequently been used to examine the evolution of cooperative behavior among agents. When the effect of representation schemes of IPD game strategies was examined, the same representation scheme was usually assigned to all agents. That is, in the literature, a population of homogeneous agents was usually used in computational experiments. In this article, we focus on a slightly different situation where every agent does not necessarily use the same representation scheme. That is, a population can be a mixture of heterogeneous agents with different representation schemes. In computational experiments, we used binary strings of different lengths (i.e., three-bit and five-bit strings) to represent IPD game strategies. We examined the evolution of cooperative behavior among heterogeneous agents in comparison with the case of homogeneous ones for the standard IPD game with typical payoff values of 0, 1, 3, and 5. Experimental results showed that the evolution of cooperative behavior was slowed down by the use of heterogeneous agents. It was also demonstrated that a faster evolution of cooperative behavior is achieved among majority agents than among minority ones in a heterogeneous population.

Self-control with spiking and non-spiking neural networks playing games

Self- control can be defined as choosing a large delayed reward over a small immediate reward, while precommitment is the making of a choice with the specific aim of denying oneself future choices. Humans recognise that they have self-control problems and attempt to overcome them by applying precommitment. Problems in exercising self-control, suggest a conflict between cognition and motivation, which has been linked to competition between higher and lower brain functions (representing the frontal lobes and the limbic system respectively). This premise of an internal process conflict, lead to a behavioural model being proposed, based on which, we implemented a computational model for studying and explaining self-control through precommitment behaviour. Our model consists of two neural networks, initially non-spiking and then spiking ones, representing the higher and lower brain systems viewed as cooperating for the benefit of the organism. The non-spiking neural networks are of simple feed forward multilayer type with reinforcement learning, one with selective bootstrap weight update rule, which is seen as myopic, representing the lower brain and the other with the temporal difference weight update rule, which is seen as far-sighted, representing the higher brain. The spiking neural networks are implemented with leaky integrate-and-fire neurons with learning based on stochastic synaptic transmission. The differentiating element between the two brain centres in this implementation is based on the memory of past actions determined by an eligibility trace time constant. As the structure of the self-control problem can be likened to the Iterated Prisoner’s Dilemma (IPD) game in that cooperation is to defection what self-control is to impulsiveness or what compromising is to insisting, we implemented the neural networks as two players, learning simultaneously but independently, competing in the IPD game. With a technique resembling the precommitment effect, whereby the payoffs for the dilemma cases in the IPD payoff matrix are differentially biased (increased or decreased), it is shown that increasing the precommitment effect (through increasing the differential bias) increases the probability of cooperating with oneself in the future, irrespective of whether the implementation is with spiking or non-spiking neural networks.

Measuring Generalization Performance in Coevolutionary Learning

Coevolutionary learning involves a training process where training samples are instances of solutions that interact strategically to guide the evolutionary (learning) process. One main research issue is with the generalization performance, i.e., the search for solutions (e.g., input-output mappings) that best predict the required output for any new input that has not been seen during the evolutionary process. However, there is currently no such framework for determining the generalization performance in coevolutionary learning even though the notion of generalization is well-understood in machine learning. In this paper, we introduce a theoretical framework to address this research issue. We present the framework in terms of game-playing although our results are more general. Here, a strategy's generalization performance is its average performance against all test strategies. Given that the true value may not be determined by solving analytically a closed-form formula and is computationally prohibitive, we propose an estimation procedure that computes the average performance against a small sample of random test strategies instead. We perform a mathematical analysis to provide a statistical claim on the accuracy of our estimation procedure, which can be further improved by performing a second estimation on the variance of the random variable. For game-playing, it is well-known that one is more interested in the generalization performance against a biased and diverse sample of good test strategies. We introduce a simple approach to obtain such a test sample through the multiple partial enumerative search of the strategy space that does not require human expertise and is generally applicable to a wide range of domains. We illustrate the generalization framework on the coevolutionary learning of the iterated prisoner's dilemma (IPD) games. We investigate two definitions of generalization performance for the IPD game based on different performance criteria, e.g., in terms of the number of wins based on individual outcomes and in terms of average payoff. We show that a small sample of test strategies can be used to estimate the generalization performance. We also show that the generalization performance using a biased and diverse set of good test strategies is lower compared to the unbiased case for the IPD game. This is the first time that generalization is defined and analyzed rigorously in coevolutionary learning. The framework allows the evaluation of the generalization performance of any coevolutionary learning system quantitatively.

Multiple Choices and Reputation in Multiagent Interactions

Coevolutionary learning provides a framework for modeling more realistic iterated prisoner's dilemma (IPD) interactions and to study conditions of how and why certain behaviors (e.g., cooperation) in a complex environment can be learned through an adaptation process guided by strategic interactions. The coevolutionary learning of cooperative behaviors can be attributed to the mechanism of direct reciprocity (e.g., repeated encounters). However, for the more complex IPD game with more choices, it is unknown precisely why the mechanism of direct reciprocity is less effective in promoting the learning of cooperative behaviors. Here, our study suggests that the evolution of defection may be a result of strategies effectively having more opportunities to exploit others when there are more choices. We note that strategies are less able to resolve the intention of an intermediate choice, e.g., whether it is a signal to engender further cooperation or a subtle exploitation. A likely consequence is strategies adapting to lower cooperation plays that offer higher payoffs in the short-term view when they cannot resolve the intention of opponents. However, cooperation in complex human interactions may also involve indirect interactions rather than direct interactions only. Following this, we study the coevolutionary learning of IPD with more choices and reputation. Here, current behavioral interactions depend not only on choices made in previous moves (direct interactions), but also choices made in past interactions that are reflected by their reputation scores (indirect interactions). The coevolutionary learning of cooperative behaviors is possible in the IPD with more choices when strategies use reputation as a mechanism to estimate behaviors of future partners and to elicit mutual cooperation play right from the start of interactions. In addition, we study the impact of the accuracy of reputation estimation in reflecting strategy behaviors of different implementations and why it is important for the evolution of cooperation. We show that the accuracy is related to how memory of games from previous generations is incorporated to calculate reputation scores and how frequently reputation scores are updated.

Kinds of kindness: classifying the causes of altruism and cooperation

Kinds of kindness: classifying the causes of altruism and cooperation

Behavioral Diversity, Choices and Noise in the Iterated Prisoner's Dilemma

Real-world dilemmas rarely involve just two choices and perfect interactions without mistakes. In the iterated prisoner's dilemma (IPD) game, intermediate choices or mistakes (noise) have been introduced to extend its realism. This paper studies the IPD game with both noise and multiple levels of cooperation (intermediate choices) in a coevolutionary environment, where players can learn and adapt their strategies through an evolutionary algorithm. The impact of noise on the evolution of cooperation is first examined. It is shown that the coevolutionary models presented in this paper are robust against low noise (when mistakes occur with low probability). That is, low levels of noise have little impact on the evolution of cooperation. On the other hand, high noise (when mistakes occur with high probability) creates misunderstandings and discourages cooperation. However, the evolution of cooperation in the IPD with more choices in a coevolutionary learning setting also depends on behavioral diversity. This paper further investigates the issue of behavioral diversity in the coevolution of strategies for the IPD with more choices and noise. The evolution of cooperation is more difficult to achieve if a coevolutionary model with low behavioral diversity is used for IPD games with higher levels of noise. The coevolutionary model with high behavioral diversity in the population is more resistant to noise. It is shown that strategy representations can have a significant impact on the evolutionary outcomes because of different behavioral diversities that they generate. The results further show the importance of behavioral diversity in coevolutionary learning.

Evolution of Iterated Prisoner's Dilemma Game Strategies in Structured Demes Under Random Pairing in Game Playing

We discuss the evolution of strategies in a spatial iterated prisoner's dilemma (IPD) game in which each player is located in a cell of a two-dimensional grid-world. Following the concept of structured demes, two neighborhood structures are used. One is for the interaction among players through the IPD game. A player in each cell plays against its neighbors defined by this neighborhood structure. The other is for mating strategies by genetic operations. A new strategy for a player is generated by genetic operations from a pair of parent strings, which are selected from its neighboring cells defined by the second neighborhood structure. After examining the effect of the two neighborhood structures on the evolution of cooperative behavior with standard pairing in game-playing, we introduce a random pairing scheme in which each player plays against a different randomly chosen neighbor at every round (i.e., every iteration) of the game. Through computer simulations, we demonstrate that small neighborhood structures facilitate the evolution of cooperative behavior under random pairing in game-playing.

Spatial strategies in a generalized spatial prisoner’s dilemma

Many strategies, such as tit-for-tat, have been proposed in the iterated prisoner’s dilemma (IPD) in which the prisoner’s dilemma (PD) is carried out repeatedly with two players. A spatial version of the iterated prisoner’s dilemma (SPD) has been studied, where a player at each site plays the IPD game with all the players in the neighborhood. However, the strategies studied in the SPD consider the past actions of a single opponent only. We studied spatial strategies that depend on the configuration of actions taken by all neighbors (as opposed to conventional temporal strategies). Since generosity can be considered as a spatial strategy, we first investigate the generosity required when an action error is involved. We also propose several spatial strategies that outperform many others.

Stabilization of tag-mediated interaction by sexual reproduction in an evolutionary agent system

The evolution of cooperation in a system of agents playing the iterated prisoner's dilemma (IPD) is investigated. We present results for the standard 2-person IPD as well as the more general N-person IPD (NIPD) game. In our computational model, agents have visible tags and choose whether to interact or not based upon these. We consider the evolutionary stability of the evolving populations. We extend previous work by introducing sexual reproduction (recombination) of agents and by analyzing its influence on the evolving populations. We observed the occasional formation of very stable cooperative societies, as opposed to previous results without sexual reproduction. These cooperative societies are able to resist invasions of “mimics” (defecting agents with the tag of a cooperating agent).

Evolution of cooperation: cooperation defeats defection in the cornfield model

“Cooperation” defines any behavior that enhances the fitness of a group (e.g. a community or species), but which, by its nature, can be exploited by selfish individuals, meaning, firstly, that selfish individuals derive an advantage from exploitation which is greater than the average advantage that accrues to unselfish individuals . Secondly, exploitation has no intrinsic fitness value except in the presence of the “cooperative behavior”. The mathematics is described by the simple Prisoner's Dilemma Game (PDG). It has previously been shown that koinophilia (the avoidance of sexual mates displaying unusual or atypical phenotypic features, such as mutations) stabilizes any inherited strategy in the simple or iterated PDG, meaning that it cannot be displaced by rare forms of alternative behavior which arise through mutation or occasional migration. In the present model equal numbers of cooperators and defectors (in the simple PDG) were randomly spread in a two-dimensional “cornfield” with uniformly distributed resources. Every individual was koinophilic, and interacted (sexually and in the PDG tournaments) only with individuals from within its immediate neighborhood. This model therefore tested whether cooperation can outcompete defection or selfishness in a straight, initially equally matched, evolutionary battle. The results show that in the absence of koinophilia cooperation was rapidly driven to extinction. With koinophilia there was a very rapid loss of cooperators in the first few generations, but thereafter cooperation slowly spread, ultimately eliminating defection completely. This result was critically dependent on sampling effects of neighborhoods. Small samples (resulting from low population densities or small neighborhood sizes) increase the probability that a chance neighborhood comes to consist predominantly of cooperators. A sexual preference for the most common phenotype in the neighborhood then makes that phenotype more common still. Once this occurs cooperation's spread becomes almost inevitable.

Patch dynamics based on Prisoner's Dilemma game: superiority of golden rule

There has been much literature on ecological model of Prisoner's Dilemma (PD) game. This game illustrates that cooperation can evolve in situations where individuals tend to look after themselves. In order to explain some behaviors of altruism in animal societies, the strategy All Cooperate (AC), often called the Golden Rule, is more appropriate than other strategies. However, very little is known about the superiority of AC. In the present article, we study patch dynamics based on non-iterated PD game, applying two different methods: island and lattice models. Each patch is assumed to be either vacant or composed of a population of AC or All Defect (AD), where AD means a selfish strategy. Both models exhibit a phase transition between a phase where both AC and AD survive, and a phase where AD is extinct. The latter phase means that AC beats AD completely. In the case of lattice model, the extinction of AD easily occurs and the abundance of AC takes a larger value, compared with the island model. Our models can be also extended to general iterated PD game; we describe the reason why AC can outperform any other strategy.

Trust and cooperation: an experimental study of PD with choice of dependence

This study examined the relationship between cooperation and trust in interpersonal trust formation. Previous studies of trust and cooperation using prisoner's dilemma (PD) games failed, methodologically as well as conceptually, to distinguish the former from the latter. In response to the criticism on the use of iterated PD games, and for the purpose of investigating dynamic relationship between trust and cooperation, an improvement in research methodology was recently proposed: namely, PD with choice of dependence (PD/D; Kakiuchi & Yamagishi, 1997; Yamagishi & Kakiuchi, 2000). We conducted an experiment to compare formation of trust relations in PD and PD/D. Result indicated a higher level of cooperation in PD/D than in PD. Further analysis of strategies used to build trustful relationship, where two partners trusted each other and reciprocated the other's trusting behavior, revealed that participants in PD/D adopted "cautious and unconditional cooperation strategy" rather than TFT strategy.

Iterated Prisoner's Dilemma: Pay-off Variance

The iterated Prisoner's Dilemma (IPD) is usually analysed by evaluating arithmetic mean pay-offs in an ESS analysis. We consider several points that the standard argument does not address. Finite population size and finite numbers of matches in the IPD game lead us to consider both pay-off variance and the sampling process in the evolutionary game. We provide a general form for the pay-off variance of the Markov strategist in the IPD game, and present a general analysis of the initial invasion process of an “all defection strategist” (ALLD) into a “tit-for-tat” (TFT) strategist population by considering stochastic processes. Finite population size, strategic error an the variances of pay-offs alter the prediction concerning the initial invasion of ALLD compared with the standard Evolutionarily Stable Strategy (ESS) analysis. Even though TFT gets the larger arithmetic mean, the variance of its pay-off is also larger when the expected iterations of the game are sufficiently large. Therefore, the boundary of the parameter of the probability of game continuity,w, above which ALLD does not have advantage to invade into the TFT poulation, becomes a bit larger than predicted by the deterministic model.

Multiagent reinforcement learning in the Iterated Prisoner's Dilemma

Reinforcement learning (RL) is based on the idea that the tendency to produce an action should be strengthened (reinforced) if it produces favorable results, and weakened if it produces unfavorable results. Q-learning is a recent RL algorithm that does not need a model of its environment and can be used on-line. Therefore, it is well suited for use in repeated games against an unknown opponent. Most RL research has been confined to single-agent settings or to multiagent settings where the agents have totally positively correlated payoffs (team problems) or totally negatively correlated payoffs (zero-sum games). This paper is an empirical study of reinforcement learning in the Iterated Prisoner's Dilemma (IPD), where the agents' payoffs are neither totally positively nor totally negatively correlated. RL is considerably more difficult in such a domain. This paper investigates the ability of a variety of Q-learning agents to play the IPD game against an unknown opponent. In some experiments, the opponent is the fixed strategy Tit-For-Tat, while in others it is another Q-learner. All the Q-learners learned to play optimally against Tit-For-Tat. Playing against another learner was more difficult because the adaptation of the other learner created a non-stationary environment, and because the other learner was not endowed with any a priori knowledge about the IPD game such as a policy designed to encourage cooperation. The learners that were studied varied along three dimensions: the length of history they received as context, the type of memory they employed (lookup tables based on restricted history windows or recurrent neural networks that can theoretically store features from arbitrarily deep in the past), and the exploration schedule they followed. Although all the learners faced difficulties when playing against other learners, agents with longer history windows, lookup table memories, and longer exploration schedules fared best in the IPD games.

From genetic evolution to emergence of game strategies

Evolution of game strategies is studied in the Erroneous Iterated Prisoner's Dilemma game, in which a player sometimes decides on an erroneous action contrary to his own strategy. Erroneous games of this kind have been known to lead to the evolution of a variety of strategies. This paper describes genetic fusion modeling as a particular source of new strategies. Successive actions are chosen according to strategies having finite memory capacity, and strategy algorithms are elaborated by genetic fusion. Such a fusion process introduces a rhizome structure in a genealogy tree. Emergence of module strategies functions as innovative source of new strategies. How the extinction of strategies and module evolution leads to ESS-free open-ended evolution is also discussed.

Brand-Name Investment of Candidates and District Homogeneity: An Ordinal Response Model

Advertising is often viewed as a sunk investment that will receive a return only if firms do not cheat their customers on quality, much like a security bond that will be lost for non-performance of a contract. In the last period a firm will cheat if there is no future return from the investment [15]. Knowing that a firm is in its last period will then reduce sales in the last period to zero, so the firms will begin to cheat the period before the last. This is the usual iterated prisoner's dilemma (PD) game with a finite life; cheating begins in the first period and continues. Demsetz [10] argues that since firms are transferable through voluntary trade, the possibly infinite life of a firm transforms this finite, iterated PD game with a known end into an iterated PD game with no known end. As Axelrod [1] suggests, the best of the known strategies for this game is the titfor-tat strategy. In this game, firms live up to quality claims if customers continue to pay for their product. Demsetz also argues that brand name cannot serve as a barrier to entry for alienable firms because the firm, with its brand name intact, can be purchased by buyers who are more efficient producers than the current producer. Since professionals and politicians who have made investments in brand name have namebrand capital that is inalienable, these markets may suffer from the last-period problem [12; 17; 18; 19]. If the seller knows the last period because he intends to quit, cheating cannot be prevented