Iterated Prisoner's Dilemma Research Articles

Engineering Design of Strategies for Winning Iterated Prisoner's Dilemma Competitions

In this paper, we investigate winning strategies for round-robin iterated Prisoner's Dilemma (IPD) competitions and evolutionary IPD competitions. Since the outcome of a single competition depends on the composition of the population of participants, we propose a statistical evaluation methodology that takes into account outcomes across varying compositions. We run several series of competitions in which the strategies of the participants are randomly chosen from a set of representative strategies. Statistics are gathered to evaluate the performance of each strategy. With this approach, the conditions for some well-known strategies to win a round-robin IPD competition are analyzed. We show that a strategy that uses simple rule-based identification mechanisms to explore and exploit the opponent outperforms well-known strategies such as tit-for-tat (TFT) in most round-robin competitions. Group strategies have an advantage over nongroup strategies in evolutionary IPD competitions. Group strategies adopt different strategies in interacting with kin members and nonkin members. A simple group strategy, Clique, which cooperates only with kin members, performs well in competing against well-known IPD strategies. We introduce several group strategies developed by combining Clique with winning strategies from round-robin competitions and evaluate their performance by adapting three parameters: sole survivor rate, extinction rate, and survival time. Simulation results show that these group strategies outperform well-known IPD strategies in evolutionary IPD competitions.

Learning to cooperate in the Iterated Prisoner’s Dilemma by means of social attachments

AbstractThe Iterated Prisoner’s Dilemma (IPD) has been used as a paradigm for studying the emergence of cooperation among individual agents. Many computer experiments show that cooperation does arise under certain conditions. In particular, the spatial version of the IPD has been used and analyzed to understand the role of local interactions in the emergence and maintenance of cooperation. It is known that individual learning leads players to the Nash equilibrium of the game, which means that cooperation is not selected. Therefore, in this paper we propose that when players have social attachment, learning may lead to a certain rate of cooperation. We perform experiments where agents play the spatial IPD considering social relationships such as belonging to a hierarchy or to coalition. Results show that learners end up cooperating, especially when coalitions emerge.

Resolving the iterated prisoner’s dilemma: theory and reality

Pairs of unrelated individuals face a prisoner's dilemma if cooperation is the best mutual outcome, but each player does best to defect regardless of his partner's behaviour. Although mutual defection is the only evolutionarily stable strategy in one-shot games, cooperative solutions based on reciprocity can emerge in iterated games. Among the most prominent theoretical solutions are the so-called bookkeeping strategies, such as tit-for-tat, where individuals copy their partner's behaviour in the previous round. However, the lack of empirical data conforming to predicted strategies has prompted the suggestion that the iterated prisoner's dilemma (IPD) is neither a useful nor realistic basis for investigating cooperation. Here, we discuss several recent studies where authors have used the IPD framework to interpret their data. We evaluate the validity of their approach and highlight the diversity of proposed solutions. Strategies based on precise accounting are relatively uncommon, perhaps because the full set of assumptions of the IPD model are rarely satisfied. Instead, animals use a diverse array of strategies that apparently promote cooperation, despite the temptation to cheat. These include both positive and negative reciprocity, as well as long-term mutual investments based on 'friendships'. Although there are various gaps in these studies that remain to be filled, we argue that in most cases, individuals could theoretically benefit from cheating and that cooperation cannot therefore be explained with the concept of positive pseudo-reciprocity. We suggest that by incorporating empirical data into the theoretical framework, we may gain fundamental new insights into the evolution of mutual reciprocal investment in nature.

Evolution of Strategies With Different Representation Schemes in a Spatial Iterated Prisoner's Dilemma Game

The iterated prisoner's dilemma (IPD) game has been frequently used to examine the evolution of cooperative behavior among agents in the field of evolutionary computation. It has been demonstrated that various factors are related to the evolution of cooperative behavior. One well-known factor is spatial relations among agents. The IPD game is often played in a 2-D grid world. Such a spatial IPD game has a neighborhood structure, which is used to choose opponents for the IPD game and parents for genetic operations. Another important factor is the choice of a representation scheme to encode the strategy of each agent. Different representation schemes often lead to different results. Whereas the choice of a representation scheme is known to be important, a mixture of different representation schemes has not been examined for the spatial IPD game in the literature. That is, a population of homogeneous agents with the same representation scheme has been usually assumed in the literature. In this paper, we introduce the use of different representation schemes in a single population to the spatial IPD game in order to examine the evolution of cooperative behavior under more general assumptions. With the use of different representation schemes, we can examine the evolution of cooperative behavior in various settings such as partial interaction through the IPD game, partial interaction through crossover, full interaction through the IPD game and crossover, and no interaction between different subpopulations of agents.

Spiking Neural Networks with Different Reinforcement Learning (RL) Schemes in a Multiagent Setting

This paper investigates the effectiveness of spiking agents when trained with reinforcement learning (RL) in a challenging multiagent task. In particular, it explores learning through reward-modulated spike-timing dependent plasticity (STDP) and compares it to reinforcement of stochastic synaptic transmission in the general-sum game of the Iterated Prisoner's Dilemma (IPD). More specifically, a computational model is developed where we implement two spiking neural networks as two "selfish" agents learning simultaneously but independently, competing in the IPD game. The purpose of our system (or collective) is to maximise its accumulated reward in the presence of reward-driven competing agents within the collective. This can only be achieved when the agents engage in a behaviour of mutual cooperation during the IPD. Previously, we successfully applied reinforcement of stochastic synaptic transmission to the IPD game. The current study utilises reward-modulated STDP with eligibility trace and results show that the system managed to exhibit the desired behaviour by establishing mutual cooperation between the agents. It is noted that the cooperative outcome was attained after a relatively short learning period which enhanced the accumulation of reward by the system. As in our previous implementation, the successful application of the learning algorithm to the IPD becomes possible only after we extended it with additional global reinforcement signals in order to enhance competition at the neuronal level. Moreover it is also shown that learning is enhanced (as indicated by an increased IPD cooperative outcome) through: (i) strong memory for each agent (regulated by a high eligibility trace time constant) and (ii) firing irregularity produced by equipping the agents' LIF neurons with a partial somatic reset mechanism.

EMOTIONS AND EMPATHY: A BRIDGE BETWEEN NATURE AND SOCIETY?

For over a decade neuroscience has uncovered that appropriate decision-making in daily life decisions results from a strong interplay between cognition and covert biases produced by emotional processes. This interplay is particularly important in social contexts: lesions in the pathways supporting these processes provoke serious impairments on social behavior. One important mechanism in social contexts is empathy, fundamental for appropriate social behavior. This paper presents arguments supporting this connection between cognition and emotion, in individual as well as in social contexts. The central claim of this paper is that biologically inspired cognitive architectures ought to include these mechanisms. A taxonomy of computational models addressing emotions is presented, together with a brief survey of the research published in this area. The Prisoner Dilemma game is used as a case study exposing the trade-off between individual rationality and cooperative behavior. Experiments using a simple implementation of empathy and emotion expression, employing an Iterated Prisoner Dilemma setup, illustrate the emergence of a cooperative behavior mutually beneficial for both players.

Critical Dynamics in the Evolution of Stochastic Strategies for the Iterated Prisoner's Dilemma

The observed cooperation on the level of genes, cells, tissues, and individuals has been the object of intense study by evolutionary biologists, mainly because cooperation often flourishes in biological systems in apparent contradiction to the selfish goal of survival inherent in Darwinian evolution. In order to resolve this paradox, evolutionary game theory has focused on the Prisoner's Dilemma (PD), which incorporates the essence of this conflict. Here, we encode strategies for the iterated Prisoner's Dilemma (IPD) in terms of conditional probabilities that represent the response of decision pathways given previous plays. We find that if these stochastic strategies are encoded as genes that undergo Darwinian evolution, the environmental conditions that the strategies are adapting to determine the fixed point of the evolutionary trajectory, which could be either cooperation or defection. A transition between cooperative and defective attractors occurs as a function of different parameters such as mutation rate, replacement rate, and memory, all of which affect a player's ability to predict an opponent's behavior. These results imply that in populations of players that can use previous decisions to plan future ones, cooperation depends critically on whether the players can rely on facing the same strategies that they have adapted to. Defection, on the other hand, is the optimal adaptive response in environments that change so quickly that the information gathered from previous plays cannot usefully be integrated for a response.

Fault-tolerant strategies in the Iterated Prisoner's Dilemma

The aim of this paper is to determine how well can strategies in the Iterated Prisoner's Dilemma (IPD) behave in the presence of errors. A highly desirable property of strategies in IPD, even without errors, is their robustness. A strategy σ is called robust, if in any population consisting of copies of two types of strategies, σ itself and some other strategy τ, the strategy σ is never worse than τ. For example, the well known simple strategy tit-for-tat is robust. However, tit-for-tat is very vulnerable to errors: already one mistake in its execution can cause it to lose the property of robustness. The ultimate resistance to errors is captured by the notion of perfect fault-tolerance: a strategy has this property, if any distortion of it on an arbitrary finite number of moves remains robust. Do there exist perfectly fault-tolerant strategies in IPD? Somewhat surprisingly, the answer is yes: indeed we describe such a strategy and we prove that it has this very strong resistance to errors. However, we prove that no strategy with bounded memory can have this property. We then go on to show that preserving robustness under distortions on any fixed initial segment of moves is possible with bounded memory.

Fluctuation in Human Subjects Iterated Prisoner’s Dilemma Games

How to examine a behavior model in social computation is a long existed problem. By data mining, characters of the fluctuation in human subjects iterated prisoner’s dilemma (IPD) experiment are studied. Fluctuation is the individual strategy changing. In IPD, fluctuation of a population is the average of the absolute value of strategy changing of individual over the whole population of an experimental session, and fluctuation of a group is over a group, and so on. Therefore, it’s natural to assume that there is a certain relationship between magnitude of fluctuation and deviation from equilibrium, namely, the more deviation from equilibrium, the more fluctuation. Here, the relationship is studied quantitatively. Employing row data from two recently articles (Duffy and Ochs [Game and Economics Behavior (2009)] and Yang, et al [Experimental Economics (2007)], we examined the above-mentioned relationship, the characters of the fluctuation near the fixed point limit and whole strategy range. In this note, the function of fluctuation is shown in three cases. (1) either at the session’s level, or at the subgame’s level, the average magnitude of fluctuation is positively and precisely related to the deviation from equilibrium; (2) the fluctuation can be quantitatively captured by random impulse model when a society is very close to pure Nash equilibrium point; (3) fluctuation can be used as a criterion for evaluating theoretical models. To sum up, the advantage of the fluctuation measurement is shown. This finding might be a useful judgment for the behavior modeling for social computations.

Evolution of cooperative behavior among heterogeneous agents with different strategy representations in an iterated prisoner’s dilemma game

The iterated prisoner’s dilemma (IPD) game has frequently been used to examine the evolution of cooperative behavior among agents. When the effect of representation schemes of IPD game strategies was examined, the same representation scheme was usually assigned to all agents. That is, in the literature, a population of homogeneous agents was usually used in computational experiments. In this article, we focus on a slightly different situation where every agent does not necessarily use the same representation scheme. That is, a population can be a mixture of heterogeneous agents with different representation schemes. In computational experiments, we used binary strings of different lengths (i.e., three-bit and five-bit strings) to represent IPD game strategies. We examined the evolution of cooperative behavior among heterogeneous agents in comparison with the case of homogeneous ones for the standard IPD game with typical payoff values of 0, 1, 3, and 5. Experimental results showed that the evolution of cooperative behavior was slowed down by the use of heterogeneous agents. It was also demonstrated that a faster evolution of cooperative behavior is achieved among majority agents than among minority ones in a heterogeneous population.

Self-control with spiking and non-spiking neural networks playing games

Self- control can be defined as choosing a large delayed reward over a small immediate reward, while precommitment is the making of a choice with the specific aim of denying oneself future choices. Humans recognise that they have self-control problems and attempt to overcome them by applying precommitment. Problems in exercising self-control, suggest a conflict between cognition and motivation, which has been linked to competition between higher and lower brain functions (representing the frontal lobes and the limbic system respectively). This premise of an internal process conflict, lead to a behavioural model being proposed, based on which, we implemented a computational model for studying and explaining self-control through precommitment behaviour. Our model consists of two neural networks, initially non-spiking and then spiking ones, representing the higher and lower brain systems viewed as cooperating for the benefit of the organism. The non-spiking neural networks are of simple feed forward multilayer type with reinforcement learning, one with selective bootstrap weight update rule, which is seen as myopic, representing the lower brain and the other with the temporal difference weight update rule, which is seen as far-sighted, representing the higher brain. The spiking neural networks are implemented with leaky integrate-and-fire neurons with learning based on stochastic synaptic transmission. The differentiating element between the two brain centres in this implementation is based on the memory of past actions determined by an eligibility trace time constant. As the structure of the self-control problem can be likened to the Iterated Prisoner’s Dilemma (IPD) game in that cooperation is to defection what self-control is to impulsiveness or what compromising is to insisting, we implemented the neural networks as two players, learning simultaneously but independently, competing in the IPD game. With a technique resembling the precommitment effect, whereby the payoffs for the dilemma cases in the IPD payoff matrix are differentially biased (increased or decreased), it is shown that increasing the precommitment effect (through increasing the differential bias) increases the probability of cooperating with oneself in the future, irrespective of whether the implementation is with spiking or non-spiking neural networks.

Spectrum sharing in iterated Prisoner’s Dilemma game based on evolutionary strategies for Cognitive Radios

We study a spectrum sharing problem where multiple systems coexist and interfere with each other. First, an analysis is proposed for distributed spectrum sharing based on Prisoners’ Dilemma (PD) in Cognitive Radios (CRs). In one-shot game, selfish and rational CRs greedily full spread their own spectrum space in order to maximize their own rates, which leads to Nash Equilibrium (N.E.). But with long term interaction, i.e., Iterated Prisoner’s Dilemma (IPD), CRs can come to cooperate and acquire the social optimal point by using different evolutionary strategies such as Tit For Tat (TFT), Generous TFT (GTFT), etc. Also we compare the performances of the different evolutionary strategies in noise-free and noisy environments for two-player games. Finally, N-player IPD (N-IPD) is simulated to verify our conclusions that TFT is a good strategy for spectrum sharing in CRs.

Relationship Between Generalization and Diversity in Coevolutionary Learning

Games have long played an important role in the development and understanding of coevolutionary learning systems. In particular, the search process in coevolutionary learning is guided by strategic interactions between solutions in the population, which can be naturally framed as game playing. We study two important issues in coevolutionary learning - generalization performance and diversity - using games. The first one is concerned with the coevolutionary learning of strategies with high generalization performance, that is, strategies that can outperform against a large number of test strategies (opponents) that may not have been seen during coevolution. The second one is concerned with diversity levels in the population that may lead to the search of strategies with poor generalization performance. It is not known if there is a relationship between generalization and diversity in coevolutionary learning. This paper investigates whether there is such a relationship in coevolutionary learning through a detailed empirical study. We systematically investigate the impact of various diversity maintenance approaches on the generalization performance of coevolutionary learning quantitatively using case studies. The problem of the iterated prisoner's dilemma (IPD) game is considered. Unlike past studies, we can measure both the generalization performance and the diversity level of the population of evolved strategies. Results from our case studies show that the introduction and maintenance of diversity do not necessarily lead to the coevolutionary learning of strategies with high generalization performance. However, if individual strategies can be combined (e.g., using a gating mechanism), there is the potential of exploiting diversity in coevolutionary learning to improve generalization performance. Specifically, when the introduction and maintenance of diversity lead to a speciated population during coevolution, where each specialist strategy is capable of outperforming different opponents, the population as a whole can have a significantly higher generalization performance compared to individual strategies.

Optimal Strategies of the Iterated Prisoner's Dilemma Problem for Multiple Conflicting Objectives

In this paper, we present a new paradigm of searching optimal strategies in the game of iterated prisoner's dilemma (IPD) using multiple-objective evolutionary algorithms. This method is more useful than the existing approaches, because it not only produces strategies that perform better in the iterated game but also finds a family of nondominated strategies, which can be analyzed to decipher properties a strategy should have to win the game in a more satisfactory manner. We present the results obtained by this new method and discuss sub-strategies found to be common among nondominated strategies. The multiobjective treatment of the IPD problem demonstrated here can be applied to other similar game-playing tasks.

Fingerprint Analysis of the Noisy Prisoner's Dilemma Using a Finite-State Representation

Fingerprinting is a technique that permits automatic classification of strategies for playing a game. In this paper, the evolution of strategies for playing the iterated prisoner's dilemma (IPD) at three different noise levels is analyzed using fingerprinting and other techniques including a novel quantity, evolutionary velocity, derived from fingerprinting. The results are at odds with initial expectations and permit the detection of a critical difference in the evolution of agents with and without noise. Noise during fitness evaluation places a larger fraction of an agent's genome under selective pressure, resulting in substantially more efficient training. In this case, efficiency is the production of superior competitive ability at a lower evolutionary velocity. Prisoner's dilemma playing agents are evolved for 6400 generations, taking samples at eight exponentially spaced epochs. This permits assessment of the change in populations over long evolutionary time. Agents are evaluated for competitive ability between those evolved for different lengths of time and between those evolved using distinct noise levels. The presence of noise during agent training is found to convey a commanding competitive advantage. A novel analysis is done in which a tournament is run with no two agents from the same evolutionary line and one third of agents from each noise level studied. This analysis simulates contributed agent tournaments without any genetic relation between agents. It is found that in early epochs the agents evolved without noise have the best average tournament rank, but that in later epochs they have the worst.

Evolution and Incremental Learning in the Iterated Prisoner's Dilemma

This paper examines the comparative performance and adaptability of evolutionary, learning, and memetic strategies to different environment settings in the iterated prisoner's dilemma (IPD). A memetic adaptation framework is developed for IPD strategies to exploit the complementary features of evolution and learning. In the paradigm, learning serves as a form of directed search to guide evolving strategies to attain eventual convergence towards good strategy traits, while evolution helps to minimize disparity in performance among learning strategies. Furthermore, a double-loop incremental learning scheme (ILS) that incorporates a classification component, probabilistic update of strategies and a feedback learning mechanism is proposed and incorporated into the evolutionary process. A series of simulation results verify that the two techniques, when employed together, are able to complement each other's strengths and compensate for each other's weaknesses, leading to the formation of strategies that will adapt and thrive well in complex, dynamic environments.

Parasites may help stabilize cooperative relationships.

BackgroundThe persistence of cooperative relationships is an evolutionary paradox; selection should favor those individuals that exploit their partners (cheating), resulting in the breakdown of cooperation over evolutionary time. Our current understanding of the evolutionary stability of mutualisms (cooperation between species) is strongly shaped by the view that they are often maintained by partners having mechanisms to avoid or retaliate against exploitation by cheaters. In contrast, we empirically and theoretically examine how additional symbionts, specifically specialized parasites, potentially influence the stability of bipartite mutualistic associations. In our empirical work we focus on the obligate mutualism between fungus-growing ants and the fungi they cultivate for food. This mutualism is exploited by specialized microfungal parasites (genus Escovopsis) that infect the ant's fungal gardens. Using sub-colonies of fungus-growing ants, we investigate the interactions between the fungus garden parasite and cooperative and experimentally-enforced uncooperative ("cheating") pairs of ants and fungi. To further examine if parasites have the potential to help stabilize some mutualisms we conduct Iterative Prisoner's Dilemma (IPD) simulations, a common framework for predicting the outcomes of cooperative/non-cooperative interactions, which incorporate parasitism as an additional factor.ResultsIn our empirical work employing sub-colonies of fungus-growing ants, we found that Escovopsis-infected sub-colonies composed of cheating populations of ants or fungi lost significantly more garden biomass than sub-colonies subjected to infection or cheating (ants or fungi) alone. Since the loss of fungus garden compromises the fitness of both mutualists, our findings suggest that the potential benefit received by the ants or fungi for cheating is outweighed by the increased concomitant cost of parasitism engendered by non-cooperation (cheating). IPD simulations support our empirical results by confirming that a purely cooperative strategy, which is unsuccessful in the classic IPD model, becomes stable when parasites are included.ConclusionHere we suggest, and provide evidence for, parasitism being an external force that has the potential to help stabilize cooperation by aligning the selfish interests of cooperative partners in opposition to a common enemy. Specifically, our empirical results and IPD simulations suggest that when two mutualists share a common enemy selection can favor cooperation over cheating, which may help explain the evolutionary stability of some mutualisms.

Measuring Generalization Performance in Coevolutionary Learning

Coevolutionary learning involves a training process where training samples are instances of solutions that interact strategically to guide the evolutionary (learning) process. One main research issue is with the generalization performance, i.e., the search for solutions (e.g., input-output mappings) that best predict the required output for any new input that has not been seen during the evolutionary process. However, there is currently no such framework for determining the generalization performance in coevolutionary learning even though the notion of generalization is well-understood in machine learning. In this paper, we introduce a theoretical framework to address this research issue. We present the framework in terms of game-playing although our results are more general. Here, a strategy's generalization performance is its average performance against all test strategies. Given that the true value may not be determined by solving analytically a closed-form formula and is computationally prohibitive, we propose an estimation procedure that computes the average performance against a small sample of random test strategies instead. We perform a mathematical analysis to provide a statistical claim on the accuracy of our estimation procedure, which can be further improved by performing a second estimation on the variance of the random variable. For game-playing, it is well-known that one is more interested in the generalization performance against a biased and diverse sample of good test strategies. We introduce a simple approach to obtain such a test sample through the multiple partial enumerative search of the strategy space that does not require human expertise and is generally applicable to a wide range of domains. We illustrate the generalization framework on the coevolutionary learning of the iterated prisoner's dilemma (IPD) games. We investigate two definitions of generalization performance for the IPD game based on different performance criteria, e.g., in terms of the number of wins based on individual outcomes and in terms of average payoff. We show that a small sample of test strategies can be used to estimate the generalization performance. We also show that the generalization performance using a biased and diverse set of good test strategies is lower compared to the unbiased case for the IPD game. This is the first time that generalization is defined and analyzed rigorously in coevolutionary learning. The framework allows the evaluation of the generalization performance of any coevolutionary learning system quantitatively.

The emergence of cooperation in the random asynchronous prisoner’s dilemma

The iterated prisoners dilemma (IPD) is a simple model for the study of the emergence of cooperative behavior in populations of selfish individuals. In this work, we challenge the assumption that players move in synchrony, and develop a general Markovian model that allows the study of a wide spectrum of scenarios. Simulations show that the relative timing of player moves, and the reward for mutual cooperation, influences the strategy that eventually dominates the final population. For a synchronous environment, reciprocal behavior appears to be the key to the evolution of cooperation, while in an asynchronous environment, guarded generosity may be a route to the evolution of cooperation.

Multiple Choices and Reputation in Multiagent Interactions

Coevolutionary learning provides a framework for modeling more realistic iterated prisoner's dilemma (IPD) interactions and to study conditions of how and why certain behaviors (e.g., cooperation) in a complex environment can be learned through an adaptation process guided by strategic interactions. The coevolutionary learning of cooperative behaviors can be attributed to the mechanism of direct reciprocity (e.g., repeated encounters). However, for the more complex IPD game with more choices, it is unknown precisely why the mechanism of direct reciprocity is less effective in promoting the learning of cooperative behaviors. Here, our study suggests that the evolution of defection may be a result of strategies effectively having more opportunities to exploit others when there are more choices. We note that strategies are less able to resolve the intention of an intermediate choice, e.g., whether it is a signal to engender further cooperation or a subtle exploitation. A likely consequence is strategies adapting to lower cooperation plays that offer higher payoffs in the short-term view when they cannot resolve the intention of opponents. However, cooperation in complex human interactions may also involve indirect interactions rather than direct interactions only. Following this, we study the coevolutionary learning of IPD with more choices and reputation. Here, current behavioral interactions depend not only on choices made in previous moves (direct interactions), but also choices made in past interactions that are reflected by their reputation scores (indirect interactions). The coevolutionary learning of cooperative behaviors is possible in the IPD with more choices when strategies use reputation as a mechanism to estimate behaviors of future partners and to elicit mutual cooperation play right from the start of interactions. In addition, we study the impact of the accuracy of reputation estimation in reflecting strategy behaviors of different implementations and why it is important for the evolution of cooperation. We show that the accuracy is related to how memory of games from previous generations is incorporated to calculate reputation scores and how frequently reputation scores are updated.