Memory-one Strategies Research Articles

Repeated prisoner's dilemma games in multi-player structured populations with crosstalk

Crosstalk represents a ubiquitous phenomenon in which strategic moves made in one game can influence actions in another. Reiter et al. [1] first integrated crosstalk with the repeated prisoner's dilemma model in structured populations. We extend their strategic space from one-step reactive strategies that consider only the actions of the opponent to memory-one strategies that consider the actions of both parties. We explore the cooperation rate of some representative memory-one strategies in repeated prisoner's dilemma games for multi-player structured populations when crosstalk exists, and focus on finding evolutionarily stable strategies that are cooperative in the memory-one strategy space under the impact of crosstalk. Simulation results in the strategic dynamics show that appropriately forgiving defection is conducive to cooperative stability in the population when crosstalk exists. Specifically, for strategies that can promote cooperation, there is a mutual restriction between tolerating the behavior of both defecting (the focused player and his opponent both defecting, DD) and forgiving unilateral defecting (the focused player cooperating while his opponent defecting, CD). That is, a more forgiving behavior of CD requires a lower tolerance for DD, while a higher tolerance of DD requires a less forgiving behavior of CD. Notably, we find two effective strategies, namely, Win-Stay-Lose-Shift (WSLS) and Firm‑but-Fair (FBF) in the facilitation of cooperation in game dynamics with crosstalk. Moreover, we reveal that the structured population can better suppress the negative impact of crosstalk on cooperation. This investigation provides complementary insights into the interplay between strategic decision-making and the influence of crosstalk within the context of memory-one strategies.

New memory-one strategies of the Iterated Prisoner’s Dilemma: a new framework to programmed human-AI interaction

With the rapid and continuing development of AI, the study of human-AI interaction is increasingly relevant. In this sense, we propose a reference framework to explore a model development in the context of social science to try to extract valuable information to the AI context. The model we choose was the Prisoner Dilemma using the Markov chain approach to study the evolution of memory-one strategies used in the Prisoner’s Dilemma in different agent-based simulation contexts using genetic algorithms programmed on the NetLogo environment. We developed the Multiplayer Prisoner’s Dilemma simulation from deterministic and probabilistic conditions, manipulating not only the probability of communication errors (noise) but also the probability of finding again the same agent. Our results suggest that the best strategies depend on the context of the game, following the rule: the lower the probability of finding the same agent again, the greater the chance of defect. Therefore, in environments with a low probability of interaction, the best strategies were the ‘Always Defect’ ones. But as the number of interactions increases, a different strategy emerges that is able to win Always Defect strategies, such as the Spiteful/grim. In addition, our results also highlight strategies that emerge in situations in which Spiteful/grim and Always Defect were banned. These are memory-one strategies with better performance than both TFT and PAVLOV under all conditions showing behaviors that are particularly deceiving but successful. The previously memory-one strategies for the Prisoner Dilemma represent a set of extensively tested strategies in contexts with different probability of encountering each other again and provide a framework for programming algorithms that interact with humans in PD-like trusted contexts.

Adaptive dynamics of memory-one strategies in the repeated donation game.

Human interactions can take the form of social dilemmas: collectively, people fare best if all cooperate but each individual is tempted to free ride. Social dilemmas can be resolved when individuals interact repeatedly. Repetition allows them to adopt reciprocal strategies which incentivize cooperation. The most basic model for direct reciprocity is the repeated donation game, a variant of the prisoner's dilemma. Two players interact over many rounds; in each round they decide whether to cooperate or to defect. Strategies take into account the history of the play. Memory-one strategies depend only on the previous round. Even though they are among the most elementary strategies of direct reciprocity, their evolutionary dynamics has been difficult to study analytically. As a result, much previous work has relied on simulations. Here, we derive and analyze their adaptive dynamics. We show that the four-dimensional space of memory-one strategies has an invariant three-dimensional subspace, generated by the memory-one counting strategies. Counting strategies record how many players cooperated in the previous round, without considering who cooperated. We give a partial characterization of adaptive dynamics for memory-one strategies and a full characterization for memory-one counting strategies.

Controlling Conditional Expectations by Zero-Determinant Strategies

Zero-determinant strategies are memory-one strategies in repeated games which unilaterally enforce linear relations between expected payoffs of players. Recently, the concept of zero-determinant strategies was extended to the class of memory-n strategies with nge 1, which enables more complicated control of payoffs by one player. However, what we can do by memory-n zero-determinant strategies is still not clear. Here, we show that memory-n zero-determinant strategies in repeated games can be used to control conditional expectations of payoffs. Equivalently, they can be used to control expected payoffs in biased ensembles, where a history of action profiles with large value of bias function is more weighted. Controlling conditional expectations of payoffs is useful for strengthening zero-determinant strategies, because players can choose conditions in such a way that only unfavorable action profiles to one player are contained in the conditions. We provide several examples of memory-n zero-determinant strategies in the repeated prisoner’s dilemma game. We also explain that a deformed version of zero-determinant strategies is easily extended to the memory-n case.

Necessary and Sufficient Condition for the Existence of Zero-Determinant Strategies in Repeated Games

Zero-determinant strategies are a class of memory-one strategies in repeated games which unilaterally enforce linear relationships between payoffs. It has long been unclear for what stage games zero-determinant strategies exist. We provide a necessary and sufficient condition for the existence of zero-determinant strategies. This condition can be interpreted as the existence of two different actions which unilaterally adjust the total value of a linear combination of payoffs. A relation between the class of stage games where zero-determinant strategies exist and other class of stage games is also provided.

Evolutionary dynamics of zero-determinant strategies in repeated multiplayer games

Several studies have confirmed the existence of zero-determinant (ZD) strategies in repeated social dilemmas since Press and Dyson’s ingenious discovery of ZD strategies in iterated prisoner’s dilemmas. However, less research studies evolutionary performance of multiplayer ZD strategies, especially from a theoretical perspective. Here, we use a state-clustering method to theoretically analyze evolutionary dynamics of two representative ZD strategies: generous ZD strategies and extortionate ZD strategies. We consider two new settings for multiplayer ZD strategies: competitions with all ZD strategies and competitions with all memory-one strategies, apart from the competitions between these strategies and some classical ones. Moreover, we investigate the influence of the level of generosity and extortion on evolutionary dynamics of generous and extortionate ZD strategies, which was commonly ignored in previous studies. Theoretical results show that players with limited generosity are at an advantageous place and extortioners extorting more severely hold their ground more readily. Our results may provide new insights into better understanding evolutionary dynamics of ZD strategies in repeated multiplayer games.

Probabilistic memory-one strategies to dominate the iterated prisoner’s dilemma over networks

The Iterated Prisoner’s Dilemma (IPD) has been a classical game theoretical scenario used to model behaviour interactions among agents. From the famous Axelrod’s tournament, and the successful results obtained by the Tit for Tat strategy, to the introduction of the zero-determinant strategies in the last decade, the game theory community has been exploring the performance of multiple strategies for years. This article grounds on such previous work, studying probabilistic memory-one strategies (PMO) and using evolutionary game theory, to analyse the criteria to find the most successful set of strategies in networked topologies. The results are nearly deterministic in discrete PMO scenarios. However, results become much more complex when moving to continuous ones, and there is no optimal strategy for a given scenario. Finally, this article describes how, using machine learning and evolutionary techniques; a cluster of agents, playing synchronously and adaptively, is able to dominate the rest of the population.

Inferring strategies from observations in long iterated Prisoner’s dilemma experiments

While many theoretical studies have revealed the strategies that could lead to and maintain cooperation in the Iterated Prisoner’s dilemma, less is known about what human participants actually do in this game and how strategies change when being confronted with anonymous partners in each round. Previous attempts used short experiments, made different assumptions of possible strategies, and led to very different conclusions. We present here two long treatments that differ in the partner matching strategy used, i.e. fixed or shuffled partners. Here we use unsupervised methods to cluster the players based on their actions and then Hidden Markov Model to infer what the memory-one strategies are in each cluster. Analysis of the inferred strategies reveals that fixed partner interaction leads to behavioral self-organization. Shuffled partners generate subgroups of memory-one strategies that remain entangled, apparently blocking the self-selection process that leads to fully cooperating participants in the fixed partner treatment. Analyzing the latter in more detail shows that AllC, AllD, TFT- and WSLS-like behavior can be observed. This study also reveals that long treatments are needed as experiments with less than 25 rounds capture mostly the learning phase participants go through in these kinds of experiments.

Exploring the inducement for social dilemma and cooperation promotion mechanisms in structured populations

Investigating and addressing the issue of social dilemma is a continuing concern within the subject of behavioral sciences and collective cooperation. The research to date has mostly focused on outside control methods dealing with the existing dilemma rather than exploring the reason resulting in that inside. This work probes into the inherent inducement for cooperation dilemma. We construct a gaming environment in which thousands of random memory-one strategies interact with each other on different complex topologies and record dominant strategies winning out from self-organized evolution. After clustering and analyzing evolutionary results, we find that the randomness and heterogeneity of population structure will strengthen the fitness of defectors. Besides, the cooperative willingness of individuals in betrayal situation has a decisive impact on the cooperation level of groups at equilibrium state. Further, we present two innovative treatments, enforcement and punishment mechanisms, to promote collective cooperation. The promoting effect is checked on various spatially networked topologies in the iterated prisoner's dilemma game. Simulation results indicate that irrespective of the underlying interaction network, the introduced promotion mechanisms are universally effective in subduing the evolutionary advantage of defectors, which favors the emergence and sustainability of cooperative behaviors. Our novel conclusions may provide some new perspectives for enhancing cooperation and elevating social welfare within biological systems and human societies.

Cooperation in alternating interactions with memory constraints

In repeated social interactions, individuals often employ reciprocal strategies to maintain cooperation. To explore the emergence of reciprocity, many theoretical models assume synchronized decision making. In each round, individuals decide simultaneously whether to cooperate or not. Yet many manifestations of reciprocity in nature are asynchronous. Individuals provide help at one time and receive help at another. Here, we explore such alternating games in which players take turns. We mathematically characterize all Nash equilibria among memory-one strategies. Moreover, we use evolutionary simulations to explore various model extensions, exploring the effect of discounted games, irregular alternation patterns, and higher memory. In all cases, we observe that mutual cooperation still evolves for a wide range of parameter values. However, compared to simultaneous games, alternating games require different strategies to maintain cooperation in noisy environments. Moreover, none of the respective strategies are evolutionarily stable.

Symmetric equilibrium of multi-agent reinforcement learning in repeated prisoner’s dilemma

We investigate the repeated prisoner’s dilemma game where both players alternately use reinforcement learning to obtain their optimal memory-one strategies. We theoretically solve the simultaneous Bellman optimality equations of reinforcement learning. We find that the Win-stay Lose-shift strategy, the Grim strategy, and the strategy which always defects can form symmetric equilibrium of the mutual reinforcement learning process amongst all deterministic memory-one strategies.

Using a theory of mind to find best responses to memory-one strategies

Memory-one strategies are a set of Iterated Prisoner’s Dilemma strategies that have been praised for their mathematical tractability and performance against single opponents. This manuscript investigates best response memory-one strategies with a theory of mind for their opponents. The results add to the literature that has shown that extortionate play is not always optimal by showing that optimal play is often not extortionate. They also provide evidence that memory-one strategies suffer from their limited memory in multi agent interactions and can be out performed by optimised strategies with longer memory. We have developed a theory that has allowed to explore the entire space of memory-one strategies. The framework presented is suitable to study memory-one strategies in the Prisoner’s Dilemma, but also in evolutionary processes such as the Moran process. Furthermore, results on the stability of defection in populations of memory-one strategies are also obtained.

Experiment research on the evolution of cooperation and network game theory

<p indent=0mm>Focusing on the mechanisms of the evolution and emergence of cooperative behavior, this paper reviews the recent development of direct reciprocity and network reciprocity from perspectives of both theoretical and experimental research based on two-player Prisoner’s Dilemma and multi-player Public Goods games. Repeated interactions provide the necessary condition for the emergence of direct reciprocity. The zero-determinant strategy, which is a new theory in direct reciprocity, has been deeply investigated from both theory and experiment. The zero-determinant strategy is a kind of memory-one strategies, which can unilaterally determine the payoff relationship in repeated games and may provide a novel tool to explore human decision behavior. Investigations on networked game experiments revealed that dynamic networks can facilitate and maintain human cooperation, while the influence of static networks on cooperation remains controversial. Furthermore, it is summarized that networked game experiments provide an efficient approach to studying the formation mechanism of network structure and the emergence of cooperation in complex systems.

Experimental criteria to identify efficient probabilistic memory-one strategies for the iterated prisoner’s dilemma

Many works and articles about probabilistic strategies for the prisoner’s dilemma have already been realised. Notably Press & Dyson 2012 article has lead to renewed interest in the subject. In this article, with the help of a systematic study of probabilistic memory-one strategies, we show that there is a basic criterion to configure and anticipate their success. This criterion, identified through the study of large homogeneous sets of strategies, is then compared to other similar criteria. Our experimental method has allowed us to discover new strategies that are efficient not only in probabilistic environments, but also in more general, probabilistic or non-probabilistic environments. We test the robustness of our results by various methods and compare the new strategies obtained with the best strategies currently known.

Reactive learning strategies for iterated games.

In an iterated game between two players, there is much interest in characterizing the set of feasible pay-offs for both players when one player uses a fixed strategy and the other player is free to switch. Such characterizations have led to extortionists, equalizers, partners and rivals. Most of those studies use memory-one strategies, which specify the probabilities to take actions depending on the outcome of the previous round. Here, we consider 'reactive learning strategies', which gradually modify their propensity to take certain actions based on past actions of the opponent. Every linear reactive learning strategy, p *, corresponds to a memory one-strategy, p , and vice versa. We prove that for evaluating the region of feasible pay-offs against a memory-one strategy, , we need to check its performance against at most 11 other strategies. Thus, is the convex hull in of at most 11 points. Furthermore, if p is a memory-one strategy, with feasible pay-off region , and p * is the corresponding reactive learning strategy, with feasible pay-off region , then is a subset of . Reactive learning strategies are therefore powerful tools in restricting the outcomes of iterated games.

Extortion Boosts Cooperation through Redistributing Strategies in Assortative Networked Systems

Networked evolutionary game theory is widely used to study the evolutionary dynamics in complex systems. In this work, we mainly investigate the influence of degree correlations of networks on the evolution of three memory-one strategies, i.e., unconditional cooperation, unconditional defection, and extortion. It is found that compared with the degree-uncorrelated network, assortative networks may inhibit the evolution of cooperation. The more assortative the network is, the less cooperation remains eventually. However, through micro dynamics analysis we reveal that by introducing extortion strategies, cooperation and defection are redistributed on the network, resulting in that cooperators can occupy the hub nodes while defectors survive among small-degree nodes. This work may provide potential clues to the coordination in multi-agent systems.

Chaos and unpredictability in evolution of cooperation in continuous time.

Cooperators benefit others with paying costs. Evolution of cooperation crucially depends on the cost-benefit ratio of cooperation, denoted as c. In this work, we investigate the infinitely repeated prisoner's dilemma for various values of c with four of the representative memory-one strategies, i.e., unconditional cooperation, unconditional defection, tit-for-tat, and win-stay-lose-shift. We consider replicator dynamics which deterministically describes how the fraction of each strategy evolves over time in an infinite-sized well-mixed population in the presence of implementation error and mutation among the four strategies. Our finding is that this three-dimensional continuous-time dynamics exhibits chaos through a bifurcation sequence similar to that of a logistic map as c varies. If mutation occurs with rate μ≪1, the position of the bifurcation sequence on the c axis is numerically found to scale as μ^{0.1}, and such sensitivity to μ suggests that mutation may have nonperturbative effects on evolutionary paths. It demonstrates how the microscopic randomness of the mutation process can be amplified to macroscopic unpredictability by evolutionary dynamics.

Evolutionary fate of memory-one strategies in repeated prisoner’s dilemma game in structured populations

We study evolutionary spatial prisoner’s dilemma game involving a one-step memory mechanism of the individuals whenever making strategy updating. In particular, during the process of strategy updating, each individual keeps in mind all the outcome of the action pairs adopted by himself and each of his neighbors in the last interaction, and according to which the individuals decide what actions they will take in the next round. Computer simulation results imply that win-stay-lose-shift like strategy win out of the memory-one strategy set in the stationary state. This result is robust in a large range of the payoff parameter, and does not depend on the initial state of the system. Furthermore, theoretical analysis with mean field and quasi-static approximation predict the same result. Thus, our studies suggest that win-stay-lose-shift like strategy is a stable dominant strategy in repeated prisoner’s dilemma game in homogeneous structured populations.

Good strategies for the Iterated Prisoner's Dilemma: Smale vs. Markov

In 1980 Steven Smale introduced a class of strategies for the Iterated Prisoner's Dilemma which used as data the running average of the previous payoff pairs. This approach is quite different from the Markov chain approach, common before and since, which used as data the outcome of the just previous play, the memory-one strategies. Our purpose here is to compare these two approaches focusing upon good strategies which, when used by a player, assure that the only way an opponent can obtain at least the cooperative payoff is to behave so that both players receive the cooperative payoff. In addition, we prove a version for the Smale approach of the so-called Folk Theorem concerning the existence of Nash equilibria in repeated play. We also consider the dynamics when certain simple Smale strategies are played against one another.

Partners or rivals? Strategies for the iterated prisoner's dilemma

Within the class of memory-one strategies for the iterated Prisoner's Dilemma, we characterize partner strategies, competitive strategies and zero-determinant strategies. If a player uses a partner strategy, both players can fairly share the social optimum; but a co-player preferring an unfair solution will be penalized by obtaining a reduced payoff. A player using a competitive strategy never obtains less than the co-player. A player using a zero-determinant strategy unilaterally enforces a linear relation between the two players' payoffs. These properties hold for every strategy used by the co-player, whether memory-one or not.