Players Update Research Articles

Errors can increase cooperation in finite populations

I use an evolutionary game to investigate how the level of noise influences cooperation and efficiency in a dynamic setting. Players choose strategies to play indefinitely repeated prisoner's dilemmas; the strategies are represented by finite automata, and complexity costs are imposed. Players update their strategies based on the successfulness of the strategies. Using both theoretical analysis and computational experiments, I show that the presence of noise dramatically changes the system dynamics. The effect of noise interacts with the benefit of cooperation: noise can increase cooperation, but only when its level is low and the benefit of cooperation is high. In the noise-free environment, I observe constant oscillations between cooperation and defection. In contrast, the presence of noise makes Win-Stay Lose-Shift (WSLS) a successful strategy when the benefit of cooperation is sufficiently high, making cooperation relatively stable and leading to an efficient outcome.

Bayesian Reinforcement Learning-Based Coalition Formation for Distributed Resource Sharing by Device-to-Device Users in Heterogeneous Cellular Networks

This paper investigates the problem of distributed resource sharing in a device-to-device enabled heterogeneous network, where the various device pairs choose their transmission channels, modes, base stations (BSs), and power levels without any control by the BSs based only on the locally-observable information. This problem is represented as a Bayesian coalition formation game, where the players (device pairs) create coalitions to maximize their long-term rewards with no prior knowledge of the values of potential coalitions and the types of their members. To minimize these uncertainties, a novel Bayesian reinforcement learning (RL) model is derived. In this model, the players update (through repeated coalition formation) their beliefs about the types and coalitional values to reach a stable coalitional agreement. The proposed Bayesian RL-based coalition formation algorithms are implemented in a long-term evolution advanced network and evaluated using simulations. The algorithms show a superior performance when compared with other relevant resource allocation schemes and achieve near-optimal results after a relatively small number of RL iterations.

Power-law distributed temporal heterogeneity of human activities promotes cooperation on complex networks

An evolutionary prisoner’s dilemma game (PDG) with players located on Barabási–Albert scale-free networks is studied. The impact of players’ heterogeneous temporal activity pattern on the evolution of cooperation is investigated. To this end, the normal procedure that players update their strategies immediately after a round of game is discarded. Instead, players update strategies according to their assigned reproduction time, which follows a power-law distribution. We find that the temporal heterogeneity of players’ activities facilitates the prosperity of cooperation, indicating the important role of hubs in the maintenance of cooperation on scale-free networks. When the reproduction time is assigned to individuals negatively related to their degrees, a fluctuation of the cooperation level with the increase of the exponent β is observed.

Bribe and Punishment: An Evolutionary Game-Theoretic Analysis of Bribery.

Harassment bribes, paid by citizens to corrupt officers for services the former are legally entitled to, constitute one of the most widespread forms of corruption in many countries. Nation states have adopted different policies to address this form of corruption. While some countries make both the bribe giver and the bribe taker equally liable for the crime, others impose a larger penalty on corrupt officers. We examine the consequences of asymmetric and symmetric penalties by developing deterministic and stochastic evolutionary game-theoretic models of bribery. We find that the asymmetric penalty scheme can lead to a reduction in incidents of bribery. However, the extent of reduction depends on how the players update their strategies over time. If the interacting members change their strategies with a probability proportional to the payoff of the alternative strategy option, the reduction in incidents of bribery is less pronounced. Our results indicate that changing from a symmetric to an asymmetric penalty scheme may not suffice in achieving significant reductions in incidents of harassment bribery.

Crucial role of strategy updating for coexistence of strategies in interaction networks.

Network models are useful tools for studying the dynamics of social interactions in a structured population. After a round of interactions with the players in their local neighborhood, players update their strategy based on the comparison of their own payoff with the payoff of one of their neighbors. Here we show that the assumptions made on strategy updating are of crucial importance for the strategy dynamics. In the first step, we demonstrate that seemingly small deviations from the standard assumptions on updating have major implications for the evolutionary outcome of two cooperation games: cooperation can more easily persist in a Prisoner's Dilemma game, while it can go more easily extinct in a Snowdrift game. To explain these outcomes, we develop a general model for the updating of states in a network that allows us to derive conditions for the steady-state coexistence of states (or strategies). The analysis reveals that coexistence crucially depends on the number of agents consulted for updating. We conclude that updating rules are as important for evolution on a network as network structure and the nature of the interaction.

Dynamics to equilibrium in network games: individual behavior and global response.

Various social contexts can be depicted as games of strategic interactions on networks, where an individual’s welfare depends on both her and her partners’ actions. Whereas much attention has been devoted to Bayes-Nash equilibria in such games, here we look at strategic interactions from an evolutionary perspective. To this end, we present the results of a numerical simulations program for these games, which allows us to find out whether Nash equilibria are accessible by adaptation of player strategies, and in general to identify the attractors of the evolution. Simulations allow us to go beyond a global characterization of the cooperativeness at equilibrium and probe into individual behavior. We find that when players imitate each other, evolution does not reach Nash equilibria and, worse, leads to very unfavorable states in terms of welfare. On the contrary, when players update their behavior rationally, they self-organize into a rich variety of Nash equilibria, where individual behavior and payoffs are shaped by the nature of the game, the social network’s structure and the players’ position within the network. Our results allow to assess the validity of mean-field approaches we use to describe the dynamics of these games. Interestingly, our dynamically-found equilibria generally do not coincide with (but show qualitatively the same features of) those resulting from theoretical predictions in the context of one-shot games under incomplete information.

Online Learning of Nash Equilibria in Congestion Games

We study the repeated, nonatomic congestion game, in which multiple populations of players share resources and make, at each iteration, a decentralized decision on which resources to utilize. We investigate the following question: given a model of how individual players update their strategies, does the resulting dynamics of strategy profiles converge to the set of Nash equilibria of the one-shot game? We consider in particular a model in which players update their strategies using algorithms with sublinear discounted regret. We show that the resulting sequence of strategy profiles converges to the set of Nash equilibria in the sense of Cesaro means. However, convergence of the actual sequence is not guaranteed in general. We show that it can be guaranteed for a class of algorithms with a sublinear discounted regret and which satisfy an additional condition. We call such algorithms AREP (approximate replicator) algorithms, as they can be interpreted as a discrete-time approximation of the replicator equat...

Evolutionary games on the lattice: best-response dynamics

The best-response dynamics is an example of an evolutionary game where players update their strategy in order to maximize their payoff. The main objective of this paper is to study a stochastic spatial version of this game based on the framework of interacting particle systems in which players are located on an infinite square lattice. In the presence of two strategies, and calling a strategy selfish or altruistic depending on a certain ordering of the coefficients of the underlying payoff matrix, a simple analysis of the nonspatial mean-field approximation of the spatial model shows that a strategy is evolutionary stable if and only if it is selfish, making the system bistable when both strategies are selfish. The spatial and nonspatial models agree when at least one strategy is altruistic. In contrast, we prove that in the presence of two selfish strategies and in any spatial dimension, only the most selfish strategy remains evolutionary stable. The main ingredients of the proof are monotonicity results and a coupling between the best-response dynamics properly rescaled in space with bootstrap percolation to compare the infinite time limits of both systems.

Near-Potential Games

Potential games are a special class of games for which many adaptive user dynamics converge to a Nash equilibrium. In this article, we study properties of near-potential games, that is, games that are close in terms of payoffs to potential games, and show that such games admit similar limiting dynamics. We first focus on finite games in strategic form. We introduce a distance notion in the space of games and study the geometry of potential games and sets of games that are equivalent, with respect to various equivalence relations, to potential games. We discuss how, given an arbitrary game, one can find a nearby game in these sets. We then study dynamics in near-potential games by focusing on continuous-time perturbed best response dynamics. We characterize the limiting behavior of this dynamics in terms of the upper contour sets of the potential function of a close potential game and approximate equilibria of the game. Exploiting structural properties of approximate equilibrium sets, we strengthen our result and show that for games that are sufficiently close to a potential game, the sequence of mixed strategies generated by this dynamics converges to a small neighborhood of equilibria whose size is a function of the distance from the set of potential games. In the second part of the article, we study continuous games and show that our approach for characterizing the limiting sets in near-potential games extends to continuous games. In particular, we consider continuous-time best response dynamics and a variant of it (where players update their strategies only if there is at least ε utility improvement opportunity) in near-potential games where the strategy sets are compact and convex subsets of a Euclidean space. We show that these update rules converge to a neighborhood of equilibria (or the maximizer of the potential function), provided that the potential function of the nearby potential game satisfies some structural properties. Our results generalize the known convergence results for potential games to near-potential games.

Aspiration-Based Risk Preference in Public Goods Game on Scale-free Networks

The public goods game on complex networks is being studied as a model for the evolution of cooperation. In the evolutionary games, players update their strategies in accordance with certain rules. This paper presents a new learning rule, in keeping with player's aspiration-based risk preference. A definition of risk preference factor is also proposed to describe the aspiration level. Our new model proves that a player will potentially take more dominant strategy if the payoff is below expectation. Our sufficient experiments and the simulation results on scale-free network indicate that our new strategy-updating rule affects the evolution of cooperation. For small multiplication factors, the frequency of cooperation increases as the aspiration level increases. For large multiplication factors, intermediate levels of aspiration prove to be optimal for the successful evolution of public cooperation.

Prisoner's dilemma game on adaptive networks under limited foresight

We study the emergence of cooperation in an environment where players in prisoner's dilemma game (PDG) not only update their strategies but also change their interaction relations. Different from previous studies in which players update their strategies according to the imitation rule, in this article, the strategies are updated with limited foresight. We find that two absorbing states—full cooperation and full defection—can be reached, assuming that players can delete interaction relations unilaterally, but new relations can only be created with the mutual consent of both partners. Simulation experiments show that high levels of cooperation in large populations can be achieved when the temptation to defect in PDG is low. Moreover, we explore the factors which influence the level of cooperation. These results provide new insights into the cooperation in social dilemma and into corresponding control strategies. © 2012Wiley Periodicals, Inc. Complexity, 2012

Simultaneous policy update algorithms for learning the solution of linear continuous-time H∞ state feedback control

It is well known that the H∞ state feedback control problem can be viewed as a two-player zero-sum game and reduced to find a solution of the algebra Riccati equation (ARE). In this paper, we propose a simultaneous policy update algorithm (SPUA) for solving the ARE, and develop offline and online versions. The offline SPUA is a model-based approach, which obtains the solution of the ARE by solving a sequence of Lyapunov equations (LEs). Its convergence is established rigorously by constructing a Newton’s sequence for the fixed point equation. The online SPUA is a partially model-free approach, which takes advantage of the thought of reinforcement learning (RL) to learn the solution of the ARE online without requiring the internal system dynamics, wherein both players update their action policies simultaneously. The convergence of the online SPUA is proved by demonstrating that it is mathematically equivalent to the offline SPUA. Finally, by conducting comparative simulation studies on an F-16 aircraft plant and a power system, the results show that both the offline SPUA and the online SPUA can find the solution of the ARE, and achieve much better convergence than the existing methods.

Evolution of cooperation on adaptively weighted networks

Recent studies have explored interactions between evolutionary game dynamics and population structure. Yet most studies so far mainly paid attention to unweighted and static networks. Here we explore evolutionary games played on dynamically weighted networks. Players update their strategies according to the payoffs they obtain. Players also update weights of their adjacent links depending on payoffs they gain through those links; profitable links are reinforced whereas unprofitable ones are weakened. The system is characterized by two time scales, the one for strategy update, β S , and the other for weight adjustment, β W . We find that, under a mean-field approximation, the asymptotic behavior of the system is described by the replicator equation with an effective payoff matrix, which is a combination of the original game matrix A and its transpose, A T. Both analytical and numerical results show that such an adaptive weight adjustment mechanism dramatically promotes evolution of cooperation.

Dumbing down rational players: Learning and teaching in an experimental game

This paper uses experimental data to examine the existence of a teaching strategy among boundedly rational players. If players realize that their own actions modify their opponents’ beliefs and actions, they might play certain actions to this specific end and forego immediate payoffs if the expected payoff gain from a teaching strategy is high enough. Our results support the existence of a teaching strategy in several ways. After exhibiting some regularities consistent with teaching, we examine more precisely the existence of such a strategy. First we show that players update their beliefs in order to take account of the reaction of their opponents to their own action. Second, we examine whether players actually use a teaching strategy by playing an action that induces a poor immediate payoff but is likely to modify the opponent’s behavior so that a preferable outcome might emerge in the future. We find strong evidence of such a strategy in the data and confirm this finding within a logistic model that suggests that the future expected payoff that could arise from a teaching strategy has indeed a significant impact on choice probabilities. Finally, we investigate the effective impact of a teaching strategy on achieved outcomes and find that more tenacious teachers can successfully use such a strategy in order to reach their favorite outcome at the expense of their opponents.

Disagreement leads to complete learning: Sequential choice with continuous types

Players with observable, continuous payoff types choose actions sequentially. Players update beliefs about the payoff-relevant state of nature using private signals and information about previous signals in the history of types and actions. Heterogeneity in preferences ensures complete learning.

The prisoner’s dilemma on co-evolving networks under perfect rationality

We consider the prisoner’s dilemma being played repeatedly on a dynamic network, where agents may choose their actions as well as their co-players. This leads to co-evolution of network structure and strategy patterns of the players. Individual decisions are made fully rationally and are based on local information only. They are made such that links to defecting agents are resolved and that cooperating agents build up new links. The exact form of the updating scheme is motivated by profit maximization and not by imitation. If players update their decisions in a synchronized way the system exhibits oscillatory dynamics: Periods of growing cooperation (and total linkage) alternate with periods of increasing defection. The cyclical behavior is reduced and the system stabilizes at significant total cooperation levels when players are less synchronized. In this regime we find emergent network structures resembling ‘complex’ and hierarchical topology. The exponent of the power-law degree distribution ( γ ∼ 8.6 ) perfectly matches empirical results for human communication networks.

SEQUENTIAL TWO‐PLAYER GAMES WITH AMBIGUITY*

If players' beliefs are strictly nonadditive, the Dempster–Shafer updating rule can be used to define beliefs off the equilibrium path. We define an equilibrium concept in sequential two‐person games where players update their beliefs with the Dempster–Shafer updating rule. We show that in the limit as uncertainty tends to zero, our equilibrium approximates Bayesian Nash equilibrium. We argue that our equilibrium can be used to define a refinement of Bayesian Nash equilibrium by imposing context‐dependent constraints on beliefs under uncertainty.

Cooperation and self-regulation in a model of agents playing different games.

A simple model for cooperation between "selfish" agents, which play an extended version of the prisoner's dilemma game, in which they use arbitrary payoffs, is presented and studied. A continuous variable, representing the probability of cooperation, p(k)(t) in [0,1], is assigned to each agent k at time t. At each time step t a pair of agents, chosen at random, interact by playing the game. The players update their p(k)(t) using a criterion based on the comparison of their utilities with the simplest estimate for expected income. The agents have no memory and do not use strategies based on direct reciprocity or "tags." Depending on the payoff matrix, the system self-organizes-after a transient-into stationary states characterized by their average probability of cooperation p(eq) and average equilibrium per-capita income p(eq),U(infinity). It turns out that the model exhibits some results that contradict the intuition. In particular, some games that a priori seem to favor defection most, may produce a relatively high degree of cooperation. Conversely, other games, which one would bet lead to maximum cooperation, indeed are not the optimal for producing cooperation.

Almost global convergence to p -dominant equilibrium

A population of players repeatedly plays an n strategy symmetric game. Players update their strategies by sampling the behavior of k opponents and playing a best response to the distribution of strategies in the sample. Suppose the game possesses a -dominant strategy which is initially played by a positive fraction of the population. Then if the population size is large enough, play converges to the -dominant equilibrium with arbitrarily high probability.

The Statistical Mechanics of Best-Response Strategy Revision

This paper compares local and global strategic interaction when players update using the (myopic) best-response rule. I show that randomizing the order in which players update their strategic choice suffices to achieve coordination on the risk-dominant strategy in symmetric 2 × 2 coordination games. The "persistant randomness" which is necessary to achieve similar coordination with global interaction is replaced under local interaction by spatial variation in the initial condition. An extension of the risk-dominance idea gives the same convergence result for K × K games with strategic complementarities. Similar results for K × K pure coordination games and potential games are also presented. Journal of Economic Literature Classification Number: C78.