Exploring cooperative evolution with tunable payoff’s loners using reinforcement learning

Abstract

Imitation and replication have emerged as a paradigm in numerous studies that explore the evolution of cooperative behavior. Since they embrace the essence of natural selection, it is widely recognized in exploring the evolution of biological behaviors. However, it is not easy to express the way individuals select and optimize in these simple and elegant ways in the complex and variable interactive environments. Currently, reinforcement learning is widely used in the study of strategy updating dynamics and agent learning processes in game theory. Therefore, we introduce the Q-learning algorithms into the voluntary public goods game to explore the impact of cooperative evolution. Simulation results demonstrate that when the synergy factor is large and the adjust loner payoff’s multiply factor is smaller, the number of cooperators becomes gradually consistent. As the synergy factor increases, the evolution of the proportion of defectors become nonlinear. Furthermore, we further explore the Q-table and strategy updating processes of agents in the steady state under a smaller multiply factor that adjusts the loner’s payoff. The results find inconsistency between the average Q-values and the steady state population strategy distribution. Subsequently, we explain the reason for the inconsistency by analyzing strategy sequence, namely that there are a number of agents who constantly change strategies in the population, and the Q-values of these agents have an impact on the overall Q-values. In addition, evolutionary snapshots of agent strategy sequences are observed. The results find that the agent’s strategic selection shows greater instability when the proportions of cooperators, defectors, and loners in the population are relatively balanced. Finally, the effect of parameters in the Q-learning algorithm on cooperative behavior is analyzed. This study hopes to provide valuable insights into understanding the dynamics of cooperation in complex social interactions.