Predator-prey Game Research Articles

The impact of sociality regimes on heterogeneous cooperative-competitive multi-agent reinforcement learning: a study with the predator-prey game

ABSTRACT The performance in multi-agent reinforcement learning (MARL) scenarios has usually been analysed in homogeneous teams with a few choices for the sociality regime (selfish, egalitarian, or altruistic). In this paper we analyse both homogeneous and heterogeneous teams in a variation of sociality regimes in the predator-prey game, using a novel normalisation of the weights so that the sum of all rewards is independent of the sociality regime. We find that the selfish regime is advantageous for both predator and prey teams, and for both homogeneous and heterogeneous teams. In particular, rewards are about 100% higher for the predator team when switching from the egalitarian to selfish regime and more than 400% higher from the altruistic regime. For the prey, the increase is around 40% and 100% respectively. The results are similar for homogeneous and heterogeneous situations. The takeaway message is that any study of homogeneous and heterogeneous cooperative-competitive multi-agent reinforcement learning teams should also take into account the sociality regimes before making conclusions on the preference of any algorithm.

Hierarchical relationship modeling in multi-agent reinforcement learning for mixed cooperative–competitive environments

In multi-agent reinforcement learning (MARL), information fusion through relationship modeling can effectively learn behavior strategies. However, the high dynamics among heterogeneous interactive agents in mixed cooperative–competitive environments pose difficulties for relational modeling. Traditional MARL solutions concatenate all agents’ states based on the global relationship, which is unrealistic and unscalable under large-scale conditions. Other methods fuse local information by modeling neighbor relationships, but local features lead to suboptimal strategies. From this perspective, we propose a novel relational information fusion approach for mixed tasks to fuse local and global features by modeling heterogeneous relationships through a hierarchical graph. During training, remote agents’ global features are fused through second-order graph representation to help strategy optimization. During decision making, the practicality and scalability of strategy are improved by fusing neighbor agents’ local features through first-order graph representation. Our approach consistently outperforms the state-of-the-art MARL methods in several multi-agent tasks, such as the Predator–Prey and Soccer Games. In particular, it achieves an 83.7% win rate, which is 11.5% higher than baselines in the 4 vs. 4 Soccer Game, and can scale from 4 vs. 4 to 13 vs. 13 in the Predator–Prey Game while maintaining good performance.

Attention Enhanced Reinforcement Learning for Multi agent Cooperation.

In this article, a novel method, called attention enhanced reinforcement learning (AERL), is proposed to address issues including complex interaction, limited communication range, and time-varying communication topology for multi agent cooperation. AERL includes a communication enhanced network (CEN), a graph spatiotemporal long short-term memory network (GST-LSTM), and parameters sharing multi-pseudo critic proximal policy optimization (PS-MPC-PPO). Specifically, CEN based on graph attention mechanism is designed to enlarge the agents' communication range and to deal with complex interaction among the agents. GST-LSTM, which replaces the standard fully connected (FC) operator in LSTM with graph attention operator, is designed to capture the temporal dependence while maintaining the spatial structure learned by CEN. PS-MPC-PPO, which extends proximal policy optimization (PPO) in multi agent systems with parameters' sharing to scale to environments with a large number of agents in training, is designed with multi-pseudo critics to mitigate the bias problem in training and accelerate the convergence process. Simulation results for three groups of representative scenarios including formation control, group containment, and predator-prey games demonstrate the effectiveness and robustness of AERL.

The predator-prey game: Revisiting industrial melanism and optimal foraging theory outdoors with biology undergraduates

Teaching natural selection and adaptations in undergraduate biology classrooms is often undertaken with the example of the <i>Biston</i> peppered moth, a well-documented case of industrial melanism. However, the idea of optimal foraging theory, a behavioral ecological model that includes predators searching for prey, may be overlooked when teaching this classic example of natural selection and predator/prey dynamics. To this end, we developed a simulated predator/prey activity to teach both of these concepts using different size and color toy lizards, moths, and snakes as part of an outdoor laboratory. Students overwhelmingly viewed the laboratory as an engaging way to learn about natural selection (100%, n=115), and how predators forage (Likert median score=5, n=115). We recommend biology instructors across science academic levels (high school and college) incorporate or modify this activity for student-based data collection, as it concomitantly engages undergraduates while providing a hands-on approach to biological and evolutionary theory of natural selection.

Regularization-Adapted Anderson Acceleration for multi-agent reinforcement learning

Originating from model-free reinforcement learning (RL), many modern multi-agent reinforcement learning (MARL) algorithms are usually armed with the paradigm of Centralized Training with Decentralized Execution (CTDE) to mitigate the non-stationary problem and make the training process stable. However, these methods still suffer from sample inefficiency and slow training convergence as in the single-agent reinforcement learning setting. Many common methods aiming to tackle these problems utilize the experience buffer and parallel training mechanism to speed up the training process, which would cost more computing resources and may still underuse the sampled experiences. In this paper, we propose Regularization-Adapted Anderson Acceleration (RA3) for model-free, off-policy MARL algorithms. Under the CTDE paradigm, this specific RA3 approach treats the joint action-value function update as a fixed-point iteration task and speeds up the training process with the same amount of sampled experiences as in the baseline algorithms. Furthermore, our RA3 employs an adaptive regularization strategy related to Bellman residuals to stabilize the update process and enhance the training performance. Experimental results demonstrate that the improved learning speed and superior performance of our proposed method are significantly improved on the predator–prey game and the challenging StarCraft II micromanagement benchmark tasks.

A phenology of fear: Investigating scale and seasonality in predator-prey games between wolves and white-tailed deer.

Predators and prey engage in games where each player must counter the moves of the other, and these games include multiple phases operating at different spatiotemporal scales. Recent work has highlighted potential issues related to scale-sensitive inferences in predator-prey interactions, and there is growing appreciation that these may exhibit pronounced but predictable dynamics. Motivated by previous assertions about effects arising from foraging games between white-tailed deer and canid predators (coyotes and wolves), we used a large and year-round network of trail cameras to characterize deer and predator foraging games, with a particular focus on clarifying its temporal scale and seasonal variation. Linear features were strongly associated with predator detection rates, suggesting these play a central role in canid foraging tactics by expediting movement. Consistent with expectations for prey contending with highly mobile predators, deer responses were more sensitive to proximal risk metrics at finer spatiotemporal scales, suggesting that coarser but more commonly used scales of analysis may miss useful insights into prey risk-response. Time allocation appears to be a key tactic for deer risk management and was more strongly moderated by factors associated with forage or evasion heterogeneity (forest cover, snow and plant phenology) than factors associated with the likelihood of predator encounter (linear features). Trade-offs between food and safety appeared to vary as much seasonally as spatially, with snow and vegetation phenology giving rise to a "phenology of fear". Deer appear free to counter predators during milder times of year, but a combination of poor-foraging state, reduced forage availability, greater movements costs, and reproductive state dampen responsiveness during winter. Pronounced intra-annual variation in predator-prey interactions may be common in seasonal environments.

Hybrid attention-oriented experience replay for deep reinforcement learning and its application to a multi-robot cooperative hunting problem

Multiple robots complete a cooperative hunting task by obtaining environmental information and autonomously learning hunting decision-making strategies. However, with the increase in the number of environment participants, it becomes difficult for robots to process a large amount of environmental information. Thus, a multi-robot cooperative hunting decision-making method called hybrid attention-oriented experience replay in multi-agent deep deterministic policy gradient (HAER-MADDPG) is proposed. First, a hybrid attention module is designed to pay greater attention to key information in a large amount of environmental information by integrating it with the multi-agent deep deterministic policy gradient (MADDPG). The method then combines hybrid attention and prioritized experience replay to improve the utilization of experience samples. Finally, the proposed algorithm is tested through a predator–prey game. The results show that the effectiveness, convergence speed, and scalability of the proposed algorithm are better than those of the baseline algorithms. In addition, HAER-MADDPG is effectively applied to a hunting task with real robots.

Incorporating thermodynamics in predator-prey games predicts the diel foraging patterns of poikilothermic predators.

Models of foraging behaviour typically assume that prey do not adapt to temporal variation in predation risk, such as by avoiding foraging at certain times of the day. When this behavioural plasticity is considered-such as in predator-prey games-the role of abiotic factors is usually ignored. An abiotic factor that exerts strong influence on the physiology and behaviour of many animals is ambient temperature, although it is often ignored from game models as it is implicitly assumed that both predators and prey are homothermic. However, poikilotherms' performance may be reduced in cold conditions due to reduced muscle function, limiting the prey-capture ability of predators and the predator-avoidance and foraging abilities of prey. Here, we use a game-theoretic predator-prey model in which diel temperature changes influence foraging gains and costs to predict the evolutionarily stable diel activity of predators. Our model predicts the range of patterns observed in nature, including nocturnal, diurnal, crepuscular and a previously unexplained post-sunset crepuscular pattern observed in some sharks. In general, smaller predators are predicted to be more diurnal than larger ones. The safety of prey when not foraging is critical, explaining why predators in coral reef systems (with safe refuges) may often have different foraging patterns to pelagic predators. We make a range of testable predictions that will enable the further evaluation of this theoretical framework for understanding diel foraging patterns in poikilotherms.

Modelling the sequential behaviours of simultaneous predator and prey patch use

Models investigating predator–prey games often fix one player's behaviour while allowing the other to make patch use choices. Here, we use adaptive dynamics to model changes in behavioural rates of a predator and prey, while explicitly allowing both to make decisions simultaneously. Our model focuses on a two-patch situation. The prey can move between an open patch, allowing for energy intake but also predation risk, and a refuge with no possibility of energy intake or predation risk. The predator can move between that patch, where it can attack prey, and an alternative patch that provides a fixed rate of energy intake, and consequently imposes a missed opportunity cost (MOC) of using the focal patch. The model predicts how the rates of movement by the players are influenced by specific parameters, including the predator's lethality and MOC, as well as the information available to the players. For example, our model predicts that the time a prey takes to emerge from its refuge and the time a predator waits for the prey to emerge depend on the sequence of events leading to the prey entering the refuge. For instance, the predator would leave the patch more quickly and the prey would emerge from the refuge more quickly if the prey entered the refuge after a failed attack rather than as part of its time allocation. • We use adaptive dynamics to model behavioural changes in predator–prey patch use. • We model a two-patch choice: open vs environment (predator); open vs refuge (prey). • Players can make decisions simultaneously and use information from recent encounters. • Specific parameters (risk, opportunity) affect each player's rate of movements. • Adaptive movement rules depend on predator lethality and missed opportunity costs.

Rare extinction events in cyclic predator–prey games

In the May–Leonard model of three cyclically competing species, we analyze the statistics of rare events in which all three species go extinct due to strong but rare fluctuations. These fluctuations are from the tails of the probability distribution of species concentrations. They render a coexistence of three populations unstable even if the coexistence is stable in the deterministic limit. We determine the mean time to extinction (MTE) by using a WKB-ansatz in the master equation that represents the stochastic description of this model. This way, the calculation is reduced to a problem of classical mechanics and amounts to solving a Hamilton–Jacobi equation with zero-energy Hamiltonian. We solve the corresponding Hamilton’s equations of motion in six-dimensional phase space numerically by using the Iterative Action Minimization Method. This allows to project on the optimal path to extinction, starting from a parameter choice where the three-species coexistence-fixed point undergoes a Hopf bifurcation and becomes stable. Specifically for our system of three species, extinction events can be triggered along various paths to extinction, differing in their intermediate steps. We compare our analytical predictions with results from Gillespie simulations for two-species extinctions, complemented by an analytical calculation of the MTE in which the remaining third species goes extinct. From Gillespie simulations we also analyze how the distributions of times to extinction (TE) change upon varying the bifurcation parameter. Our results shed some light on the sensitivity of the TE to system parameters. Even within the same model and the same dynamical regime, which allows a stable coexistence of species in the deterministic limit, the MTE depends on the distance from the bifurcation point in a way that contains the system size dependence in the exponent. It is challenging and worthwhile to quantify how rare the rare events of extinction are.

Where and when to hunt? Decomposing predation success of an ambush carnivore.

Predator-prey games emerge when predators and prey dynamically respond to the behavior of one another, driving the outcomes of predator-prey interactions. Predation success is a function of the combined probabilities of encountering and capturing prey, which are influenced by both prey behavior and environmental features. While the relative importance of encounter and capture probabilities have been evaluated in a spatial framework, temporal variation in prey behavior and intrinsic catchability are likely to also affect the distribution of predation events. Using a single-predator-single-prey (puma-vicuña) system, we evaluated which factors predict predation events across both temporal and spatial dimensions of the components of predation by testing the prey-abundance hypothesis (predators select for high encounter probability) and the prey-catchability hypothesis (predators select for high relative capture probability) in time and space. We found that for both temporal and spatial analyses, neither the prey-abundance hypothesis nor the prey-catchability hypothesis alone predicted kill frequency or distribution; puma kill frequency was static throughout the diel cycle and pumas consistently selected a single habitat type when hunting, despite temporal and spatial variation in encounter rates and intrinsic catchability. Our integrated spatiotemporal analysis revealed that an interaction between time of day and habitat influences kill probability, suggesting that trade-offs in the temporal and spatial components of predation drive the probability of predation events. These findings reinforce the importance of examining both the temporal and spatial patterns of the components of predation, rather than unidimensional measures of predator or prey behavior, to comprehensively describe the feedbacks between predator and prey in the predator-prey game.

Predator-prey games on complex networks

We study the predator-prey game, an intriguing collective and dynamic process in nature, on complex networks. The game is over when the prey and the predator meet at the same node, where the associated mean first encounter time is used as the payoff value. We find that for a number of real networks, a saddle point will appear on the resultant payoff matrix, suggesting that each player has an optimal pure strategy. Interestingly, the prey and the predator exhibit distinct best responses. Specifically, the prey tends to move towards low-degree nodes to escape, while the predator chasing is more likely to hop to nodes with medium-degree. Moreover, in this situation, a scaling behavior of the value of the game indicates that the lifetime of the prey is very long in an infinite-size environment. Furthermore, we find that the saddle point of the predator-prey game occurs more readily on homogeneous networks rather than on heterogeneous ones. Our work provides a basic modeling framework for studying various collective behavior of interacting entities on complex networks.

Oceanic diel vertical migrations arising from a predator-prey game

The deep scattering layer is an ubiquitous aggregation of zooplankton and forage fish in the ocean. It features the striking phenomenon of diel vertical migration, where animals remain in deep, dark water in daylight hours and migrate upwards at dusk. The common explanation for this is that prey avoid vulnerability to visual predators. Here, we develop a game-theoretic explanation for the deep scattering layer and its diel vertical migrations, focusing on one generic predator species and one generic prey species. The model is formulated in continuous time and space, and by neglecting the cost of locomotion, it allows fine-grained predictions of vertical distributions. The Nash equilibrium features a distinct deep scattering layer which undertakes diel vertical migrations, the range of which increases with predator abundance. Maximum feeding rates are predicted to occur at dawn and dusk. Predator interference emerges from the game dynamics in the form of a complicated functional dependency of gross encounter rates on predator abundance. In turn, the growth rate of the prey decreases monotonically with predator abundance. In addition to providing a mechanistic explanation for the striking phenomenon of diel vertical migrations, the model yields quantitative predictions of vertical distributions and diel patterns in feeding intake, which may be compared with acoustic data, observed individual behavior, or stomach data.

How does the presence of a conspecific individual change the behavioral game that a predator plays with its prey?

Behavioral games predators play among themselves may have profound effects on behavioral games predators play with their prey. We studied the behavioral game between predators and prey within the framework of social foraging among predators. We tested how conspecific interactions among predators (little egret) change the predator-prey behavioral game and foraging success. To do so, we examined foraging behavior of egrets alone and in pairs (male and female) in a specially designed aviary consisting of three equally spaced pools with identical initial prey (comet goldfish) densities. Each pool was comprised of a risky microhabitat, rich with food, and a safe microhabitat with no food, forcing the fish to trade off food and safety. When faced with two versus one egret, we found that fish significantly reduced activity in the risky habitat. Egrets in pairs suffered reduced foraging success (negative intraspecific density dependence) and responded to fish behavior and to their conspecific by changing their visiting regime at the different pools-having shorter, more frequent visits. The time egret spent on each visit allowed them to match their long-term capture success rate across the environment to their capture success rate in the pool, which satisfies one aspect of optimality. Overall, egrets in pairs allocated more time for foraging and changed their foraging tactics to focus more on fish under cover and fish 'peeping' out from their shelter. These results suggest that both prey and predator show behavioral flexibility and can adjust to changing conditions as needed in this foraging game.

Rock-paper-scissors played within competing domains in predator-prey games

We consider (N, r) games of prey and predation with N species and r < N prey and predators, acting in a cyclic way. Further basic reactions include reproduction, decay and diffusion over a one- or two-dimensional regular grid, without a hard constraint on the occupation number per site, so in a ‘bosonic’ implementation. For special combinations of N and r and appropriate parameter choices we observe games within games, that is different coexisting games, depending on the spatial resolution. As a concrete and simplest example we analyze the (6,3) game. Once the players segregate from a random initial distribution, domains emerge, which effectively play a (2,1)-game on the coarse scale of domain diameters, while agents inside the domains play (3,1) (rock-paper-scissors), leading to spiral formation with species chasing each other. The (2,1)-game has a winner in the end, so that the coexistence of domains is transient, while agents inside the remaining domain coexist, until demographic fluctuations lead to extinction of all but one species in the very end. This means that we observe a dynamical generation of multiple space and time scales with emerging re-organization of players upon segregation, starting from a simple set of rules on the smallest scale (that of the grid) and changed rules from the coarser perspective. These observations are based on Gillespie simulations. We discuss the deterministic limit derived from a van Kampen expansion. In this limit we perform a linear stability analysis and numerically integrate the resulting equations. The linear stability analysis predicts the number of forming domains, their composition in terms of species; it explains the instability of interfaces between domains, which drives their extinction; spiral patterns are identified as motion along heteroclinic cycles. The numerical solutions reproduce the observed patterns of the Gillespie simulations including even extinction events, so that the mean-field analysis here is very conclusive, which is due to the specific implementation of rules.

Cooperation can emerge in prisoner’s dilemma from a multi-species predator prey replicator dynamic

In this paper we study a generalized variation of the replicator dynamic that involves several species and sub-species that may interact. We show how this dynamic comes about from a specific finite-population model, but also show that one must take into consideration the dynamic nature of the population sizes (and hence proportions) in order to make the model complete. We provide expressions for these population dynamics to produce a kind of multi-replicator dynamic. We then use this replicator dynamic to show that cooperation can emerge as a stable behavior when two species each play prisoner’s dilemma as their intra-species game and a form of zero-sum predator prey game as their inter-species game. General necessary and sufficient conditions for cooperation to emerge as stable are provided for a number of game classes. We also showed an example using Hawk–Dove where both species can converge to stable (asymmetric) mixed strategies.

Optimal search and ambush for a hider who can escape the search region

Search games for a mobile or immobile hider traditionally have the hider permanently confined to a compact ‘search region’ making eventual capture inevitable. Hence the payoff can be taken as time until capture. However in many real life search problems it is possible for the hider to escape an area in which he was known to be located (e.g. Bin Laden from Tora Bora) or for a prey animal to escape a predator’s hunting territory. We model and solve such continuous time problems with escape where we take the probability of capture to be the searcher’s payoff.We assume the searcher, while cruise searching, can cover the search region at unit rate of area, for a given time horizon T known to the hider. The hider can stay still or choose any time to flee the region. To counter this, the searcher can also adopt an ambush mode which will capture a fleeing hider. The searcher wins the game if he either finds the hider while cruise searching or ambushes him while he is attempting to flee; the hider wins if he flees successfully (while the searcher is cruising) or has not been found by time T. The optimal searcher strategy involves decreasing the ambush probability over time, to a limit of zero. This surprising behaviour is opposite to that found recently by Alpern et al. (2011, 2013) in a predator-prey game with similar dynamics but without the possibility of the hider escaping. Our work also complements that of Zoroa et al. (2015) on searching for multiple prey and Gal and Casas (2014) for a combined model of search and pursuit.

Reactive responses of zebras to lion encounters shape their predator–prey space game at large scale

The predator–prey space game and the costs associated with risk effects are affected by prey 1) proactive adjustments (when prey modify their behaviour in response to an a priori assessment of the risk level) and 2) reactive adjustments (when prey have detected an immediate threat). Proactive adjustments are generally well‐studied, whereas the frequency, strength and duration of reactive adjustments remain largely unknown. We studied the space use and habitat selection of GPS‐collared zebrasEquus quaggafrom 2 to 48 h after an encounter with lionsPanthera leo. Lion–zebra encounters generally occurred close to artificial waterholes (&lt; 1 km). Two hours after an encounter, zebras were more likely to have fled than stay when the encounter occurred in more risky bushy areas. During their flight, zebras selected grasslands more than usual, getting great visibility. Regardless of their initial response, zebras finally fled at the end of the night and reached areas located far from waterholes where encounters with lions are less frequent. The large‐scale flights (∼4–5 km) of zebras led to a local zebra depression for lions. Zebras that had fled immediately after the encounter resumed their behaviour of coming close to waterholes on the following day. However, zebras that had initially stayed remained far from waterholes for an extra 24 h, remaining an elusive prey for longer. The delay in the flight decision had different short‐term consequences on the lion–zebra game. We reveal that the spatial context of the encounter shapes the immediate response of prey, and that encountering predators induces strong behavioural responses: prey flee towards distant, safer, areas and have a constrained use of key resource areas which are at the heart of the predator–prey game at larger spatio‐temporal scales. Nighttime encounters were infrequent (once every 35 days on average), zebra responses were short‐lived (&lt; 36 h) but occurred over a large spatial scale (several km).

Logging‐induced changes in habitat network connectivity shape behavioral interactions in the wolf–caribou–moose system

Habitat connectivity influences the distribution dynamics of animals. Connectivity can therefore shape trophic interactions, but little empirical evidence is available, especially for large mammals. In forest ecosystems, logging alters functional connectivity among habitat patches, and such activities can affect the spatial game between large herbivores and their predators. We used graph theory to evaluate how harvesting‐induced changes in habitat connectivity influence patch choice and residency time of GPS‐collared caribou (Rangifer tarandus caribou) and moose (Alces alces) in winter in the boreal forest. We then investigated the predator–prey game by assessing how GPS‐collared wolves (Canis lupus) adjusted their movements to changes in landscape properties and in the networks of their prey species. We built prey habitat networks using minimum planar graphs organized around species‐specific, highly selected habitat patches (i.e., network nodes). We found that spatial dynamics of large herbivores were influenced not only by the intrinsic quality of habitat patches, but also by the connectivity of those network nodes. Caribou and moose selected nodes that were connected by a high number of links, and moose also spent relatively more time in those nodes. By limiting node accessibility, human disturbances influenced travel decisions. Caribou and moose avoided nodes that were surrounded by a high proportion of cuts and roads, but once within these nodes, moose stayed longer than in other nodes. Caribou selectively moved among nodes with low distance costs, and their residency time increased with distance costs required to reach the nodes. Wolves selected their prey's nodes, where vegetation consumed by caribou and moose was highly abundant. Furthermore, wolves discriminated among those nodes by selecting the most connected ones. In fact, selection by wolves was stronger for their prey's nodes than for the prey's utilization distribution per se, a difference that increased with the level of human disturbance. Considering the difficulty of keeping track of highly mobile prey, predators may benefit by targeting not only their prey's resource patches, but also the most highly connected patches. Matrix quality and connectivity are therefore key elements shaping the predator–prey spatial game in human‐altered landscapes because of their impact on the spatial dynamics of the interacting species.

Evolutionary Inversion of Swarm Emergence Using Disjunctive Combs Control

Given simple agent rules, a swarm's emergent behavior can be difficult to predict. The inverse problem is even more difficult: Given a desired emergent behavior, what are the rules by which swarm agents should operate? Disjunctive fuzzy control is proposed as a method to model swarm agents. Compared to more commonly used conjunctive fuzzy control such as that proposed by Mamdani, disjunctive fuzzy control is robustly fault tolerant and disjointly connected. Swarms are inherently disjunctive. Instead of agents working in coordination with one another, each swarm agent contributes individually to the result. The disjunctive attribute can also be applied at the sensor level for each individual agent. Disjunctive control allows adaptation of the describing membership function, as is commonly done in conjunctive control. The inversion process is illustrated with numerous simulation examples, including a predator-prey game, gang warfare, and escaping agents. The swarm is instructed what to do but not how to do it. Imposition of fitness constraints and repeated generations of evolutionary molding of agent performance can then result in unexpected emergent behaviors of the swarm, e.g., use of decoys, self-sacrifice, flanking maneuvers, and shielding of the weak.