Lose-shift Strategy Research Articles

A Comparison of Rapid Rule-Learning Strategies in Humans and Monkeys.

Interspecies comparisons are key to deriving an understanding of the behavioral and neural correlates of human cognition from animal models. We perform a detailed comparison of the strategies of female macaque monkeys to male and female humans on a variant of the Wisconsin Card Sorting Test (WCST), a widely studied and applied task that provides a multiattribute measure of cognitive function and depends on the frontal lobe. WCST performance requires the inference of a rule change given ambiguous feedback. We found that well-trained monkeys infer new rules three times more slowly than minimally instructed humans. Input-dependent hidden Markov model-generalized linear models were fit to their choices, revealing hidden states akin to feature-based attention in both species. Decision processes resembled a win-stay, lose-shift strategy with interspecies similarities as well as key differences. Monkeys and humans both test multiple rule hypotheses over a series of rule-search trials and perform inference-like computations to exclude candidate choice options. We quantitatively show that perseveration, random exploration, and poor sensitivity to negative feedback account for the slower task-switching performance in monkeys.

Cognitive flexibility in urban yellow mongooses, Cynictis penicillata

Cognitive flexibility enables animals to alter their behaviour and respond appropriately to environmental changes. Such flexibility is important in urban settings where environmental changes occur rapidly and continually. We studied whether free-living, urban-dwelling yellow mongooses, Cynictis penicillata, in South Africa, are cognitively flexible in reversal learning and attention task experiments (n = 10). Reversal learning was conducted using two puzzle boxes that were distinct visually and spatially, each containing a preferred or non-preferred food type. Once mongooses learned which box contained the preferred food type, the food types were reversed. The mongooses successfully unlearned their previously learned response in favour of learning a new response, possibly through a win-stay, lose-shift strategy. Attention task experiments were conducted using one puzzle box surrounded by zero, one, two or three objects, introducing various levels of distraction while solving the task. The mongooses were distracted by two and three distractions but were able to solve the task despite the distractions by splitting their attention between the puzzle box task and remaining vigilant. However, those exposed to human residents more often were more vigilant. We provide the first evidence of cognitive flexibility in urban yellow mongooses, which enables them to modify their behaviour to urban environments.

Action Sequence Learning Is Impaired in Genetically Modified Mice with the Suppressed GABAergic Transmission from the Thalamic Reticular Nucleus to the Thalamus.

The thalamic reticular nucleus (TRN) is a thin sheet of GABAergic neurons surrounding the thalamus, and it regulates the activity of thalamic relay neurons. The TRN has been reported to be involved in sensory gating, attentional regulation, and some other functions. However, little is known about the contribution of the TRN to sequence learning. In the present study, we examined whether the TRN is involved in reward-based learning of action sequence with no eliciting stimuli (operant conditioning), by analyzing the performance of male and female Avp-Vgat-/- mice (Vgatflox/flox mice crossed to an Avp-Cre driver line) on tasks conducted in an operant box having three levers. Our histological and electrophysiological data demonstrated that in adult Avp-Vgat-/- mice, vesicular GABA transporter (VGAT) was absent in most TRN neurons and the GABAergic transmission from the TRN to the thalamus was largely suppressed. The performance on a task in which mice needed to press an active lever for food reward showed that simple operant learning of lever pressing and learning of win-stay and lose-shift strategies are not affected in Avp-Vgat-/- mice. In contrast, the performance on a task in which mice needed to press three levers in a correct order for food reward showed that learning of the order of lever pressing (action sequence learning) was impaired in Avp-Vgat-/- mice. These results suggest that the TRN plays an important role in action sequence learning.

Limiting Dynamics for Q-Learning with Memory One in Symmetric Two-Player, Two-Action Games

We develop a method based on computer algebra systems to represent the mutual pure strategy best-response dynamics of symmetric two-player, two-action repeated games played by players with a one-period memory. We apply this method to the iterated prisoner’s dilemma, stag hunt, and hawk-dove games and identify all possible equilibrium strategy pairs and the conditions for their existence. The only equilibrium strategy pair that is possible in all three games is the win-stay, lose-shift strategy. Lastly, we show that the mutual best-response dynamics are realized by a sample batch Q-learning algorithm in the infinite batch size limit.

Cognitive flexibility and aging in coyotes (Canis latrans).

Cognitive flexibility evolves in species that live in complex and dynamic social systems and habitats and may enable species to better cope with anthropogenic habitat modification. Aging may also impact the cognitive abilities of canids. Coyotes (Canis latrans) and domestic dogs (Canis lupus familiaris) differ markedly in their social and trophic ecology but have both been highly successful in adapting to human-modified ecosystems. Aging dogs develop a form of dementia that mirrors Alzheimer's disease in humans, but it is unknown whether similar cognitive deficits develop with age in coyotes and other wild canids. In this study, we modified a spatial serial reversal-learning test that was sensitive to cognitive aging in pet dogs to test cognitive flexibility in captive coyotes. We also performed a second experiment using a color discrimination task to test for flexible rule learning. A total of 19 of 20 coyotes demonstrated the ability to track shifts in spatial reward contingencies and learned to rapidly complete reversals by using a win-stay, lose-shift strategy. In addition, coyotes inhibited prepotent win-stay choices to acquire the color discrimination task. These findings suggest that behavioral flexibility may help coyotes to detect and respond appropriately to both rapid fluctuations and gradual changes in ecological conditions. Performance did not differ between coyotes and previously tested dogs, but similar to dogs, behavioral flexibility declined with age in adult coyotes. Thus, cognitive decline and flexibility may be conserved among canines pending additional studies on other Canis species. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

A test of the win-stay–lose-shift foraging strategy and its adaptive value in albatrosses

Foraging site selection and site fidelity can have implications for many ecological processes. The degree of site fidelity differs greatly not just between species but also within populations. Some of this variation may be explained by a win-stay–lose-shift (WSLS) strategy, where an individual returns to its most recent foraging area only if the previous visit was profitable. However, the use and adaptive value of this strategy have mostly been tested in captivity, largely because of the difficulty in obtaining accurate measures of profitability (foraging efficiency) in the wild. Here, we used a rare combination of data on movements of breeding black-browed albatrosses, Thalassarche melanophris , tracked using satellite transmitters, and on chick meal mass obtained from automatic nest balances, to test whether individuals adopted a WSLS strategy, and how this strategy performed in terms of provisioning rate. We found results consistent with the use of a WSLS strategy, and that the strategy had some adaptive value, albeit rather limited. Our observational study of free-living seabirds corroborates previous experimental results suggesting that animals account for recent foraging success in their decision making and can adapt their strategy to local resource dynamics. • Site fidelity in albatrosses was higher when the previous trip was more profitable. • Wild seabirds showed evidence of a win-staye–lose-shift (WSLS) foraging strategy. • This strategy yielded small adaptive payoffs in terms of meal deliveries to chicks.

Symmetric equilibrium of multi-agent reinforcement learning in repeated prisoner’s dilemma

We investigate the repeated prisoner’s dilemma game where both players alternately use reinforcement learning to obtain their optimal memory-one strategies. We theoretically solve the simultaneous Bellman optimality equations of reinforcement learning. We find that the Win-stay Lose-shift strategy, the Grim strategy, and the strategy which always defects can form symmetric equilibrium of the mutual reinforcement learning process amongst all deterministic memory-one strategies.

Sex-dependent effects of chronic intermittent voluntary alcohol consumption on attentional, not motivational, measures during probabilistic learning and reversal.

BackgroundForced alcohol (ethanol, EtOH) exposure has been shown to cause significant impairments on reversal learning, a widely-used assay of cognitive flexibility, specifically on fully-predictive, deterministic versions of this task. However, previous studies have not adequately considered voluntary EtOH consumption and sex effects on probabilistic reversal learning. The present study aimed to fill this gap in the literature.MethodsMale and female Long-Evans rats underwent either 10 weeks of voluntary intermittent 20% EtOH access or water only (H2O) access. Rats were then pretrained to initiate trials and learn stimulus-reward associations via touchscreen response, and subsequently required to select between two visual stimuli, rewarded with probability 0.70 or 0.30. In the final phase, reinforcement contingencies were reversed.ResultsWe found significant sex differences on several EtOH-drinking variables, with females reaching a higher maximum EtOH consumption, exhibiting more high-drinking days, and escalating their EtOH at a quicker rate compared to males. During early abstinence, EtOH drinkers (and particularly EtOH-drinking females) made more initiation omissions and were slower to initiate trials than H2O drinking controls, especially during pretraining. A similar pattern in trial initiations was also observed in discrimination, but not in reversal learning. EtOH drinking rats were unaffected in their reward collection and stimulus response times, indicating intact motivation and motor responding. Although there were sex differences in discrimination and reversal phases, performance improved over time. We also observed sex-independent drinking group differences in win-stay and lose-shift strategies specific to the reversal phase.ConclusionsFemales exhibit increased vulnerability to EtOH effects in early learning: there were sex-dependent EtOH effects on attentional measures during pretraining and discrimination phases. We also found sex-independent EtOH effects on exploration strategies during reversal. Future studies should aim to uncover the neural mechanisms for changes in attention and exploration in both acute and prolonged EtOH withdrawal.

Behavioural and neural modulation of win-stay but not lose-shift strategies as a function of outcome value in Rock, Paper, Scissors.

Competitive environments in which individuals compete for mutually-exclusive outcomes require rational decision making in order to maximize gains but often result in poor quality heuristics. Reasons for the greater reliance on lose-shift relative to win-stay behaviour shown in previous studies were explored using the game of Rock, Paper, Scissors and by manipulating the value of winning and losing. Decision-making following a loss was characterized as relatively fast and relatively inflexible both in terms of the failure to modulate the magnitude of lose-shift strategy and the lack of significant neural modulation. In contrast, decision-making following a win was characterized as relatively slow and relatively flexible both in terms of a behavioural increase in the magnitude of win-stay strategy and a neural modulation of feedback-related negativity (FRN) and stimulus-preceding negativity (SPN) following outcome value modulation. The win-stay/lose-shift heuristic appears not to be a unified mechanism, with the former relying on System 2 processes and the latter relying on System 1 processes. Our ability to play rationally appears more likely when the outcome is positive and when the value of wins are low, highlighting how vulnerable we can be when trying to succeed during competition.

Market Imitation and Win-Stay Lose-Shift Strategies Emerge as Unintended Patterns in Market Direction Guesses.

Decisions made in our everyday lives are based on a wide variety of information so it is generally very difficult to assess what are the strategies that guide us. Stock market provides a rich environment to study how people make decisions since responding to market uncertainty needs a constant update of these strategies. For this purpose, we run a lab-in-the-field experiment where volunteers are given a controlled set of financial information -based on real data from worldwide financial indices- and they are required to guess whether the market price would go “up” or “down” in each situation. From the data collected we explore basic statistical traits, behavioural biases and emerging strategies. In particular, we detect unintended patterns of behavior through consistent actions, which can be interpreted as Market Imitation and Win-Stay Lose-Shift emerging strategies, with Market Imitation being the most dominant. We also observe that these strategies are affected by external factors: the expert advice, the lack of information or an information overload reinforce the use of these intuitive strategies, while the probability to follow them significantly decreases when subjects spends more time to make a decision. The cohort analysis shows that women and children are more prone to use such strategies although their performance is not undermined. Our results are of interest for better handling clients expectations of trading companies, to avoid behavioural anomalies in financial analysts decisions and to improve not only the design of markets but also the trading digital interfaces where information is set down. Strategies and behavioural biases observed can also be translated into new agent based modelling or stochastic price dynamics to better understand financial bubbles or the effects of asymmetric risk perception to price drops.

Win-stay lose-shift strategy in formation changes in football

Managerial decision making is likely to be a dominant determinant of performance of teams in team sports. Here we use Japanese and German football data to investigate correlates between temporal patterns of formation changes across matches and match results. We found that individual teams and managers both showed win-stay lose-shift behavior, a type of reinforcement learning. In other words, they tended to stick to the current formation after a win and switch to a different formation after a loss. In addition, formation changes did not statistically improve the results of succeeding matches.The results indicate that a swift implementation of a new formation in the win-stay lose-shift manner may not be a successful managerial rule of thumb.

A Game of Hide and Seek: Expectations of Clumpy Resources Influence Hiding and Searching Patterns.

Resources are often distributed in clumps or patches in space, unless an agent is trying to protect them from discovery and theft using a dispersed distribution. We uncover human expectations of such spatial resource patterns in collaborative and competitive settings via a sequential multi-person game in which participants hid resources for the next participant to seek. When collaborating, resources were mostly hidden in clumpy distributions, but when competing, resources were hidden in more dispersed (random or hyperdispersed) patterns to increase the searching difficulty for the other player. More dispersed resource distributions came at the cost of higher overall hiding (as well as searching) times, decreased payoffs, and an increased difficulty when the hider had to recall earlier hiding locations at the end of the experiment. Participants’ search strategies were also affected by their underlying expectations, using a win-stay lose-shift strategy appropriate for clumpy resources when searching for collaboratively-hidden items, but moving equally far after finding or not finding an item in competitive settings, as appropriate for dispersed resources. Thus participants showed expectations for clumpy versus dispersed spatial resources that matched the distributions commonly found in collaborative versus competitive foraging settings.

File sharing in cloud computing using win stay lose shift strategy

Traditionally, users in clouds are assumed to be willing to store and share files in clouds, where the security and efficiency issues are discussed. However, few studies involve the incentives to share files in clouds. In this paper, we delve into users' incentives for using the cloud system to store and share their files towards the view of game theory. More specifically, the process of file sharing is reduced to an infinitely repeated prisoner's dilemma PD game and the action of sharing files in clouds is considered as the action of cooperation in the infinitely repeated PD game. We incorporate win stay lose shift WSLS strategy into file sharing and simulate it compared with tit-for-tat TFT strategy in clouds. Simulation results show that WSLS is an optimal strategy for users to share their files in clouds. Furthermore, WSLS is robust for unintentional deviation and returns to mutual cooperation after deviation.

Strategy switches and co-action equilibria in a minority game

We propose an analytically tractable variation of the minority game in which rational agents use probabilistic strategies. In our model, N agents choose between two alternatives repeatedly, and those who are in the minority get a pay-off 1, others zero. The agents optimize the expectation value of their discounted future pay-off, the discount parameter being λ. We propose an alternative to the standard Nash equilibrium, called co-action equilibrium, which gives higher expected pay-off for all agents. The optimal choice of probabilities of different actions are determined exactly in terms of simple self-consistent equations. The optimal strategy is characterized by N real parameters, which are non-analytic functions of λ, even for a finite number of agents. The solution for N≤7 is worked out explicitly indicating the structure of the solution for larger N. For large enough future time horizon, the optimal strategy switches from random choice to a win-stay lose-shift strategy, with the shift probability depending on the current state and λ.

Effects of a novel CB1 agonist on visual attention in male rats: Role of strategy and expectancy in task accuracy.

The effects of cannabinoid CB1 agonists (including Δ9-tetrahydrocannabinol, the main psychoactive component of marijuana) on attention are uncertain, with reports of impairments, no effects, and occasionally performance enhancements. To better understand these effects, we sought to uncover a role of changing online (within-session) strategy as a possible mediator of the effects of the novel, potent CB1 agonist AM 4054, on a model of sustained attention in male Sprague-Dawley rats. In this operant, two-choice reaction time (RT) task, AM 4054 decreased accuracy in an asymmetric manner; that is, performance was spared on one lever but impaired on the other. Furthermore, this pattern was enhanced by the outcome of the previous trial such that AM 4054 strengthened a win-stay strategy on the "preferred" lever and a lose-shift strategy on the "nonpreferred" lever. This pattern is often found in tests of expectancy; therefore, in a second experiment AM 4054 enhanced expectancy that we engendered by altering the probability of the two stimulus cues. Accuracy was impaired in reporting the less frequent cue, but only after two or more presentations of the more frequent cue. Taking the results of the experiments together, AM 4054 engendered expectancy by increasing the role of previous trial location and outcome on performance of future trials, diminishing stimulus control (and therefore, accuracy). This novel effect of CB1 receptor agonism may contribute to the deleterious effects of cannabinoids on attention.

Adversarial scheduling in discrete models of social dynamics

In this paper we advocate the study of discrete models of social dynamics underadversarial scheduling. The approach we propose forms part of a foundational basis for agenerative approach to social science(Epstein 2007). We highlight the feasibility of the adversarial scheduling approach by using it to study thePrisoners's Dilemma Game with Pavlov update, a dynamics that has already been investigated under random update in Kittock (1994), Dyeret al. (2002), Mossel and Roch (2006) and Dyer and Velumailum (2011). The model is specified by letting players at the nodes of an underlying graphGrepeatedly play the Prisoner's Dilemma against their neighbours. The players adapt their strategies based on the past behaviour of their opponents by applying the so-called win–stay lose–shift strategy. With random scheduling, starting from any initial configuration, the system reaches the fixed point in which all players cooperate with high probability. On the other hand, under adversarial scheduling the following results hold:—A scheduler that can selectbothgame participants can preclude the system from reaching the unique fixed point on most graph topologies.—A non-adaptive scheduler that is only allowed to chooseoneof the participants is no more powerful than a random scheduler. With this restriction, even an adaptive scheduler is not significantly more powerful than the random scheduler, provided it is ‘reasonably fair’.

Generosity Pays in the Presence of Direct Reciprocity: A Comprehensive Study of 2×2 Repeated Games

By applying a technique previously developed to study ecosystem assembly [Capitán et al., Phys. Rev. Lett. 103, 168101 (2009)] we study the evolutionary stable strategies of iterated 22 games. We focus on memory-one strategies, whose probability to play a given action depends on the actions of both players in the previous time step. We find the asymptotically stable populations resulting from all possible invasions of any known stable population. The results of this invasion process are interpreted as transitions between different populations that occur with a certain probability. Thus the whole process can be described as a Markov chain whose states are the different stable populations. With this approach we are able to study the whole space of symmetric 22 games, characterizing the most probable results of evolution for the different classes of games. Our analysis includes quasi-stationary mixed equilibria that are relevant as very long-lived metastable states and is compared to the predictions of a fixation probability analysis. We confirm earlier results on the success of the Pavlov strategy in a wide range of parameters for the iterated Prisoner's Dilemma, but find that as the temptation to defect grows there are many other possible successful strategies. Other regions of the diagram reflect the equilibria structure of the underlying one-shot game, albeit often some non-expected strategies arise as well. We thus provide a thorough analysis of iterated 22 games from which we are able to extract some general conclusions. Our most relevant finding is that a great deal of the payoff parameter range can still be understood by focusing on win-stay, lose-shift strategies, and that very ambitious ones, aspiring to obtaining always a high payoff, are never evolutionary stable.

Evolution of cooperation facilitated by reinforcement learning with adaptive aspiration levels

Repeated interaction between individuals is the main mechanism for maintaining cooperation in social dilemma situations. Variants of tit-for-tat (repeating the previous action of the opponent) and the win-stay lose-shift strategy are known as strong competitors in iterated social dilemma games. On the other hand, real repeated interaction generally allows plasticity (i.e., learning) of individuals based on the experience of the past. Although plasticity is relevant to various biological phenomena, its role in repeated social dilemma games is relatively unexplored. In particular, if experience-based learning plays a key role in promotion and maintenance of cooperation, learners should evolve in the contest with nonlearners under selection pressure. By modeling players using a simple reinforcement learning model, we numerically show that learning enables the evolution of cooperation. We also show that numerically estimated adaptive dynamics appositely predict the outcome of evolutionary simulations. The analysis of the adaptive dynamics enables us to capture the obtained results as an affirmative example of the Baldwin effect, where learning accelerates the evolution to optimality.

TWO-STEP "WIN-STAY, LOSE-SHIFT" AND LEARNING TO COOPERATE IN THE REPEATED PRISONER'S DILEMMA

The standard win-stay, lose-shift behavior strategy in the repeated Prisoner's Dilemma game prescribes the players that win and lose in a current game round to keep and to change, respectively, their current actions, in the next round. Winning and losing are understood as receiving one of two upper values and one of two lower values, respectively, among the four admissible values for the players' benefits. In particular, a player acting as a cooperator against cooperation wins and therefore is not allowed to switch to defection in the next round with a hope to gain more (provided his/her rival keeps cooperating). This constraint can be viewed as too strong for a selfish player. Here, we discuss a two-step win-stay, lose-shift behavior that differs from the traditional win-stay lose-shift one in understanding of winning and losing. A player wins if his/her benefit is no smaller that in the previous round, and loses otherwise. This pattern is in a sense more selfish; in particular, a switch from cooperation (against cooperation) to defection is not forbidden. Another confirmation of a more selfish character of the two-step win-stay, lose-shift behavior, compared to the standard win-stay, lose-shift one, is that the former does not bring two individuals playing the repeated Prisoner's Dilemma game to mutual cooperation. In this paper, our goal is to understand to what degree one can relax the two-step win-stay, lose-shift behavior in selfishness so as to reach mutual cooperation, anyway. We deal with two models of the repeated Prisoner's Dilemma game — a game of two individuals and a game in a group of players. In the game of two individuals, a relaxed two-step win-stay, lose-shift behavior assumes that the players use mixed strategies; here, relaxation is associated with patience. In the game in a group of players, relaxation is achieved through conformity, a tendency to join the majority. We show that even a small degree of conformity is enough to teach a two-step win-stay, lose-shift group to cooperate.

Evolutionary dynamics, intrinsic noise, and cycles of cooperation

We use analytical techniques based on an expansion in the inverse system size to study the stochastic evolutionary dynamics of finite populations of players interacting in a repeated prisoner's dilemma game. We show that a mechanism of amplification of demographic noise can give rise to coherent oscillations in parameter regimes where deterministic descriptions converge to fixed points with complex eigenvalues. These quasicycles between cooperation and defection have previously been observed in computer simulations; here we provide a systematic and comprehensive analytical characterization of their properties. We are able to predict their power spectra as a function of the mutation rate and other model parameters and to compare the relative magnitude of the cycles induced by different types of underlying microscopic dynamics. We also extend our analysis to the iterated prisoner's dilemma game with a win-stay lose-shift strategy, appropriate in situations where players are subject to errors of the trembling-hand type.