Counterfactual Feedback Research Articles

The effect of reward prediction errors on subjective affect depends on outcome valence and decision context.

The valence of an individual's emotional response to an event is often thought to depend on their prior expectations for the event: better-than-expected outcomes produce positive affect and worse-than-expected outcomes produce negative affect. In recent years, this hypothesis has been instantiated within influential computational models of subjective affect that assume the valence of affect is driven by reward prediction errors. However, there remain a number of open questions regarding this association. In this project, we investigated the moderating effects of outcome valence and decision context (Experiment 1: free vs. forced choices; Experiment 2: trials with vs. trials without counterfactual feedback) on the effects of reward prediction errors on subjective affect. We conducted two online experiments (N = 300 in total) of general-population samples recruited via Prolific to complete a risky decision-making task with an embedded high-resolution sampling of subjective affect. Hierarchical Bayesian computational modeling revealed that both outcome amount and reward prediction errors influenced subjective affect, but that the effects of reward prediction errors were moderated by both outcome valence and decision context. Specifically, we found evidence that only negative reward prediction errors (worse-than-expected outcomes) influenced subjective affect, with no significant effect of positive reward prediction errors (better-than-expected outcomes). Moreover, these effects were only apparent on trials in which participants made a choice freely (but not on forced-choice trials) and when counterfactual feedback was absent (but not when counterfactual feedback was present). These results deepen our understanding of the effects of reward prediction errors on subjective affect. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Statistical information about reward timing is insufficient for promoting optimal persistence decisions

When deciding how long to keep waiting for delayed rewards that will arrive at an uncertain time, different distributions of possible reward times dictate different optimal strategies for maximizing reward. When reward timing distributions are heavy-tailed (e.g., waiting on hold) there is a point at which waiting is no longer advantageous because the opportunity cost of waiting is too high. Alternatively, when reward timing distributions have more predictable timing (e.g., uniform), it is advantageous to wait as long as necessary for the reward. Although people learn to approximate optimal strategies, little is known about how this learning occurs. One possibility is that people learn a general cognitive representation of the probability distribution that governs reward timing and then infer a strategy from that model of the environment. Another possibility is that they learn an action policy in a way that depends more narrowly on direct task experience, such that general knowledge of the reward timing distribution is insufficient for expressing the optimal strategy. Here, in a series of studies in which participants decided how long to persist for delayed rewards before quitting, we provided participants with information about the reward timing distribution in several ways. Whether the information was provided through counterfactual feedback (Study 1), previous exposure (Studies 2a and 2b), or description (Studies 3a and 3b), it did not obviate the need for direct, feedback-driven learning in a decision context. Therefore, learning when to quit waiting for delayed rewards might depend on task-specific experience, not solely on probabilistic reasoning.

Neural responses in macaque prefrontal cortex are linked to strategic exploration.

Humans have been shown to strategically explore. They can identify situations in which gathering information about distant and uncertain options is beneficial for the future. Because primates rely on scarce resources when they forage, they are also thought to strategically explore, but whether they use the same strategies as humans and the neural bases of strategic exploration in monkeys are largely unknown. We designed a sequential choice task to investigate whether monkeys mobilize strategic exploration based on whether information can improve subsequent choice, but also to ask the novel question about whether monkeys adjust their exploratory choices based on the contingency between choice and information, by sometimes providing the counterfactual feedback about the unchosen option. We show that monkeys decreased their reliance on expected value when exploration could be beneficial, but this was not mediated by changes in the effect of uncertainty on choices. We found strategic exploratory signals in anterior and mid-cingulate cortex (ACC/MCC) and dorsolateral prefrontal cortex (dlPFC). This network was most active when a low value option was chosen, which suggests a role in counteracting expected value signals, when exploration away from value should to be considered. Such strategic exploration was abolished when the counterfactual feedback was available. Learning from counterfactual outcome was associated with the recruitment of a different circuit centered on the medial orbitofrontal cortex (OFC), where we showed that monkeys represent chosen and unchosen reward prediction errors. Overall, our study shows how ACC/MCC-dlPFC and OFC circuits together could support exploitation of available information to the fullest and drive behavior towards finding more information through exploration when it is beneficial.

Layer-specific profiles of factual and counterfactual feedback signals in human early visual cortex during navigation – 7T fMRI study

Layer-specific profiles of factual and counterfactual feedback signals in human early visual cortex during navigation – 7T fMRI study

Decreased transfer of value to action in Tourette syndrome

ObjectiveTourette syndrome is a neurodevelopmental disorder putatively associated with a hyperdopaminergic state. Therefore, it seems plausible that excessive dopamine transmission in Tourette syndrome alters the ability to learn based on rewards and punishments. We tested whether Tourette syndrome patients exhibited altered reinforcement learning and corresponding feedback-related EEG deflections. MethodsWe used a reinforcement learning task providing factual and counterfactual feedback in a sample of 15 Tourette syndrome patients and matched healthy controls whilst recording EEG. The paradigm presented various reward probabilities to enforce adaptive adjustments. We employed a computational model to derive estimates of the prediction error, which we used for single-trial regression analysis of the EEG data. ResultsWe found that Tourette syndrome patients showed increased choice stochasticity compared to controls. The feedback-related negativity represented an axiomatic prediction error for factual feedback and did not differ between groups. We observed attenuated P3a modulation specifically for factual feedback in Tourette syndrome patients, representing impaired coding of attention allocation. ConclusionOur findings indicate that cortical prediction error coding is unaffected by Tourette syndrome. Nonetheless, the transfer of learned values into choice formation is degraded, in line with a hyperdopaminergic state.

When choosing means losing: Regret enhances repetitive negative thinking in high brooders

Past research suggests a relation between rumination (i.e., a form of repetitive negative thinking and a well-established vulnerability factor for depression) and regret (i.e., negative emotions containing self-blame, connected to cognitions about how past personal actions might have achieved better outcomes). However, these relations have not yet been investigated in experimental designs, and it has not been investigated how regret is related to both trait and state components of rumination. Therefore, in the present study we examined the temporal dynamics of repetitive negative thinking (state) in relation to regret, and how this is affected by interindividual differences in tendencies to engage in ruminative brooding (trait). Healthy females (N = 69) performed a sequential risk-taking task which included counterfactual feedback about alternative personal choice outcomes, and in which they experienced goal nonattainment (i.e., no monetary reward). Repetitive negative thoughts (state) prior to the task, and the tendency for ruminative brooding (trait), were both independently associated with reported regret after goal nonattainment, suggesting that both state and trait components of repetitive negative thinking render individuals more prone to experience regret. Furthermore, there was an interaction between regret after goal nonattainment and brooding (trait), in the sense that regret contributed to increases in (state) repetitive negative thinking, only among high brooders. The current results suggest that, among brooding-inclined individuals, the experience of regret may be a specific mechanism in aggravating repetitive negative thinking, which in turn may contribute to the development of psychopathology.

Contrasting temporal difference and opportunity cost reinforcement learning in an empirical money-emergence paradigm

Money is a fundamental and ubiquitous institution in modern economies. However, the question of its emergence remains a central one for economists. The monetary search-theoretic approach studies the conditions under which commodity money emerges as a solution to override frictions inherent to interindividual exchanges in a decentralized economy. Although among these conditions, agents' rationality is classically essential and a prerequisite to any theoretical monetary equilibrium, human subjects often fail to adopt optimal strategies in tasks implementing a search-theoretic paradigm when these strategies are speculative, i.e., involve the use of a costly medium of exchange to increase the probability of subsequent and successful trades. In the present work, we hypothesize that implementing such speculative behaviors relies on reinforcement learning instead of lifetime utility calculations, as supposed by classical economic theory. To test this hypothesis, we operationalized the Kiyotaki and Wright paradigm of money emergence in a multistep exchange task and fitted behavioral data regarding human subjects performing this task with two reinforcement learning models. Each of them implements a distinct cognitive hypothesis regarding the weight of future or counterfactual rewards in current decisions. We found that both models outperformed theoretical predictions about subjects' behaviors regarding the implementation of speculative strategies and that the latter relies on the degree of the opportunity costs consideration in the learning process. Speculating about the marketability advantage of money thus seems to depend on mental simulations of counterfactual events that agents are performing in exchange situations.

Emotional responses to behavioral economic incentives for health behavior change

Many people aim to change their lifestyle, but have trouble acting on their intentions. Behavioral economic incentives and related emotions can support commitment to personal health goals, but the related emotions remain unexplored. In a regret lottery, winners who do not attain their health goals do not get their prize but receive feedback on what their forgone earnings would have been. This counterfactual feedback should provoke anticipated regret and increase commitment to health goals. We explored which emotions were actually expected upon missing out on a prize due to unsuccessful weight loss and which incentive-characteristics influence their likelihood and intensity. Participants reported their expected emotional response after missing out on a prize in one of 12 randomly presented incentive-scenarios, which varied in incentive type, incentive size and deadline distance. Participants primarily reported feeling disappointment, followed by regret. Regret was expected most when losing a lottery prize (vs. a fixed incentive) and intensified with prize size. Multiple features of the participant and the lottery incentive increase the occurrence and intensity of regret. As such, our findings can be helpful in designing behavioral economic incentives that leverage emotions to support health behavior change.

Commitment Lotteries Promote Physical Activity Among Overweight Adults-A Cluster Randomized Trial.

The World Health Organization has identified physical inactivity as the fourth leading risk factor for global mortality. People often intend to engage in physical activity on a regular basis, but have trouble doing so. To realize their health goals, people can voluntarily accept deadlines with consequences that restrict undesired future behaviors (i.e., commitment devices). We examined if lottery-based deadlines that leverage regret aversion would help overweight individuals in attaining their goal of attending their gym twice per week. At each deadline a lottery winner was drawn from all participants. The winners were only eligible for their prize if they attained their gym-attendance goals. Importantly, nonattending lottery winners were informed about their forgone prize. The promise of this counterfactual feedback was designed to evoke anticipated regret and emphasize the deadlines. Six corporate gyms with a total of 163 overweight participants were randomized to one of three arms. We compared (i) weekly short-term lotteries for 13 weeks; (ii) the same short-term lotteries in combination with an additional long-term lottery after 26 weeks; and (iii) a control arm without lotteries. After 13 weeks, participants in the lottery arms attained their attendance goals more often than participants in the control arm. After 26 weeks, we observe a decline in goal attainment in the short-term lottery arm and the highest goal attainment in the long-term lottery arm. With novel applications, the current research adds to a growing body of research that demonstrates the effectiveness of commitment devices in closing the gap between health goals and behavior. This trial is registered in the Dutch Trial Register. Identifier: NTR5559.

Emotional Responses to Behavioral Economic Incentives for Health Behavior Change

Many people aim to change their lifestyle, but have trouble acting on their intentions. Behavioral economic incentives and related emotions can support commitment to personal health goals, but the related emotions remain unexplored. In a regret lottery, winners who do not attain their health goals do not get their prize but receive feedback on what their forgone earnings would have been. This counterfactual feedback should provoke anticipated regret and increase commitment to health goals. We explored which emotions were actually expected upon missing out on a prize due to unsuccessful weight loss and which incentive-characteristics influence their likelihood and intensity. Participants reported their expected emotional response after missing out on a prize in one of 12 randomly presented incentive-scenarios, which varied in incentive type, incentive size and deadline distance. Participants primarily reported feeling disappointment, followed by regret. Regret was expected most when losing a lottery prize (vs. a fixed incentive) and intensified with prize size. Multiple features of the participant and the lottery incentive increase the occurrence and intensity of regret. As such, our findings can be helpful in designing behavioral economic incentives that leverage emotions to support health behavior change.

Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing.

Previous studies suggest that factual learning, that is, learning from obtained outcomes, is biased, such that participants preferentially take into account positive, as compared to negative, prediction errors. However, whether or not the prediction error valence also affects counterfactual learning, that is, learning from forgone outcomes, is unknown. To address this question, we analysed the performance of two groups of participants on reinforcement learning tasks using a computational model that was adapted to test if prediction error valence influences learning. We carried out two experiments: in the factual learning experiment, participants learned from partial feedback (i.e., the outcome of the chosen option only); in the counterfactual learning experiment, participants learned from complete feedback information (i.e., the outcomes of both the chosen and unchosen option were displayed). In the factual learning experiment, we replicated previous findings of a valence-induced bias, whereby participants learned preferentially from positive, relative to negative, prediction errors. In contrast, for counterfactual learning, we found the opposite valence-induced bias: negative prediction errors were preferentially taken into account, relative to positive ones. When considering valence-induced bias in the context of both factual and counterfactual learning, it appears that people tend to preferentially take into account information that confirms their current choice.

Specific effect of a dopamine partial agonist on counterfactual learning: evidence from Gilles de la Tourette syndrome

The dopamine partial agonist aripiprazole is increasingly used to treat pathologies for which other antipsychotics are indicated because it displays fewer side effects, such as sedation and depression-like symptoms, than other dopamine receptor antagonists. Previously, we showed that aripiprazole may protect motivational function by preserving reinforcement-related signals used to sustain reward-maximization. However, the effect of aripiprazole on more cognitive facets of human reinforcement learning, such as learning from the forgone outcomes of alternative courses of action (i.e., counterfactual learning), is unknown. To test the influence of aripiprazole on counterfactual learning, we administered a reinforcement learning task that involves both direct learning from obtained outcomes and indirect learning from forgone outcomes to two groups of Gilles de la Tourette (GTS) patients, one consisting of patients who were completely unmedicated and the other consisting of patients who were receiving aripiprazole monotherapy, and to healthy subjects. We found that whereas learning performance improved in the presence of counterfactual feedback in both healthy controls and unmedicated GTS patients, this was not the case in aripiprazole-medicated GTS patients. Our results suggest that whereas aripiprazole preserves direct learning of action-outcome associations, it may impair more complex inferential processes, such as counterfactual learning from forgone outcomes, in GTS patients treated with this medication.

The Computational Development of Reinforcement Learning during Adolescence.

Adolescence is a period of life characterised by changes in learning and decision-making. Learning and decision-making do not rely on a unitary system, but instead require the coordination of different cognitive processes that can be mathematically formalised as dissociable computational modules. Here, we aimed to trace the developmental time-course of the computational modules responsible for learning from reward or punishment, and learning from counterfactual feedback. Adolescents and adults carried out a novel reinforcement learning paradigm in which participants learned the association between cues and probabilistic outcomes, where the outcomes differed in valence (reward versus punishment) and feedback was either partial or complete (either the outcome of the chosen option only, or the outcomes of both the chosen and unchosen option, were displayed). Computational strategies changed during development: whereas adolescents’ behaviour was better explained by a basic reinforcement learning algorithm, adults’ behaviour integrated increasingly complex computational features, namely a counterfactual learning module (enabling enhanced performance in the presence of complete feedback) and a value contextualisation module (enabling symmetrical reward and punishment learning). Unlike adults, adolescent performance did not benefit from counterfactual (complete) feedback. In addition, while adults learned symmetrically from both reward and punishment, adolescents learned from reward but were less likely to learn from punishment. This tendency to rely on rewards and not to consider alternative consequences of actions might contribute to our understanding of decision-making in adolescence.

Financial Sector Reforms in Nigeria: Issues and Challenges

One of the most challenging debates of modern history is whether financial development causes economicgrowth or a consequence of economic activity. There are on-going reforms in the sector in Nigeria, thus theimportance of a review of the various reforms. Evidence as to whether and how reforms are remedying thetraditional weaknesses of the Financial Sector is so far limited in Nigeria. Ongoing reforms to improve banks’corporate governance and internal systems suggest that the prospects for the financial sector to performprofitably and prudently, while reducing volatility in the system exist. The paper adopts an empirical reviewapproach for its analysis. Paper suggests therefore, that the present reforms be reviewed and sustained in anorderly manner, for appropriate channeling of resources for investment and productive purposes. Efforts shouldbe concentrated on the linkages of the sector with macro accounts and where financial development appears tohave been the weakest. Furthermore, advancement of the financial sector vis-a-vis instruments should be theprimary focus for the authorities. A counterfactual feedback mechanism should also be integrated within thefinancial sector for an appropriate signaling for the economic productive base.