Probabilistic Reinforcement Learning Research Articles

Altered learning from positive feedback in adolescents with anorexia nervosa.

Anorexia nervosa (AN) is characterized by severe restriction of calorie intake, which persists despite serious medical and psychological sequelae of starvation. Several prior studies have identified impaired feedback learning among individuals with AN, but whether it reflects a disturbance in learning from positive feedback (i.e., reward), negative feedback (i.e., punishment), or both, and the extent to which this impairment is related to severity and duration of illness, has not been clarified. Participants were female adolescents with AN (n = 76) and healthy teen volunteers (HC; n = 38) between the ages of 12-18 years who completed a probabilistic reinforcement learning task. A Bayesian reinforcement learning model was used to calculate separate learning rates for positive and negative feedback. Exploratory analyses examined associations between feedback learning and duration of illness, eating disorder severity, and self/parent reports of reward and punishment sensitivity. Adolescents with AN had a significantly lower rate of learning from positive feedback relative to HC. Patients and HC did not differ in learning from negative feedback or on overall task performance measures. Feedback learning parameters were not significantly associated with duration of illness, eating disorder severity, or questionnaire-based reports of reward and punishment sensitivity. Adolescents with AN showed a circumscribed deficit in learning from reward that was not associated with duration of illness or reported sensitivity to reward or punishment. Subsequent longitudinal research should explore whether differences in learning from positive feedback relate to course of illness in youth with AN.

Uncertainty of treatment efficacy moderates placebo effects on reinforcement learning

The placebo-reward hypothesis postulates that positive effects of treatment expectations on health (i.e., placebo effects) and reward processing share common neural underpinnings. Moreover, experiments in humans and animals indicate that reward uncertainty increases striatal dopamine, which is presumably involved in placebo responses and reward learning. Therefore, treatment uncertainty analogously to reward uncertainty may affect updating from rewards after placebo treatment. Here, we address whether different degrees of uncertainty regarding the efficacy of a sham treatment affect reward sensitivity. In an online between-subjects experiment with N = 141 participants, we systematically varied the provided efficacy instructions before participants first received a sham treatment that consisted of listening to binaural beats and then performed a probabilistic reinforcement learning task. We fitted a Q-learning model including two different learning rates for positive (gain) and negative (loss) reward prediction errors and an inverse gain parameter to behavioral decision data in the reinforcement learning task. Our results yielded an inverted-U-relationship between provided treatment efficacy probability and learning rates for gain, such that higher levels of treatment uncertainty, rather than of expected net efficacy, affect presumably dopamine-related reward learning. These findings support the placebo-reward hypothesis and suggest harnessing uncertainty in placebo treatment for recovering reward learning capabilities.

Reduced brain connectivity underlying value-based choices and outcomes in stimulant use disorder

BACKGROUND: Patients with stimulant use disorder (SUD) show impairments when making value-based choices that are associated with disruptions in neural processing across brain networks. Making optimal choices requires learning from outcomes to update knowledge and further optimise ongoing behaviour. The optimal functioning of neural systems that underpin the ability to make favourable choices is an essential component for life functioning, and successful recovery in patients with SUD. Therefore, we sought to investigate the neural processes that underpin value-based choices in SUD patients. We hypothesise that patients with SUD have reduced functional connectivity while making financial choices during a probabilistic reinforcement learning task. METHODS: We investigated connectivity associated with loss and reward value-based choices and their outcomes in patients with SUD and healthy control participants during a pharmacological magnetic resonance imaging study. Participants received a single dose of a dopamine receptor agonist, pramipexole, and a dopamine receptor antagonist, amisulpride, in a randomised, double-blind, placebo-controlled, balanced, crossover design. Functional task-related connectivity was analysed taking a whole brain connectomics approach to identify networks that are differentially modulated by dopaminergic receptor functioning. RESULTS: SUD patients showed widespread reductions in connectivity during both reward and loss value-based choices and outcomes, which were negatively correlated with the duration of stimulant drug use. Disturbances to functional brain connectivity in SUD patients during task performance were not modulated acutely by either amisulpride or pramipexole. CONCLUSIONS: Reductions in brain connectivity, particularly when making value-based choices and processing outcomes, may underlie learning impairments in SUD patients. Given that acute dopaminergic modulation did not improve brain connectivity impairments in SUD patients, it is likely that alternative treatments are needed.

Depth recordings of the mouse homologue of the Reward Positivity.

We recently advanced a rodent homologue for the reward-specific, event-related potential component observed in humans known as the Reward Positivity. We sought to determine the cortical source of this signal in mice to further test the nature of this homology. While similar reward-related cortical signals have been identified in rats, these recordings were all performed in cingulate gyrus. Given the value-dependent nature of this event, we hypothesized that more ventral prelimbic and infralimbic areas also contribute important variance to this signal. Depth probes assessed local field activity in 29 mice (15 males) while they completed multiple sessions of a probabilistic reinforcement learning task. Using a priori regions of interest, we demonstrated that the depth of recording in the cortical midline significantly correlated with the size of reward-evoked delta band spectral activity as well as the single trial correlation between delta power and reward prediction error. These findings provide important verification of the validity of this translational biomarker of reward responsiveness, learning, and valuation.

Impaired Punishment Learning in Conduct Disorder

Conduct disorder (CD) has been associated with deficits in the use of punishment to guide reinforcement learning (RL) and decision making. This may explain the poorly planned and often impulsive antisocial and aggressive behavior in affected youths. Here, we used a computational modeling approach to examine differences in RL abilities between CD youths and typically developing controls (TDCs). Specifically, we tested 2 competing hypotheses that RL deficits in CD reflect either reward dominance (also known as reward hypersensitivity) or punishment insensitivity (also known as punishment hyposensitivity). The study included 92 CD youths and 130 TDCs (aged 9-18 years, 48% girls) who completed a probabilistic RL task with reward, punishment, and neutral contingencies. Using computational modeling, we investigated the extent to which the 2 groups differed in their learning abilities to obtain reward and/or to avoid punishment. RL model comparisons showed that a model with separate learning rates per contingency explained behavioral performance best. Importantly, CD youths showed lower learning rates than TDCs specifically for punishment, whereas learning rates for reward and neutral contingencies did not differ. Moreover, callous-unemotional (CU) traits did not correlate with learning rates in CD. CD youths have a highly selective impairment in probabilistic punishment learning, regardless of their CU traits, whereas reward learning appears to be intact. In summary, our data suggest punishment insensitivity rather than reward dominance in CD. Clinically, the use of punishment-based intervention techniques to achieve effective discipline in patients with CD may be a less helpful strategy than reward-based techniques.

Guided probabilistic reinforcement learning for sampling-efficient maintenance scheduling of multi-component system

In recent years, multi-agent deep reinforcement learning has progressed rapidly as reflected by its increasing adoptions in industrial applications. This paper proposes a Guided Probabilistic Reinforcement Learning (Guided-PRL) model to tackle maintenance scheduling of multi-component systems in the presence of uncertainty with the goal of minimizing the overall life-cycle cost. The proposed Guided-PRL is deeply rooted in the Actor-Critic (AC) scheme. Since traditional AC falls short in sampling efficiency and suffers from getting stuck in local minima in the context of multi-agent reinforcement learning, it is thus challenging for the actor network to converge to a solution of desirable quality even when the critic network is properly configured. To address these issues, we develop a generic framework to facilitate effective training of the actor network, and the framework consists of environmental reward modeling, degradation formulation, state representation, and policy optimization. The convergence speed of the actor network is significantly improved with a guided sampling scheme for environment exploration by exploiting rules-based domain expert policies. To handle data scarcity, the environmental modeling and policy optimization are approximated with Bayesian models for effective uncertainty quantification. The Guided-PRL model is evaluated using the simulations of a 12-component system as well as GE90 and CFM56 engines. Compared with four alternative deep reinforcement learning schemes, the Guided-PRL lowers life-cycle cost by 34.92% to 88.07%. In comparison with rules-based expert policies, the Guided-PRL decreases the life-cycle cost by 23.26% to 51.36%.

L-DOPA and oxytocin influence the neurocomputational mechanisms of self-benefitting and prosocial reinforcement learning

Humans learn through reinforcement, particularly when outcomes are unexpected. Recent research suggests similar mechanisms drive how we learn to benefit other people, that is, how we learn to be prosocial. Yet the neurochemical mechanisms underlying such prosocial computations remain poorly understood. Here, we investigated whether pharmacological manipulation of oxytocin and dopamine influence the neurocomputational mechanisms underlying self-benefitting and prosocial reinforcement learning. Using a double-blind placebo-controlled cross-over design, we administered intranasal oxytocin (24 IU), dopamine precursor l-DOPA (100 mg + 25 mg carbidopa), or placebo over three sessions. Participants performed a probabilistic reinforcement learning task with potential rewards for themselves, another participant, or no one, during functional magnetic resonance imaging. Computational models of reinforcement learning were used to calculate prediction errors (PEs) and learning rates. Participants behavior was best explained by a model with different learning rates for each recipient, but these were unaffected by either drug. On the neural level, however, both drugs blunted PE signaling in the ventral striatum and led to negative signaling of PEs in the anterior mid-cingulate cortex, dorsolateral prefrontal cortex, inferior parietal gyrus, and precentral gyrus, compared to placebo, and regardless of recipient. Oxytocin (versus placebo) administration was additionally associated with opposing tracking of self-benefitting versus prosocial PEs in dorsal anterior cingulate cortex, insula and superior temporal gyrus. These findings suggest that both l-DOPA and oxytocin induce a context-independent shift from positive towards negative tracking of PEs during learning. Moreover, oxytocin may have opposing effects on PE signaling when learning to benefit oneself versus another.

Social feedback enhances learning in Williams syndrome

Williams syndrome (WS) is a rare genetic condition characterized by high social interest and approach motivation as well as intellectual disability and anxiety. Despite the fact that social stimuli are believed to have an increased intrinsic reward value in WS, it is not known whether this translates to learning and decision making. Genes homozygously deleted in WS are linked to sociability in the general population, making it a potential model condition for understanding the social brain. Probabilistic reinforcement learning was studied with either social or non-social rewards for correct choices. Social feedback improved learning in individuals with Williams syndrome but not in typically developing controls or individuals with other intellectual disabilities. Computational modeling indicated that these effects on social feedback were mediated by a shift towards higher weight given to rewards relative to punishments and increased choice consistency. We conclude that reward learning in WS is characterized by high volatility and a tendency to learn how to avoid punishment rather than how to gain rewards. Social feedback can partly normalize this pattern and promote adaptive reward learning.

Irrelevant Threats Linger and Affect Behavior in High Anxiety.

Threat-related information attracts attention and disrupts ongoing behavior, and particularly so for more anxious individuals. Yet, it is unknown how and to what extent threat-related information leave lingering influences on behavior (e.g., by impeding ongoing learning processes). Here, human male and female participants (N = 47) performed probabilistic reinforcement learning tasks where irrelevant distracting faces (neutral, happy, or fearful) were presented together with relevant monetary feedback. Behavioral modeling was combined with fMRI data (N = 27) to explore the neurocomputational bases of learning relevant and irrelevant information. In two separate studies, individuals with high trait anxiety showed increased avoidance of objects previously paired with the combination of neutral monetary feedback and fearful faces (but not neutral or happy faces). Behavioral modeling revealed that high anxiety increased the integration of fearful faces during feedback learning, and fMRI results (regarded as provisional, because of a relatively small sample size) further showed that variance in the prediction error signal, uniquely accounted for by fearful faces, correlated more strongly with activity in the right DLPFC for more anxious individuals. Behavioral and neuronal dissociations indicated that the threat-related distractors did not simply disrupt learning processes. By showing that irrelevant threats exert long-lasting influences on behavior, our results extend previous research that separately showed that anxiety increases learning from aversive feedbacks and distractibility by threat-related information. Our behavioral results, combined with the proposed neurocomputational mechanism, may help explain how increased exposure to irrelevant affective information contributes to the acquisition of maladaptive behaviors in more anxious individuals.SIGNIFICANCE STATEMENT In modern-day society, people are increasingly exposed to various types of irrelevant information (e.g., intruding social media announcements). Yet, the neurocomputational mechanisms influenced by irrelevant information during learning, and their interactions with increasingly distracted personality types are largely unknown. Using a reinforcement learning task, where relevant feedback is presented together with irrelevant distractors (emotional faces), we reveal an interaction between irrelevant threat-related information (fearful faces) and interindividual anxiety levels. fMRI shows provisional evidence for an interaction between anxiety levels and the coupling between activity in the DLPFC and learning signals specifically elicited by fearful faces. Our study reveals how irrelevant threat-related information may become entrenched in the anxious psyche and contribute to long-lasting abnormal behaviors.

Average reward rates enable motivational transfer across independent reinforcement learning tasks

Outcomes and feedbacks on performance may influence behavior beyond the context in which it was received, yet it remains unclear what neurobehavioral mechanisms may account for such lingering influences on behavior. The average reward rate (ARR) has been suggested to regulate motivated behavior, and was found to interact with dopamine-sensitive cognitive processes, such as vigilance and associative memory encoding. The ARR could therefore provide a bridge between independent tasks when these are performed in temporal proximity, such that the reward rate obtained in one task could influence performance in a second subsequent task. Reinforcement learning depends on the coding of prediction error signals by dopamine neurons and their downstream targets, in particular the nucleus accumbens. Because these brain regions also respond to changes in ARR, reinforcement learning may be vulnerable to changes in ARR. To test this hypothesis, we designed a novel paradigm in which participants (n = 245) performed two probabilistic reinforcement learning tasks presented in interleaved trials. The ARR was controlled by an "induction" task which provided feedback with a low (p = 0.58), a medium (p = 0.75), or a high probability of reward (p = 0.92), while the impact of ARR on reinforcement learning was tested by a second "reference" task with a constant reward probability (p = 0.75). We find that performance was significantly lower in the reference task when the induction task provided low reward probabilities (i.e., during low levels of ARR), as compared to the medium and high ARR conditions. Behavioral modeling further revealed that the influence of ARR is best described by models which accumulates average rewards (rather than average prediction errors), and where the ARR directly modulates the prediction error signal (rather than affecting learning rates or exploration). Our results demonstrate how affective information in one domain may transfer and affect motivated behavior in other domains. These findings are particularly relevant for understanding mood disorders, but may also inform abnormal behaviors attributed to dopamine dysfunction.

Hippocampal Contribution to Probabilistic Feedback Learning: Modeling Observation- and Reinforcement-based Processes.

Simple probabilistic reinforcement learning is recognized as a striatum-based learning system, but in recent years, has also been associated with hippocampal involvement. This study examined whether such involvement may be attributed to observation-based learning (OL) processes, running in parallel to striatum-based reinforcement learning. A computational model of OL, mirroring classic models of reinforcement-based learning (RL), was constructed and applied to the neuroimaging data set of Palombo, Hayes, Reid, and Verfaellie [2019. Hippocampal contributions to value-based learning: Converging evidence from fMRI and amnesia. Cognitive, Affective & Behavioral Neuroscience, 19(3), 523-536]. Results suggested that OL processes may indeed take place concomitantly to reinforcement learning and involve activation of the hippocampus and central orbitofrontal cortex. However, rather than independent mechanisms running in parallel, the brain correlates of the OL and RL prediction errors indicated collaboration between systems, with direct implication of the hippocampus in computations of the discrepancy between the expected and actual reinforcing values of actions. These findings are consistent with previous accounts of a role for the hippocampus in encoding the strength of observed stimulus-outcome associations, with updating of such associations through striatal reinforcement-based computations. In addition, enhanced negative RL prediction error signaling was found in the anterior insula with greater use of OL over RL processes. This result may suggest an additional mode of collaboration between the OL and RL systems, implicating the error monitoring network.

Probabilistic Reinforcement Learning and Anhedonia.

Despite the prominence of anhedonic symptoms associated with diverse neuropsychiatric conditions, there are currently no approved therapeutics designed to attenuate the loss of responsivity to previously rewarding stimuli. However, the search for improved treatment options for anhedonia has been reinvigorated by a recent reconceptualization of the very construct of anhedonia, including within the Research Domain Criteria (RDoC) initiative. This chapter will focus on the RDoC Positive Valence Systems construct of reward learning generally and sub-construct of probabilistic reinforcement learning specifically. The general framework emphasizes objective measurement of a subject's responsivity to reward via reinforcement learning under asymmetrical probabilistic contingencies as a means to quantify reward learning. Indeed, blunted reward responsiveness and reward learning are central features of anhedonia and have been repeatedly described in major depression. Moreover, these probabilistic reinforcement techniques can also reveal neurobiological mechanisms to aid development of innovative treatment approaches. In this chapter, we describe how investigating reward learning can improve our understanding of anhedonia via the four RDoC-recommended tasks that have been used to probe sensitivity to probabilistic reinforcement contingencies and how such task performance is disrupted in various neuropsychiatric conditions. We also illustrate how reverse translational approaches of probabilistic reinforcement assays in laboratory animals can inform understanding of pharmacological and physiological mechanisms. Next, we briefly summarize the neurobiology of probabilistic reinforcement learning, with a focus on the prefrontal cortex, anterior cingulate cortex, striatum, and amygdala. Finally, we discuss treatment implications and future directions in this burgeoning area.

Age-Energy Tradeoff Optimization for Packet Delivery in Fading Channels

Power control policies that minimize the weighted combination of age of information (AoI) and total energy consumption are studied in this paper. It is assumed that the status update information is acquired at the transmitter through sensors and then sent in packets to the receiver at fixed rate over Rayleigh fading channels. Retransmission mechanism is introduced to guarantee the reliability of the received update packets, and a limit on the maximum number of transmission rounds is imposed. On the one hand, with the channel distribution information (CDI) available at the transmitter, the age-energy tradeoff is optimized by formulating the problem as a constrained Markov decision process (CMDP) which takes into account the sensing power consumption as well as the average transmission power constraint. Employing the Lagrangian relaxation, the CMDP problem is transformed into an unconstrained Markov decision process (MDP) problem. A threshold-based policy is also provided. Subsequently, a dynamic programming approach is proposed to obtain the optimal power allocation policy. On the other hand, when the environment is unknown, two probabilistic reinforcement learning algorithms are proposed to optimize the age-energy tradeoff. Through simulation results, it is shown that the age-energy tradeoffs can be improved by the proposed optimal policies compared with benchmark schemes.

The Role of Dopaminergic Genes in Probabilistic Reinforcement Learning in Schizophrenia Spectrum Disorders.

Schizophrenia spectrum disorders (SZ) are characterized by impairments in probabilistic reinforcement learning (RL), which is associated with dopaminergic circuitry encompassing the prefrontal cortex and basal ganglia. However, there are no studies examining dopaminergic genes with respect to probabilistic RL in SZ. Thus, the aim of our study was to examine the impact of dopaminergic genes on performance assessed by the Probabilistic Selection Task (PST) in patients with SZ in comparison to healthy control (HC) subjects. In our study, we included 138 SZ patients and 188 HC participants. Genetic analysis was performed with respect to the following genetic polymorphisms: rs4680 in COMT, rs907094 in DARP-32, rs2734839, rs936461, rs1800497, and rs6277 in DRD2, rs747302 and rs1800955 in DRD4 and rs28363170 and rs2975226 in DAT1 genes. The probabilistic RL task was completed by 59 SZ patients and 95 HC subjects. SZ patients performed significantly worse in acquiring reinforcement contingencies during the task in comparison to HCs. We found no significant association between genetic polymorphisms and RL among SZ patients; however, among HC participants with respect to the DAT1 rs28363170 polymorphism, individuals with 10-allele repeat genotypes performed better in comparison to 9-allele repeat carriers. The present study indicates the relevance of the DAT1 rs28363170 polymorphism in RL in HC participants.

Increased Ventromedial Prefrontal Cortex Activity in Adolescence Benefits Prosocial Reinforcement Learning

Learning which of our behaviors benefit others contributes to forming social relationships. An important period for the development of (pro)social behavior is adolescence, which is characterized by transitions in social connections. It is, however, unknown how learning to benefit others develops across adolescence and what the underlying cognitive and neural mechanisms are. In this functional neuroimaging study, we assessed learning for self and others (i.e., prosocial learning) and the concurring neural tracking of prediction errors across adolescence (ages 9–21, N = 74). Participants performed a two-choice probabilistic reinforcement learning task in which outcomes resulted in monetary consequences for themselves, an unknown other, or no one. Participants from all ages were able to learn for themselves and others, but learning for others showed a more protracted developmental trajectory. Prediction errors for self were observed in the ventral striatum and showed no age-related differences. However, prediction error coding for others showed an age-related increase in the ventromedial prefrontal cortex. These results reveal insights into the computational mechanisms of learning for others across adolescence, and highlight that learning for self and others show different age-related patterns.

Delaying feedback compensates for impaired reinforcement learning in developmental dyslexia

A theoretical framework suggests that developmental dyslexia is characterized by abnormalities in brain structures underlying the procedural learning and memory systems while the declarative learning and memory systems are presumed to remain intact or even enhanced (Procedural Deficit Hypothesis). This notion has been supported by a substantial body of research, which focused on each system independently. However, less attention has been paid to interactions between these memory systems which may provide insights as to learning situations and conditions in which learning in dyslexia can be improved. The current study was undertaken to examine these important but unresolved issues. To this end, probabilistic reinforcement learning and episodic memory tasks were examined in participants with dyslexia and neurotypicals simultaneously within a single task. Feedback timing presentation was manipulated, building on prior research indicating that delaying feedback timing shifts striatal-based probabilistic learning, to become more hippocampal-dependent. It was hypothesized that if the procedural learning and memory systems are impaired in dyslexia, performance will be impaired under conditions that encourage procedural memory engagement (immediate feedback trials) but not under conditions that promote declarative memory processing (long delayed feedback trials). It was also predicted that the ability to incidentally acquire episodic information would be preserved in dyslexia. The results supported these predictions. Participants with dyslexia were impaired in probabilistic learning of cue-outcome associations compared to neurotypicals in an immediate feedback condition, but not when feedback on choices was presented after a long delay. Furthermore, participants with dyslexia demonstrated similar performance to neurotypicals in a task requiring incidental episodic memory formation. These findings attest to a dissociation between procedural-based and declarative-based learning in developmental dyslexia within a single task, a finding that adds discriminative validity to the Procedural Deficit Hypothesis. Just as important, the present findings suggest that training conditions designed to shift the load from midbrain/striatal systems to declarative memory mechanisms have the potential to compensate for impaired learning in developmental dyslexia.

HIV Transgenic Rats Demonstrate Superior Task Acquisition and Intact Reversal Learning in the Within-Session Probabilistic Reversal Learning Task.

The HIV transgenic (HIVtg) rat is a commonly used animal model of chronic HIV infection that exhibits a wide range of cognitive deficits. To date, relatively little work has been conducted on these rats' capacity for reversal learning, an assay of executive function and cognitive flexibility used in humans. The present study sought todetermine the impact of HIV genotype on probabilistic reversal learning, effortful motivation, and spontaneous locomotion/exploration in rats. Male (n = 8) and female (n = 8) HIVtg rats and wildtype (WT) controlswere utilized. Cognitive flexibility was assessed via the Probabilistic Reversal Learning Task (PRLT), which reinforced responses to two stimuli on differential probabilistic schedules that periodically reversed. Effortful motivation and locomotor/exploratory behavior were assessed via the Progressive Ratio Breakpoint Task (PRBT) and the Behavioral Pattern Monitor (BPM), respectively. Regardless of sex, HIVtg rats required fewer trials to ascertain initial PRLT reward schedules than WT rats, and completed the same number of reversals. Secondary behaviors suggested that HIVtg PRLT performance was facilitated by a speed-accuracy tradeoff strategy. No main or interactive effects of genotype were observed in the PRBT or BPM. Relative to WT controls, HIVtg rats exhibited superior probabilistic reinforcement learning. Reversal learning was unaffected by HIV genotype, as was effortful motivation and exploratory behavior. These findings contrast with previous characterizations of the HIVtg rat, thus indicating a nuanced cognitive profile that is dependent upon suchtask specifications as within- versus between-session assessment and probabilistic versus deterministic reward schedules.

A causal role of estradiol in human reinforcement learning

The sex hormone estradiol is hypothesized to play a key role in human cognition, and reward processing specifically, via increased dopamine D1-receptor signalling.However, the effect of estradiol on reward processing in men has never been established. To fill this gap, we performed a double-blind placebo-controlled study in which men (N = 100) received either a single dose of estradiol (2 mg) or a placebo. Subjects performed a probabilistic reinforcement learning task where they had to choose between two options with varying reward probabilities to maximize monetary reward. Results showed that estradiol administration increased reward sensitivity compared to placebo. This effect was observed in subjects' choices, how much weight they assigned to their previous choices, and subjective reports about the reward probabilities. Furthermore, effects of estradiol were moderated by reward sensitivity, as measured through the BIS/BAS questionnaire. Using reinforcement learning models, we found that behavioral effects of estradiol were reflected in increased learning rates. These results demonstrate a causal role of estradiol within the framework of reinforcement learning, by enhancing reward sensitivity and learning. Furthermore, they provide preliminary evidence for dopamine-related genetic variants moderating the effect of estradiol on reward processing.

Modeling changes in probabilistic reinforcement learning during adolescence.

In the real world, many relationships between events are uncertain and probabilistic. Uncertainty is also likely to be a more common feature of daily experience for youth because they have less experience to draw from than adults. Some studies suggest probabilistic learning may be inefficient in youths compared to adults, while others suggest it may be more efficient in youths in mid adolescence. Here we used a probabilistic reinforcement learning task to test how youth age 8-17 (N = 187) and adults age 18-30 (N = 110) learn about stable probabilistic contingencies. Performance increased with age through early-twenties, then stabilized. Using hierarchical Bayesian methods to fit computational reinforcement learning models, we show that all participants’ performance was better explained by models in which negative outcomes had minimal to no impact on learning. The performance increase over age was driven by 1) an increase in learning rate (i.e. decrease in integration time scale); 2) a decrease in noisy/exploratory choices. In mid-adolescence age 13-15, salivary testosterone and learning rate were positively related. We discuss our findings in the context of other studies and hypotheses about adolescent brain development.

Impaired Learning From Negative Feedback in Stimulant Use Disorder: Dopaminergic Modulation.

BackgroundDrug-induced alterations to the dopamine system in stimulant use disorder (SUD) are hypothesized to impair reinforcement learning (RL). Computational modeling enables the investigation of the latent processes of RL in SUD patients, which could elucidate the nature of their impairments.MethodsWe investigated RL in 44 SUD patients and 41 healthy control participants using a probabilistic RL task that assesses learning from reward and punishment separately. In an independent sample, we determined the modulatory role of dopamine in RL following a single dose of the dopamine D2/3 receptor antagonist amisulpride (400 mg) and the agonist pramipexole (0.5 mg) in a randomised, double-blind, placebo-controlled, crossover design. We analyzed task performance using computational modelling and hypothesized that RL impairments in SUD patients would be differentially modulated by a dopamine D2/3 receptor antagonist and agonist.ResultsComputational analyses in both samples revealed significantly reduced learning rates from punishment in SUD patients compared with healthy controls, whilst their reward learning rates were not measurably impaired. In addition, the dopaminergic receptor agents modulated RL parameters differentially in both groups. Both amisulpride and pramipexole impaired RL parameters in healthy participants, but ameliorated learning from punishment in SUD patients.ConclusionOur findings suggest that RL impairments seen in SUD patients are associated with altered dopamine function.