Aversive Prediction Error Research Articles

Threat learning in space: how stimulus-outcome spatial compatibility modulates conditioned skin conductance response

A central question in Pavlovian conditioning concerns the critical conditions that drive the acquisition and maintenance of the stimulus-outcome association. The spatial relationship between the conditioned (CS) and unconditioned (US) stimuli is considered to exert strong effects on learning. However, how spatial information modulates Pavlovian learning remains mostly unexplored in humans. Here, we test how the compatibility between the CS and the US location influences the acquisition, extinction, and recovery (following reinstatement) of Pavlovian conditioned threat. Participants (N = 20) completed a differential threat conditioning task in which visual CSs appeared on the same (compatible) or opposite (incompatible) hemispace as the US delivery (aversive shock to one hand), while their skin conductance response served as an index of learning. Results show that initial threat expectations were biased in favor of compatible CSs before conditioning. Nevertheless, this bias was revised during acquisition to reflect current stimulus-outcome contingencies. Computational modeling suggested that this effect occurred through a higher reliance on positive aversive prediction errors for incompatible CSs, thereby facilitating learning of their association with the US. Additionally, the conditioned response to incompatible CSs was associated with initially slower extinction and a greater recovery after threat reinstatement. These findings suggest that spatial information conveyed by stimuli and outcomes can be flexibly used to enact defensive responses to the current source of danger, highlighting the adaptive nature of Pavlovian learning.

Positioning the brainstem within the neural network of threat prediction

In a recent study, Strickland and McDannald dissected the role of brainstem networks in threat prediction. Using probabilistic threat discrimination in rats, the authors demonstrated that brainstem neurons estimate threat probability and generate positive aversive prediction errors after unexpected outcomes. Their findings suggest that, beyond organizing defensive behaviors, brainstem neurons are involved in threat prediction computations.

Nucleus accumbens dopamine tracks aversive stimulus duration and prediction but not value or prediction error.

There is active debate on the role of dopamine in processing aversive stimuli, where inferred roles range from no involvement at all, to signaling an aversive prediction error (APE). Here, we systematically investigate dopamine release in the nucleus accumbens core (NAC), which is closely linked to reward prediction errors, in rats exposed to white noise (WN, a versatile, underutilized, aversive stimulus) and its predictive cues. Both induced a negative dopamine ramp, followed by slow signal recovery upon stimulus cessation. In contrast to reward conditioning, this dopamine signal was unaffected by WN value, context valence, or probabilistic contingencies, and the WN dopamine response shifted only partially toward its predictive cue. However, unpredicted WN provoked slower post-stimulus signal recovery than predicted WN. Despite differing signal qualities, dopamine responses to simultaneous presentation of rewarding and aversive stimuli were additive. Together, our findings demonstrate that instead of an APE, NAC dopamine primarily tracks prediction and duration of aversive events.

Asymmetric representation of aversive prediction errors in Pavlovian threat conditioning

Survival in biological environments requires learning associations between predictive sensory cues and threatening outcomes. Such aversive learning may be implemented through reinforcement learning algorithms that are driven by the signed difference between expected and encountered outcomes, termed prediction errors (PEs). While PE-based learning is well established for reward learning, the role of putative PE signals in aversive learning is less clear. Here, we used functional magnetic resonance imaging in humans (21 healthy men and women) to investigate the neural representation of PEs during maintenance of learned aversive associations. Four visual cues, each with a different probability (0, 33, 66, 100%) of being followed by an aversive outcome (electric shock), were repeatedly presented to participants. We found that neural activity at omission (US-) but not occurrence of the aversive outcome (US+) encoded PEs in the medial prefrontal cortex. More expected omission of aversive outcome was associated with lower neural activity. No neural signals fulfilled axiomatic criteria, which specify necessary and sufficient components of PE signals, for signed PE representation in a whole-brain search or in a-priori regions of interest. Our results might suggest that, different from reward learning, aversive learning does not involve signed PE signals that are represented within the same brain region for all conditions.

The Roles of Basolateral Amygdala Parvalbumin Neurons in Fear Learning.

The basolateral amygdala (BLA) is obligatory for fear learning. This learning is linked to BLA excitatory projection neurons whose activity is regulated by complex networks of inhibitory interneurons, dominated by parvalbumin (PV)-expressing GABAergic neurons. The roles of these GABAergic interneurons in learning to fear and learning not to fear, activity profiles of these interneurons across the course of fear learning, and whether or how these change across the course of learning all remain poorly understood. Here, we used PV cell-type-specific recording and manipulation approaches in male transgenic PV-Cre rats during pavlovian fear conditioning to address these issues. We show that activity of BLA PV neurons during the moments of aversive reinforcement controls fear learning about aversive events, but activity during moments of nonreinforcement does not control fear extinction learning. Furthermore, we show expectation-modulation of BLA PV neurons during fear learning, with greater activity to an unexpected than expected aversive unconditioned stimulus (US). This expectation-modulation was specifically because of BLA PV neuron sensitivity to aversive prediction error. Finally, we show that BLA PV neuron function in fear learning is conserved across these variations in prediction error. We suggest that aversive prediction-error modulation of PV neurons could enable BLA fear-learning circuits to retain selectivity for specific sensory features of aversive USs despite variations in the strength of US inputs, thereby permitting the rapid updating of fear associations when these sensory features change.SIGNIFICANCE STATEMENT The capacity to learn about sources of danger in the environment is essential for survival. This learning depends on complex microcircuitries of inhibitory interneurons in the basolateral amygdala. Here, we show that parvalbumin-positive GABAergic interneurons in the rat basolateral amygdala are important for fear learning during moments of danger, but not for extinction learning during moments of safety, and that the activity of these neurons is modulated by expectation of danger. This may enable fear-learning circuits to retain selectivity for specific aversive events across variations in expectation, permitting the rapid updating of learning when aversive events change.

Apical intercalated cell cluster: A distinct sensory regulator in the amygdala.

GABAergic neurons regulate different aspects of information processing in the amygdala. Among these are clusters of intercalated cells (ITCs), which have been implicated in fear-related behaviors. Although a few of the ITC clusters have been studied, the functional role of apical ITCs (apITCs) is unknown. Here, we combine monosynaptic rabies tracing with optogenetics and demonstrate that apITCs receive synaptic input from medial geniculate nucleus (MGm), posterior intralaminar nucleus (PIN), and medial dorsal nucleus of the thalamus and from a diverse range of cortical areas including temporal association, entorhinal, insular, piriform, and somatosensory cortex. Upon fear learning, PIN/MGm inputs are strengthened, indicative of their involvement in fear behaviors. 3-D reconstruction of apITCs reveals local arborization and innervation of the dorsal striatum and lateral amygdala. We further show that apITCs provide sensory feedforward inhibition to LA principal cells, a putative mechanism for controlling plasticity during fear learning.

Importance of the Habenula for Avoidance Learning Including Contextual Cues in the Human Brain: A Preliminary fMRI Study.

Human habenula studies are gradually advancing, primarily through the use of functional magnetic resonance imaging (fMRI) analysis of passive (Pavlovian) conditioning tasks as well as probabilistic reinforcement learning tasks. However, no studies have particularly targeted aversive prediction errors, despite the essential importance for the habenula in the field. Complicated learned strategies including contextual contents are involved in making aversive prediction errors during the learning process. Therefore, we examined habenula activation during a contextual learning task. We performed fMRI on a group of 19 healthy controls. We assessed the manually traced habenula during negative outcomes during the contextual learning task. The Beck Depression Inventory-Second Edition (BDI-II), the State-Trait-Anxiety Inventory (STAI), and the Temperament and Character Inventory (TCI) were also administered. The left and right habenula were activated during aversive outcomes and the activation was associated with aversive prediction errors. There was also a positive correlation between TCI reward dependence scores and habenula activation. Furthermore, dynamic causal modeling (DCM) analyses demonstrated the left and right habenula to the left and right hippocampus connections during the presentation of contextual stimuli. These findings serve to highlight the neural mechanisms that may be relevant to understanding the broader relationship between the habenula and learning processes.

Beyond arousal: Prediction error related to aversive events promotes episodic memory formation.

Stimuli encoded shortly before an aversive event are typically well remembered. Traditionally, this emotional memory enhancement has been attributed to beneficial effects of physiological arousal on memory formation. Here, we proposed an additional mechanism and tested whether memory formation is driven by the unpredictable nature of aversive events (i.e., aversive prediction errors). In a combined Pavlovian fear conditioning and incidental memory paradigm, participants saw initially neutral pictures from 2 distinct categories, 1 of which was associated with a risk to receive an electric shock. During encoding, we measured both physiological arousal and explicit prediction errors to explain memory differences in a surprise recognition test that followed approximately 24 hr later. In a first experiment, we show that physiological arousal, expressed as outcome-related skin conductance responses, was associated with improved recognition memory, corroborating arousal-based models. Critically, unsigned binary prediction errors derived from explicit shock expectancy ratings in each trial were also linked to enhanced recognition and model fits showed that the impact of prediction errors on memory was dissociable from the impact of arousal. In a second experiment, we replicated and extended the findings of the first experiment by demonstrating that the memory-promoting effect of prediction errors remained even after controlling for arousal. The present data point to prediction error-related learning as a cognitive mechanism that contributes to the emotional enhancement of memory, above and beyond the well-established effects of arousal in emotional memory formation. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Recent advances in understanding the role of phasic dopamine activity.

The latest animal neurophysiology has revealed that the dopamine reward prediction error signal drives neuronal learning in addition to behavioral learning and reflects subjective reward representations beyond explicit contingency. The signal complies with formal economic concepts and functions in real-world consumer choice and social interaction. An early response component is influenced by physical impact, reward environment, and novelty but does not fully code prediction error. Some dopamine neurons are activated by aversive stimuli, which may reflect physical stimulus impact or true aversiveness, but they do not seem to code general negative value or aversive prediction error. The reward prediction error signal is complemented by distinct, heterogeneous, smaller and slower changes reflecting sensory and motor contributors to behavioral activation, such as substantial movement (as opposed to precise motor control), reward expectation, spatial choice, vigor, and motivation. The different dopamine signals seem to defy a simple unifying concept and should be distinguished to better understand phasic dopamine functions.

Distinct Processing of Aversive Experience in Amygdala Subregions

BackgroundThe amygdala is an anatomically complex medial temporal brain structure whose subregions are considered to serve distinct functions. However, their precise role in mediating human aversive experience remains ill understood. MethodsWe used functional magnetic resonance imaging in 39 healthy volunteers with varying levels of trait anxiety to assess distinct contributions of the basolateral amygdala (BLA) and centromedial amygdala to anticipation and experience of aversive events. Additionally, we examined the relationship between any identified functional subspecialization and measures of subjective reported aversion and trait anxiety. ResultsOur results show that the centromedial amygdala is responsive to aversive outcomes but insensitive to predictive aversive cues. In contrast, the BLA encodes an aversive prediction error that quantifies whether cues and outcomes are worse than expected. A neural representation within the BLA for distinct threat levels was mirrored in self-reported subjective anxiety across individuals. Furthermore, high trait-anxious individuals were characterized by indiscriminately heightened BLA activity in response to aversive cues, regardless of actual threat level. ConclusionsOur results demonstrate that amygdala subregions are distinctly engaged in processing of aversive experience, with elevated and undifferentiated BLA responses to threat emerging as a potential neurobiological mediator of vulnerability to anxiety disorders.

Effects of Positive and Negative Expectations on Human Pain Perception Engage Separate But Interrelated and Dependently Regulated Cerebral Mechanisms.

Expectations substantially influence pain perception, but the relationship between positive and negative expectations remains unclear. Recent evidence indicates that the integration between pain-related expectations and prediction errors is crucial for pain perception, which suggests that aversive prediction error-associated regions, such as the anterior insular cortex (aIC) and rostral anterior cingulate cortex (rACC), may play a pivotal role in expectation-induced pain modulation and help to delineate the relationship between positive and negative expectations. In a stimulus expectancy paradigm combining fMRI in healthy volunteers of both sexes, we found that, although positive and negative expectations respectively engaged the right aIC and right rACC to modulate pain, their associated activations and pain rating changes were significantly correlated. When positive and negative expectations modulated pain, the right aIC and rACC exhibited opposite coupling with periaqueductal gray (PAG) and the mismatch between actual and expected pain respectively modulated their coupling with PAG and thalamus across individuals. Participants' certainty about expectations predicted the extent of pain modulation, with positive expectations involving connectivity between aIC and hippocampus, a region regulating anxiety, and negative expectations engaging connectivity between rACC and lateral orbitofrontal cortex, a region reflecting outcome value and certainty. Interestingly, the strength of these certainty-related connectivities was also significantly associated between positive and negative expectations. These findings suggest that aversive prediction-error-related regions interact with pain-processing circuits to underlie stimulus expectancy effects on pain, with positive and negative expectations engaging dissociable but interrelated neural responses that are dependently regulated by individual certainty about expectations.SIGNIFICANCE STATEMENT Positive and negative expectations substantially influence pain perception, but their relationship remains unclear. Using fMRI in a stimulus expectancy paradigm, we found that, although positive and negative expectations engaged separate brain regions encoding the mismatch between actual and expected pain and involved opposite functional connectivities with the descending pain modulatory system, they produced significantly correlated pain rating changes and brain activation. Moreover, participants' certainty about expectations predicted the magnitude of both types of pain modulation, with the underlying functional connectivities significantly correlated between positive and negative expectations. These findings advance current understanding about cognitive modulation of pain, suggesting that both types of pain modulation engage different aversive prediction error signals but are dependently regulated by individual certainty about expectations.

Polymorphisms that affect GABA neurotransmission predict processing of aversive prediction errors in humans

Learning is one of our most adaptive abilities, allowing us to adjust our expectations about future events. Aberrant learning processes may underlie disorders such as anxiety, motivating the search for the neural mechanisms that underpin learning. Animal studies have shown that the neurotransmitter GABA is required for the computation of prediction errors, the mismatches between anticipated and experienced outcomes, which drive new learning. Given that evidence from human studies is lacking, we sought to determine whether these findings extend to humans. Here, in two samples of Caucasian individuals, we investigated whether genetically determined individual differences in GABA neurotransmission predict the P3 event-related potential, an EEG component known to reflect prediction error processing. Consistent with the results of animal studies, we show that a weighted genetic risk score computed from the number of GABRB2 rs1816072 A alleles (associated with increased expression of the GABAA receptor β2 subunit gene) and the number of ErbB4 rs7598440 T alleles (associated with increased GABA concentration) predicts optimal prediction error processing during aversive classical conditioning with both visual (Experiment 1, N = 90; p = .010) and auditory (Experiment 2; N = 92; p = .031) unconditioned stimuli. Our finding that optimal processing of aversive prediction errors is reduced in individuals genetically predisposed towards decreased GABA neurotransmission suggests a potential mechanism linking GABA and anxiety. Specifically, reduced GABA signalling via GABAA receptors could result in aberrant learning from aversive experiences and vulnerability to anxiety disorders.

Roles of dopamine neurons in mediating the prediction error in aversive learning in insects

In associative learning in mammals, it is widely accepted that the discrepancy, or error, between actual and predicted reward determines whether learning occurs. The prediction error theory has been proposed to account for the finding of a blocking phenomenon, in which pairing of a stimulus X with an unconditioned stimulus (US) could block subsequent association of a second stimulus Y to the US when the two stimuli were paired in compound with the same US. Evidence for this theory, however, has been imperfect since blocking can also be accounted for by competitive theories. We recently reported blocking in classical conditioning of an odor with water reward in crickets. We also reported an “auto-blocking” phenomenon in appetitive learning, which supported the prediction error theory and rejected alternative theories. The presence of auto-blocking also suggested that octopamine neurons mediate reward prediction error signals. Here we show that blocking and auto-blocking occur in aversive learning to associate an odor with salt water (US) in crickets, and our results suggest that dopamine neurons mediate aversive prediction error signals. We conclude that the prediction error theory is applicable to both appetitive learning and aversive learning in insects.

Temporal dissociation of salience and prediction error responses to appetitive and aversive taste.

The feedback-related negativity (FRN), a frontocentral ERP occurring 200-350 ms after emotionally valued outcomes, has been posited as the neural correlate of reward prediction error, a key component of associative learning. Recent evidence challenged this interpretation and has led to the suggestion that this ERP expresses salience instead. Here, we distinguish between utility prediction error and salience by delivering or withholding hedonistically matched appetitive and aversive tastes, and measure ERPs to cues signaling each taste. We observed a typical FRN (computed as the loss-minus-gain difference wave) to appetitive taste, but a reverse FRN to aversive taste. When tested axiomatically, frontocentral ERPs showed a salience response across tastes, with a particularly early response to outcome delivery, supporting recent propositions of a fast, unsigned, and unspecific response to salient stimuli. ERPs also expressed aversive prediction error peaking at 285 ms, which conformed to the logic of an axiomatic model of prediction error. With stimuli that most resemble those used in animal models, we did not detect any frontocentral ERP signal for utility prediction error, in contrast with dominant views of the functional role of the FRN ERP. We link the animal and human literature and present a challenge for current perspectives on associative learning research using ERPs.

Cognitive emotion regulation modulates the balance of competing influences on ventral striatal aversive prediction error signals

Cognitive emotion regulation (CER) is a critical human ability to face aversive emotional stimuli in a flexible way, via recruitment of specific prefrontal brain circuits. Animal research reveals a central role of ventral striatum in emotional behavior, for both aversive conditioning, with striatum signaling aversive prediction errors (aPE), and for integrating competing influences of distinct striatal inputs from regions such as the prefrontal cortex (PFC), amygdala, hippocampus and ventral tegmental area (VTA). Translating these ventral striatal findings from animal research to human CER, we hypothesized that successful CER would affect the balance of competing influences of striatal afferents on striatal aPE signals, in a way favoring PFC as opposed to ‘subcortical’ (i.e., non-isocortical) striatal inputs. Using aversive Pavlovian conditioning with and without CER during fMRI, we found that during CER, superior regulators indeed reduced the modulatory impact of ‘subcortical’ striatal afferents (hippocampus, amygdala and VTA) on ventral striatal aPE signals, while keeping the PFC impact intact. In contrast, inferior regulators showed an opposite pattern. Our results demonstrate that ventral striatal aPE signals and associated competing modulatory inputs are critical mechanisms underlying successful cognitive regulation of aversive emotions in humans.

Early adversity disrupts the adult use of aversive prediction errors to reduce fear in uncertainty

Early life adversity increases anxiety in adult rodents and primates, and increases the risk for developing post-traumatic disorder (PTSD) in humans. We hypothesized that early adversity impairs the use of learning signals -negative, aversive prediction errors–to reduce fear in uncertainty. To test this hypothesis, we gave adolescent rats a battery of adverse experiences then assessed adult performance in probabilistic Pavlovian fear conditioning and fear extinction. Rats were confronted with three cues associated with different probabilities of foot shock: one cue never predicted shock, another cue predicted shock with uncertainty, and a final cue always predicted shock. Control rats initially acquired fear to all cues, but rapidly reduced fear to the non-predictive and uncertain cues. Early adversity rats were slower to reduce fear to the non-predictive cue and never fully reduced fear to the uncertain cue. In extinction, all cues were presented in the absence of shock. Fear to the uncertain cue in discrimination, but not early adversity itself, predicted the reduction of fear in extinction. These results demonstrate early adversity impairs the use of negative aversive prediction errors to reduce fear, especially in situations of uncertainty.

Cognitive emotion regulation enhances aversive prediction error activity while reducing emotional responses

Cognitive emotion regulation is a powerful way of modulating emotional responses. However, despite the vital role of emotions in learning, it is unknown whether the effect of cognitive emotion regulation also extends to the modulation of learning. Computational models indicate prediction error activity, typically observed in the striatum and ventral tegmental area, as a critical neural mechanism involved in associative learning. We used model-based fMRI during aversive conditioning with and without cognitive emotion regulation to test the hypothesis that emotion regulation would affect prediction error-related neural activity in the striatum and ventral tegmental area, reflecting an emotion regulation-related modulation of learning. Our results show that cognitive emotion regulation reduced emotion-related brain activity, but increased prediction error-related activity in a network involving ventral tegmental area, hippocampus, insula and ventral striatum. While the reduction of response activity was related to behavioral measures of emotion regulation success, the enhancement of prediction error-related neural activity was related to learning performance. Furthermore, functional connectivity between the ventral tegmental area and ventrolateral prefrontal cortex, an area involved in regulation, was specifically increased during emotion regulation and likewise related to learning performance. Our data, therefore, provide first-time evidence that beyond reducing emotional responses, cognitive emotion regulation affects learning by enhancing prediction error-related activity, potentially via tegmental dopaminergic pathways.

Representation of aversive prediction errors in the human periaqueductal gray.

Pain is a primary driver of learning and motivated action. It is also a target of learning, as nociceptive brain responses are shaped by learning processes. We combined an instrumental pain avoidance task with an axiomatic approach to assessing fMRI signals related to prediction errors (PEs), which drive reinforcement-based learning. We found that pain PEs were encoded in the periaqueductal gray (PAG), an important structure for pain control and learning in animal models. Axiomatic tests combined with dynamic causal modeling suggested that ventromedial prefrontal cortex, supported by putamen, provides an expected value-related input to the PAG, which then conveys PE signals to prefrontal regions important for behavioral regulation, including orbitofrontal, anterior mid-cingulate, and dorsomedial prefrontal cortices. Thus, pain-related learning involves distinct neural circuitry, with implications for behavior and pain dynamics.

Differential modulation of reinforcement learning by D2 dopamine and NMDA glutamate receptor antagonism.

The firing pattern of midbrain dopamine (DA) neurons is well known to reflect reward prediction errors (PEs), the difference between obtained and expected rewards. The PE is thought to be a crucial signal for instrumental learning, and interference with DA transmission impairs learning. Phasic increases of DA neuron firing during positive PEs are driven by activation of NMDA receptors, whereas phasic suppression of firing during negative PEs is likely mediated by inputs from the lateral habenula. We aimed to determine the contribution of DA D2-class and NMDA receptors to appetitively and aversively motivated reinforcement learning. Healthy human volunteers were scanned with functional magnetic resonance imaging while they performed an instrumental learning task under the influence of either the DA D2 receptor antagonist amisulpride (400 mg), the NMDA receptor antagonist memantine (20 mg), or placebo. Participants quickly learned to select ("approach") rewarding and to reject ("avoid") punishing options. Amisulpride impaired both approach and avoidance learning, while memantine mildly attenuated approach learning but had no effect on avoidance learning. These behavioral effects of the antagonists were paralleled by their modulation of striatal PEs. Amisulpride reduced both appetitive and aversive PEs, while memantine diminished appetitive, but not aversive PEs. These data suggest that striatal D2-class receptors contribute to both approach and avoidance learning by detecting both the phasic DA increases and decreases during appetitive and aversive PEs. NMDA receptors on the contrary appear to be required only for approach learning because phasic DA increases during positive PEs are NMDA dependent, whereas phasic decreases during negative PEs are not.

The dorsal raphe nucleus is integral to negative prediction errors in Pavlovian fear

Prediction errors are central to modern learning theories. While brain regions contributing to reward prediction errors have been uncovered, the sources of aversive prediction errors remain largely unknown. Here we used probabilistic and deterministic reinforcement procedures, followed by extinction, to examine the contribution of the dorsal raphe nucleus to negative, aversive prediction errors in Pavlovian fear. Rats with dorsal raphe lesions were able to acquire fear and reduce fear to a non-reinforced deterministic cue. However, dorsal raphe lesions impaired the reduction of fear to a probabilistic cue and fear extinction to a deterministic cue, both of which involve the use of negative prediction errors. The results point to an integral role for the dorsal raphe nucleus in negative prediction error signaling in Pavlovian fear.