Positive Prediction Errors Research Articles

Conservative Sparse Neural Network Embedded Frequency-Constrained Unit Commitment With Distributed Energy Resources

The increasing penetration of distributed energy resources (DERs) will decrease the rotational inertia of the power system and further degrade the system frequency stability. To address the above issues, this paper leverages the advanced neural network (NN) to learn the frequency dynamics and incorporates NN to facilitate system reliable operation. This paper proposes the conservative sparse neural network (CSNN) embedded frequency-constrained unit commitment (FCUC) with converter-based DERs, including the learning and optimization stages. In the learning stage, it samples the inertia parameters, calculates the corresponding frequency, and characterizes the stability region of the sampled parameters using the convex hulls to ensure stability and avoid extrapolation. For conservativeness, the positive prediction error penalty is added to the loss function to prevent possible frequency requirement violation. For the sparsity, the NN topology pruning is employed to eliminate unnecessary connections for solving acceleration. In the optimization stage, the trained CSNN is transformed into mixed-integer linear constraints using the big-M method and then incorporated to establish the data-enhanced model. The case study verifies 1) the effectiveness of the proposed model in terms of high accuracy, fewer parameters, and significant solving acceleration; 2) the stable system operation against frequency violation under contingency.

Molecularly targetable cell types in mouse visual cortex have distinguishable prediction error responses

Predictive processing postulates the existence of prediction error neurons in cortex. Neurons with both negative and positive prediction error response properties have been identified in layer 2/3 of visual cortex, but whether they correspond to transcriptionally defined subpopulations is unclear. Here we used the activity-dependent, photoconvertible marker CaMPARI2 to tag neurons in layer 2/3 of mouse visual cortex during stimuli and behaviors designed to evoke prediction errors. We performed single-cell RNA-sequencing on these populations and found that previously annotated Adamts2 and Rrad layer 2/3 transcriptional cell types were enriched when photolabeling during stimuli that drive negative or positive prediction error responses, respectively. Finally, we validated these results functionally by designing artificial promoters for use in AAV vectors to express genetically encoded calcium indicators. Thus, transcriptionally distinct cell types in layer 2/3 that can be targeted using AAV vectors exhibit distinguishable negative and positive prediction error responses.

The shadowing effect of initial expectation on learning asymmetry.

Evidence for positivity and optimism bias abounds in high-level belief updates. However, no consensus has been reached regarding whether learning asymmetries exist in more elementary forms of updates such as reinforcement learning (RL). In RL, the learning asymmetry concerns the sensitivity difference in incorporating positive and negative prediction errors (PE) into value estimation, namely the asymmetry of learning rates associated with positive and negative PEs. Although RL has been established as a canonical framework in characterizing interactions between agent and environment, the direction of learning asymmetry remains controversial. Here, we propose that part of the controversy stems from the fact that people may have different value expectations before entering the learning environment. Such a default value expectation influences how PEs are calculated and consequently biases subjects' choices. We test this hypothesis in two learning experiments with stable or varying reinforcement probabilities, across monetary gains, losses, and gain-loss mixed environments. Our results consistently support the model incorporating both asymmetric learning rates and the initial value expectation, highlighting the role of initial expectation in value updating and choice preference. Further simulation and model parameter recovery analyses confirm the unique contribution of initial value expectation in accessing learning rate asymmetry.

Opponent learning with different representations in the cortico-basal ganglia pathways can develop obsession-compulsion cycle.

Obsessive-compulsive disorder (OCD) has been suggested to be associated with impairment of model-based behavioral control. Meanwhile, recent work suggested shorter memory trace for negative than positive prediction errors (PEs) in OCD. We explored relations between these two suggestions through computational modeling. Based on the properties of cortico-basal ganglia pathways, we modeled human as an agent having a combination of successor representation (SR)-based system that enables model-based-like control and individual representation (IR)-based system that only hosts model-free control, with the two systems potentially learning from positive and negative PEs in different rates. We simulated the agent's behavior in the environmental model used in the recent work that describes potential development of obsession-compulsion cycle. We found that the dual-system agent could develop enhanced obsession-compulsion cycle, similarly to the agent having memory trace imbalance in the recent work, if the SR- and IR-based systems learned mainly from positive and negative PEs, respectively. We then simulated the behavior of such an opponent SR+IR agent in the two-stage decision task, in comparison with the agent having only SR-based control. Fitting of the agents' behavior by the model weighing model-based and model-free control developed in the original two-stage task study resulted in smaller weights of model-based control for the opponent SR+IR agent than for the SR-only agent. These results reconcile the previous suggestions about OCD, i.e., impaired model-based control and memory trace imbalance, raising a novel possibility that opponent learning in model(SR)-based and model-free controllers underlies obsession-compulsion. Our model cannot explain the behavior of OCD patients in punishment, rather than reward, contexts, but it could be resolved if opponent SR+IR learning operates also in the recently revealed non-canonical cortico-basal ganglia-dopamine circuit for threat/aversiveness, rather than reward, reinforcement learning, and the aversive SR + appetitive IR agent could actually develop obsession-compulsion if the environment is modeled differently.

Threat learning in space: how stimulus-outcome spatial compatibility modulates conditioned skin conductance response

A central question in Pavlovian conditioning concerns the critical conditions that drive the acquisition and maintenance of the stimulus-outcome association. The spatial relationship between the conditioned (CS) and unconditioned (US) stimuli is considered to exert strong effects on learning. However, how spatial information modulates Pavlovian learning remains mostly unexplored in humans. Here, we test how the compatibility between the CS and the US location influences the acquisition, extinction, and recovery (following reinstatement) of Pavlovian conditioned threat. Participants (N = 20) completed a differential threat conditioning task in which visual CSs appeared on the same (compatible) or opposite (incompatible) hemispace as the US delivery (aversive shock to one hand), while their skin conductance response served as an index of learning. Results show that initial threat expectations were biased in favor of compatible CSs before conditioning. Nevertheless, this bias was revised during acquisition to reflect current stimulus-outcome contingencies. Computational modeling suggested that this effect occurred through a higher reliance on positive aversive prediction errors for incompatible CSs, thereby facilitating learning of their association with the US. Additionally, the conditioned response to incompatible CSs was associated with initially slower extinction and a greater recovery after threat reinstatement. These findings suggest that spatial information conveyed by stimuli and outcomes can be flexibly used to enact defensive responses to the current source of danger, highlighting the adaptive nature of Pavlovian learning.

Dopamine in the dorsal bed nucleus of stria terminalis signals Pavlovian sign-tracking and reward violations.

Midbrain and striatal dopamine signals have been extremely well characterized over the past several decades, yet novel dopamine signals and functions in reward learning and motivation continue to emerge. A similar characterization of real-time sub-second dopamine signals in areas outside of the striatum has been limited. Recent advances in fluorescent sensor technology and fiber photometry permit the measurement of dopamine binding correlates, which can divulge basic functions of dopamine signaling in non-striatal dopamine terminal regions, like the dorsal bed nucleus of the stria terminalis (dBNST). Here, we record GRABDA signals in the dBNST during a Pavlovian lever autoshaping task. We observe greater Pavlovian cue-evoked dBNST GRABDA signals in sign-tracking (ST) compared to goal-tracking/intermediate (GT/INT) rats and the magnitude of cue-evoked dBNST GRABDA signals decreases immediately following reinforcer-specific satiety. When we deliver unexpected rewards or omit expected rewards, we find that dBNST dopamine signals encode bidirectional reward prediction errors in GT/INT rats, but only positive prediction errors in ST rats. Since sign- and goal-tracking approach strategies are associated with distinct drug relapse vulnerabilities, we examined the effects of experimenter-administered fentanyl on dBNST dopamine associative encoding. Systemic fentanyl injections do not disrupt cue discrimination but generally potentiate dBNST dopamine signals. These results reveal multiple dBNST dopamine correlates of learning and motivation that depend on the Pavlovian approach strategy employed.

Prediction errors arising from switches between major and minor modes in music: An fMRI study

The major and minor modes in Western music have positive and negative connotations, respectively. The present fMRI study examined listeners’ neural responses to switches between major and minor modes. We manipulated the final chords of J. S. Bach’s keyboard pieces so that each major-mode passage ended with either the major (Major-Major) or minor (Major-Minor) tonic chord, and each minor-mode passage ended with either the minor (Minor-Minor) or major (Minor-Major) tonic chord. If the final major and minor chords have positive and negative reward values respectively, the Major-Minor and Minor-Major stimuli would cause negative and positive reward prediction errors (RPEs) respectively in a listener’s brain. We found that activity in a frontoparietal network was significantly higher for Major-Minor than for Major-Major. Based on previous research, these results support the idea that a major-to-minor switch causes negative RPE. The contrast of Minor-Major minus Minor-Minor yielded activation in the ventral insula and visual cortex, speaking against the idea that a minor-to-major switch causes positive RPE. We discuss our results in relation to executive functions and the emotional connotations of major versus minor modes.

The Human Centromedial Amygdala Contributes to Negative Prediction Error Signaling during Appetitive and Aversive Pavlovian Gustatory Learning.

Prediction error (PE) is the mismatch between a prior expectation and reality, and it lies at the core of associative learning about aversive and appetitive stimuli. Human studies on fear learning have linked the amygdala to aversive PEs. In contrast, the relationship between the amygdala and PE in appetitive settings and stimuli, unlike those that induce fear, has received less research attention. Animal studies show that the amygdala is a functionally heterogeneous structure. Nevertheless, the role of the amygdala nuclei in PE signaling remains unknown in humans. To clarify the role of two subdivisions of the human amygdala, the centromedial amygdala (CMA) and basolateral amygdala (BLA), in appetitive and aversive PE signaling, we used gustatory pavlovian learning involving eating-related naturalistic outcomes. Thirty-eight right-handed individuals (19 females) participated in the study. We found that surprise with neutral feedback when a reward is expected triggers activity within the left and right CMA. When an aversive outcome is expected, surprise with neutral feedback triggers activity only within the left CMA. Notably, the BLA was not activated by those conditions. Thus, the CMA engages in negative PE signaling during appetitive and aversive gustatory pavlovian learning, whereas the BLA is not critical for this process. In addition, PE-related activity within the left CMA during aversive learning is negatively correlated with neuroticism and positively correlated with extraversion. The findings indicate the importance of the CMA in gustatory learning when the value of outcomes changes and have implications for understanding psychological conditions that manifest perturbed processing of negative PEs.SIGNIFICANCE STATEMENT A discrepancy between a prediction and an actual outcome (PE) plays a crucial role in learning. Learning improves when an outcome is more significant than expected (positive PE) and worsens when it is smaller than expected (negative PE). We found that the negative PE during appetitive and aversive taste learning is associated with increased activity of the CMA, which suggests that the CMA controls taste learning. Our findings may have implications for understanding psychological states associated with deficient learning from negative PEs, such as obesity and addictive behaviors.

Individuals with problem gambling and obsessive-compulsive disorder learn through distinct reinforcement mechanisms.

Obsessive-compulsive disorder (OCD) and pathological gambling (PG) are accompanied by deficits in behavioural flexibility. In reinforcement learning, this inflexibility can reflect asymmetric learning from outcomes above and below expectations. In alternative frameworks, it reflects perseveration independent of learning. Here, we examine evidence for asymmetric reward-learning in OCD and PG by leveraging model-based functional magnetic resonance imaging (fMRI). Compared with healthy controls (HC), OCD patients exhibited a lower learning rate for worse-than-expected outcomes, which was associated with the attenuated encoding of negative reward prediction errors in the dorsomedial prefrontal cortex and the dorsal striatum. PG patients showed higher and lower learning rates for better- and worse-than-expected outcomes, respectively, accompanied by higher encoding of positive reward prediction errors in the anterior insula than HC. Perseveration did not differ considerably between the patient groups and HC. These findings elucidate the neural computations of reward-learning that are altered in OCD and PG, providing a potential account of behavioural inflexibility in those mental disorders.

The power of the unexpected: Prediction errors enhance stereotype-based learning

Stereotyping is a ubiquitous feature of social cognition, yet surprisingly little is known about how group-related beliefs influence the acquisition of person knowledge. Accordingly, in combination with computational modeling (i.e., Reinforcement Learning Drift Diffusion Model analysis), here we used a probabilistic selection task to explore the extent to which gender stereotypes impact instrumental learning. Several theoretically interesting effects were observed. First, reflecting the impact of cultural socialization on person construal, an expectancy-based preference for stereotype-consistent (vs. stereotype-inconsistent) responses was observed. Second, underscoring the potency of unexpected information, learning rates were faster for counter-stereotypic compared to stereotypic individuals, both for negative and positive prediction errors. Collectively, these findings are consistent with predictive accounts of social perception and have implications for the conditions under which stereotyping can potentially be reduced.

A computational neuroimaging study of reinforcement learning and goal-directed exploration in schizophrenia spectrum disorders.

Prior evidence indicates that negative symptom severity and cognitive deficits, in people with schizophrenia (PSZ), relate to measures of reward-seeking and loss-avoidance behavior (implicating the ventral striatum/VS), as well as uncertainty-driven exploration (reliant on rostrolateral prefrontal cortex/rlPFC). While neural correlates of reward-seeking and loss-avoidance have been examined in PSZ, neural correlates of uncertainty-driven exploration have not. Understanding neural correlates of uncertainty-driven exploration is an important next step that could reveal insights to how this mechanism of cognitive and negative symptoms manifest at a neural level. We acquired fMRI data from 29 PSZ and 36 controls performing the Temporal Utility Integration decision-making task. Computational analyses estimated parameters corresponding to learning rates for both positive and negative reward prediction errors (RPEs) and the degree to which participates relied on representations of relative uncertainty. Trial-wise estimates of expected value, certainty, and RPEs were generated to model fMRI data. Behaviorally, PSZ demonstrated reduced reward-seeking behavior compared to controls, and negative symptoms were positively correlated with loss-avoidance behavior. This finding of a bias toward loss avoidance learning in PSZ is consistent with previous work. Surprisingly, neither behavioral measures of exploration nor neural correlates of uncertainty in the rlPFC differed significantly between groups. However, we showed that trial-wise estimates of relative uncertainty in the rlPFC distinguished participants who engaged in exploratory behavior from those who did not. rlPFC activation was positively associated with intellectual function. These results further elucidate the nature of reinforcement learning and decision-making in PSZ and healthy volunteers.

Intracranial subsecond dopamine measurements during a "sure bet or gamble" decision-making task in patients with alcohol use disorder suggest diminished dopaminergic signals about relief.

To the authors' knowledge, no data have been reported on dopamine fluctuations on subsecond timescales in humans with alcohol use disorder (AUD). In this study, dopamine release was monitored in 2 patients with and 2 without a history of AUD during a "sure bet or gamble" (SBORG) decision-making task to begin to characterize how subsecond dopamine responses to counterfactual information, related to psychological notions of regret and relief, in AUD may be altered. Measurements of extracellular dopamine levels were made once every 100 msec using human voltammetric methods. Measurements were made in the caudate during deep brain stimulation electrode implantation surgeries (for treatment of movement disorders) in patients who did (AUD, n = 2) or did not (non-AUD, n = 2) have a history of AUD. Participants performed an SBORG decision-making task in which they made choices between sure bets and 50%-chance monetary gamble outcomes. Fast changes were found in dopamine levels that appear to be modulated by "what could have been" and by patients' AUD status. Positive counterfactual prediction errors (related to relief) differentiated patients with versus without a history of AUD. Dopaminergic encoding of counterfactual information appears to differ between patients with and without AUD. The current study has a major limitation of a limited sample size, but these data provide a rare insight into dopaminergic physiology during real-time decision-making in humans with an addiction disorder. The authors hope future work will expand the sample size and determine the generalizability of the current results.

A GAN–SVR Prediction Method of the Metal Tube‐Bending Rebound with Small Samples

This paper investigates a predictive algorithm for the angle of the metal tube‐bending rebound with small samples. First, the generative adversarial network (GAN) approach is introduced to address the issues of insufficient sample data. The proposed method can realize data augmentation through a generator, enhancing training effectiveness compared to conventional model‐based and experimental prediction methods. To further reduce the problems caused by the small samples, the Wasserstein distance is utilized as the optimization objective for the GAN approach. Second, after obtaining the augmented dataset, Support Vector Regression (SVR) is employed to predict the rebound model of the metal tube‐bending. A novel predictive algorithm for the angle of the metal tube‐bending rebound based on GAN–SVR is proposed. It exhibits that the GAN–SVR owns more positive prediction ability and error when dealing with small samples than conventional GAN‐radial basis function methd (GAN–BP) and GAN–convolutional neural networks. Finally, the effectiveness of the proposed method is validated through experimental results.

Opponent Learning with Different Representations in the Cortico-Basal Ganglia Circuits.

The direct and indirect pathways of the basal ganglia (BG) have been suggested to learn mainly from positive and negative feedbacks, respectively. Since these pathways unevenly receive inputs from different cortical neuron types and/or regions, they may preferentially use different state/action representations. We explored whether such a combined use of different representations, coupled with different learning rates from positive and negative reward prediction errors (RPEs), has computational benefits. We modeled animal as an agent equipped with two learning systems, each of which adopted individual representation (IR) or successor representation (SR) of states. With varying the combination of IR or SR and also the learning rates from positive and negative RPEs in each system, we examined how the agent performed in a dynamic reward navigation task. We found that combination of SR-based system learning mainly from positive RPEs and IR-based system learning mainly from negative RPEs could achieve a good performance in the task, as compared with other combinations. In such a combination of appetitive SR-based and aversive IR-based systems, both systems show activities of comparable magnitudes with opposite signs, consistent with the suggested profiles of the two BG pathways. Moreover, the architecture of such a combination provides a novel coherent explanation for the functional significance and underlying mechanism of diverse findings about the cortico-BG circuits. These results suggest that particularly combining different representations with appetitive and aversive learning could be an effective learning strategy in certain dynamic environments, and it might actually be implemented in the cortico-BG circuits.

Positioning the brainstem within the neural network of threat prediction

In a recent study, Strickland and McDannald dissected the role of brainstem networks in threat prediction. Using probabilistic threat discrimination in rats, the authors demonstrated that brainstem neurons estimate threat probability and generate positive aversive prediction errors after unexpected outcomes. Their findings suggest that, beyond organizing defensive behaviors, brainstem neurons are involved in threat prediction computations.

Memory trace imbalance in reinforcement and punishment systems can reinforce implicit choices leading to obsessive-compulsive behavior.

We may view most of our daily activities as rational action selections; however, we sometimes reinforce maladaptive behaviors despite having explicit environmental knowledge. In this study, we model obsessive-compulsive disorder (OCD) symptoms as implicitly learned maladaptive behaviors. Simulations in the reinforcement learning framework show that agents implicitly learn to respond to intrusive thoughts when the memory trace signal for past actions decays differently for positive and negative prediction errors. Moreover, this model extends our understanding of therapeutic effects of behavioral therapy in OCD. Using empirical data, we confirm that patients with OCD show extremely imbalanced traces, which are normalized by serotonin enhancers. We find that healthy participants also vary in their obsessive-compulsive tendencies, consistent with the degree of imbalanced traces. These behavioral characteristics can be generalized to variations in the healthy population beyond the spectrum of clinical phenotypes.

The computational roots of positivity and confirmation biases in reinforcement learning.

Humans do not integrate new information objectively: outcomes carrying a positive affective value and evidence confirming one's own prior belief are overweighed. Until recently, theoretical and empirical accounts of the positivity and confirmation biases assumed them to be specific to 'high-level' belief updates. We present evidence against this account. Learning rates in reinforcement learning (RL) tasks, estimated across different contexts and species, generally present the same characteristic asymmetry, suggesting that belief and value updating processes share key computational principles and distortions. This bias generates over-optimistic expectations about the probability of making the right choices and, consequently, generates over-optimistic reward expectations. We discuss the normative and neurobiological roots of these RL biases and their position within the greater picture of behavioral decision-making theories.

Sticky me: Self-relevance slows reinforcement learning

A prominent facet of social-cognitive functioning is that self-relevant information is prioritized in perception, attention, and memory. What is not yet understood, however, is whether similar effects arise during learning. In particular, compared to other people (e.g., best friend) is information about the self acquired more rapidly? To explore this matter, here we used a probabilistic selection task in combination with computational modeling (i.e., Reinforcement Learning Drift Diffusion Model analysis) to establish how self-relevance influences learning under conditions of uncertainty (i.e., choices are based on the perceived likelihood of positive and negative outcomes). Across two experiments, a consistent pattern of effects was observed. First, learning rates for both positive and negative prediction errors were slower for self-relevant compared to friend-relevant associations. Second, self-relevant (vs. friend-relevant) learning was characterized by the exploitation (vs. exploration) of choice selections. That is, in a complex (i.e., probabilistic) decision-making environment, previously rewarded self-related outcomes were selected more often than novel — but potentially riskier — alternatives. The implications of these findings for accounts of self-function are considered.

Cardiac Cycle Affects the Asymmetric Value Updating in Instrumental Reward Learning.

This study aimed to investigate whether instrumental reward learning is affected by the cardiac cycle. To this end, we examined the effects of the cardiac cycle (systole or diastole) on the computational processes underlying the participants’ choices in the instrumental learning task. In the instrumental learning task, participants were required to select one of two discriminative stimuli (neutral visual stimuli) and immediately receive reward/punishment feedback depending on the probability assigned to the chosen stimuli. To manipulate the cardiac cycle, the presentation of discriminative stimuli was timed to coincide with either cardiac systole or diastole. We fitted the participants’ choices in the task with reinforcement learning (RL) models and estimated parameters involving instrumental learning (i.e., learning rate and inverse temperature) separately in the systole and diastole trials. Model-based analysis revealed that the learning rate for positive prediction errors was higher than that for negative prediction errors in the systole trials; however, learning rates did not differ between positive and negative prediction errors in the diastole trials. These results demonstrate that the natural fluctuation of cardiac afferent signals can affect asymmetric value updating in instrumental reward learning.

Disentangling performance-monitoring signals encoded in feedback-related EEG dynamics

The feedback-related negativity (FRN) is a well-established electrophysiological correlate of feedback-processing. However, there is still an ongoing debate whether the FRN is driven by negative or positive reward prediction errors (RPE), valence of feedback, or mere surprise. Our study disentangles independent contributions of valence, surprise, and RPE on the feedback-related neuronal signal including the FRN and P3 components using the statistical power of a sample of N = 992 healthy individuals. The participants performed a modified time-estimation task, while EEG from 64 scalp electrodes was recorded. Our results show that valence coding is present during the FRN with larger amplitudes for negative feedback. The FRN is further modulated by surprise in a valence-dependent way being more positive-going for surprising positive outcomes. The P3 was strongly driven by both global and local surprise, with larger amplitudes for unexpected feedback and local deviants. Behavioral adaptations after feedback and FRN just show small associations. Results support the theory of the FRN as a representation of a signed RPE. Additionally, our data indicates that surprising positive feedback enhances the EEG response in the time window of the P3. These results corroborate previous findings linking the P3 to the evaluation of PEs in decision making and learning tasks.