Signed Prediction Errors Research Articles

Timing along the cardiac cycle modulates neural signals of reward-based learning

Natural fluctuations in cardiac activity modulate brain activity associated with sensory stimuli, as well as perceptual decisions about low magnitude, near-threshold stimuli. However, little is known about the relationship between fluctuations in heart activity and other internal representations. Here we investigate whether the cardiac cycle relates to learning-related internal representations – absolute and signed prediction errors. We combined machine learning techniques with electroencephalography with both simple, direct indices of task performance and computational model-derived indices of learning. Our results demonstrate that just as people are more sensitive to low magnitude, near-threshold sensory stimuli in certain cardiac phases, so are they more sensitive to low magnitude absolute prediction errors in the same cycles. However, this occurs even when the low magnitude prediction errors are associated with clearly suprathreshold sensory events. In addition, participants exhibiting stronger differences in their prediction error representations between cardiac cycles exhibited higher learning rates and greater task accuracy.

The Rescorla-Wagner model, prediction error, and fear learning

The Rescorla-Wagner model remains one of the most important and influential theoretical accounts of the conditions under which Pavlovian learning occurs. Moreover, the experimental approaches that inspired the model continue to provide powerful behavioral tools to advance mechanistic understanding of how we and other animals learn to fear and learn to reduce fear. Here we consider key features of the Rescorla-Wagner model as applied to study of fear learning. We review evidence for key insights of the model. First, learning to fear and learning to reduce fear are governed by a common, signed prediction error. Second, this error drives variations in effectiveness of the shock US that are causal to whether and how much fear is learned or lost during a conditioning trial. We also consider behavioral and neural findings inconsistent with the model and which will be essential to understand and advance understanding of fear learning.

Gender differences in refraction prediction error of five formulas for cataract surgery

ObjectivesTo evaluate gender differences in optical biometry measurements and lens power calculations.MethodsEight thousand four hundred thirty-one eyes of five thousand five hundred nineteen patients who underwent cataract surgery at University of Michigan’s Kellogg Eye Center were included in this retrospective study. Data including age, gender, optical biometry, postoperative refraction, implanted intraocular lens (IOL) power, and IOL formula refraction predictions were gathered and/or calculated utilizing the Sight Outcomes Research Collaborative (SOURCE) database and analyzed.ResultsThere was a statistical difference between every optical biometry measure between genders. Despite lens constant optimization, mean signed prediction errors (SPEs) of modern IOL formulas differed significantly between genders, with predictions skewed more hyperopic for males and myopic for females for all 5 of the modern IOL formulas tested. Optimization of lens constants by gender significantly decreased prediction error for 2 of the 5 modern IOL formulas tested.ConclusionsGender was found to be an independent predictor of refraction prediction error for all 5 formulas studied. Optimization of lens constants by gender can decrease refraction prediction error for certain modern IOL formulas.

Comparative study of Ohlson and cash flow discounting models in the prediction of the stock price

Two models derived from the dividend discount model attracted the attention of researchers: the residual income model (RIM) and the Ohlson model. These models are said to be dualistic since they combine both aspects of the economic and accounting vision. We propose, in our study, to test the performance of the dualistic evaluation model and to show the importance of accounting information. To do this, we will calculate the value of a listed company according to the actuarial valuation model, namely: the available cash flow discounting model (DCF) and the Ohlson model as a dualistic model. Then, we will determine, based on the expectation and the variance of the signed prediction error (SPE), the model that comes closest to the market price in the case of a Tunisian listed company. The results found in the Tunisian context show the superiority of the Ohlson model in the prediction of stock market prices. This model underlies the traditional belief that the company value is compounded of two main parts: the net value of the investment made in it (book value) and the present value of the period benefits (earnings) that together bring the “clean surplus” concept of the shareholders’ equity value. Specifically, Ohlson (1995) motivates the adoption of the historical price model in value relevance studies, which expresses value as a function of earnings and book values

Testing the reinforcement learning hypothesis of social conformity.

Our preferences are influenced by the opinions of others. The past human neuroimaging studies on social conformity have identified a network of brain regions related to social conformity that includes the posterior medial frontal cortex (pMFC), anterior insula, and striatum. Since these brain regions are also known to play important roles in reinforcement learning (i.e., processing prediction error), it was previously hypothesized that social conformity and reinforcement learning have a common neural mechanism. However, although this view is currently widely accepted, these two processes have never been directly compared; therefore, the extent to which they shared a common neural mechanism had remained unclear. This study aimed to formally test the hypothesis. The same group of participants (n = 25) performed social conformity and reinforcement learning tasks inside a functional magnetic resonance imaging (fMRI) scanner. Univariate fMRI data analyses revealed activation overlaps in the pMFC and bilateral insula between social conflict and unsigned prediction error and in the striatum between social conflict and signed prediction error. We further conducted multivoxel pattern analysis (MVPA) for more direct evidence of a shared neural mechanism. MVPA did not reveal any evidence to support the hypothesis in any of these regions but found that activation patterns between social conflict and prediction error in these regions were largely distinct. Taken together, the present study provides no clear evidence of a common neural mechanism between social conformity and reinforcement learning.

Precision weighting of cortical unsigned prediction error signals benefits learning, is mediated by dopamine, and is impaired in psychosis

Recent theories of cortical function construe the brain as performing hierarchical Bayesian inference. According to these theories, the precision of prediction errors plays a key role in learning and decision-making, is controlled by dopamine and contributes to the pathogenesis of psychosis. To test these hypotheses, we studied learning with variable outcome-precision in healthy individuals after dopaminergic modulation with a placebo, a dopamine receptor agonist bromocriptine or a dopamine receptor antagonist sulpiride (dopamine study n = 59) and in patients with early psychosis (psychosis study n = 74: 20 participants with first-episode psychosis, 30 healthy controls and 24 participants with at-risk mental state attenuated psychotic symptoms). Behavioural computational modelling indicated that precision weighting of prediction errors benefits learning in health and is impaired in psychosis. FMRI revealed coding of unsigned prediction errors, which signal surprise, relative to their precision in superior frontal cortex (replicated across studies, combined n = 133), which was perturbed by dopaminergic modulation, impaired in psychosis and associated with task performance and schizotypy (schizotypy correlation in 86 healthy volunteers). In contrast to our previous work, we did not observe significant precision-weighting of signed prediction errors, which signal valence, in the midbrain and ventral striatum in the healthy controls (or patients) in the psychosis study. We conclude that healthy people, but not patients with first-episode psychosis, take into account the precision of the environment when updating beliefs. Precision weighting of cortical prediction error signals is a key mechanism through which dopamine modulates inference and contributes to the pathogenesis of psychosis.

Neural Mechanisms of Social and Nonsocial Reward Prediction Errors in Adolescents with Autism Spectrum Disorder

Autism spectrum disorder (ASD) is characterized by impaired predictive abilities; however, the neural mechanisms subsuming reward prediction errors in ASD are poorly understood. In the current study, we investigated neural responses during social and nonsocial reward prediction errors in 22 adolescents with ASD (ages 12-17) and 20 typically developing control adolescents (ages 12-18). Participants performed a reward prediction error task using both social (i.e., faces) and nonsocial (i.e., objects) rewards during a functional magnetic resonance imaging scan. Reward prediction errors were defined in two ways: (a) the signed prediction error, the difference between the experienced and expected reward; and (b) the thresholded unsigned prediction error, the difference between expected and unexpected outcomes regardless of magnitude. During social reward prediction errors, the ASD group demonstrated the following differences relative to the TD group: (a) signed prediction error: decreased activation in the right precentral gyrus and increased activation in the right frontal pole; and (b) thresholded unsigned prediction error: increased activation in the right anterior cingulate gyrus and bilateral precentral gyrus. Groups did not differ in brain activation during nonsocial reward prediction errors. Within the ASD group, exploratory analyses revealed that reaction times and social-communication impairments were related to precentral gyrus activation during social prediction errors. These findings elucidate the neural mechanisms of social reward prediction errors in ASD and suggest that ASD is characterized by greater neural atypicalities during social, relative to nonsocial, reward prediction errors in ASD. Autism Res 2020, 13: 715-728. © 2020 International Society for Autism Research, Wiley Periodicals, Inc. LAY SUMMARY: We used brain imaging to evaluate differences in brain activation in adolescents with autism while they performed tasks that involved learning about social and nonsocial information. We found no differences in brain responses during the nonsocial condition, but differences during the social condition of the learning task. This study provides evidence that autism may involve different patterns of brain activation when learning about social information.

Neurocomputational Changes in Inhibitory Control Associated With Prolonged Exposure Therapy.

Posttraumatic stress disorder (PTSD) is associated with inhibitory control dysfunction that extends beyond difficulties inhibiting trauma-related intrusions. Inhibitory learning has been proposed as a potential mechanism of change underlying the effectiveness of extinction-based therapies such as prolonged exposure (PE), a first-line treatment for PTSD. To identify neurocognitive markers of change in inhibitory learning associated with PE, we applied a Bayesian learning model to the analysis of neuroimaging data collected during an inhibitory control task, both before and after PE treatment. Veterans (N = 20) with combat-related PTSD completed a stop-signal task (SST) while undergoing fMRI at time points immediately before and after PE treatment. Participants exhibited a small, significant improvement in performance on the SST, as demonstrated by longer reaction times and improved inhibition accuracy. Amplitude of neural activation associated with a signed prediction error (SPE; i.e., the discrepancy between actual outcome and model-based expectation of needing to stop) in the right caudate decreased from baseline to posttreatment assessment. Change in model-based activation was modulated by performance accuracy, with a decrease in positive SPE activation observed on successful trials, d = 0.79, and a reduction in negative SPE activation on error trials, d = 0.74. The decrease in SPE-related activation on successful stop trials was correlated with PTSD symptom reduction. These results are consistent with the notion that PE may help broadly strengthen inhibitory learning and the development of more accurate model-based predictions, which may thus facilitate change in cognitions in response to trauma-related cues and help reduce PTSD symptoms.

Learning in Visual Regions as Support for the Bias in Future Value-Driven Choice.

Reinforcement learning can bias decision-making toward the option with the highest expected outcome. Cognitive learning theories associate this bias with the constant tracking of stimulus values and the evaluation of choice outcomes in the striatum and prefrontal cortex. Decisions however first require processing of sensory input, and to date, we know far less about the interplay between learning and perception. This functional magnetic resonance imaging study (N = 43) relates visual blood oxygen level–dependent (BOLD) responses to value beliefs during choice and signed prediction errors after outcomes. To understand these relationships, which co-occurred in the striatum, we sought relevance by evaluating the prediction of future value-based decisions in a separate transfer phase where learning was already established. We decoded choice outcomes with a 70% accuracy with a supervised machine learning algorithm that was given trial-by-trial BOLD from visual regions alongside more traditional motor, prefrontal, and striatal regions. Importantly, this decoding of future value-driven choice outcomes again highlighted an important role for visual activity. These results raise the intriguing possibility that the tracking of value in visual cortex is supportive for the striatal bias toward the more valued option in future choice.

Neural Mechanisms of Reward Prediction Error in Autism Spectrum Disorder.

Few studies have explored neural mechanisms of reward learning in ASD despite evidence of behavioral impairments of predictive abilities in ASD. To investigate the neural correlates of reward prediction errors in ASD, 16 adults with ASD and 14 typically developing controls performed a prediction error task during fMRI scanning. Results revealed greater activation in the ASD group in the left paracingulate gyrus during signed prediction errors and the left insula and right frontal pole during thresholded unsigned prediction errors. Findings support atypical neural processing of reward prediction errors in ASD in frontostriatal regions critical for prediction coding and reward learning. Results provide a neural basis for impairments in reward learning that may contribute to traits common in ASD (e.g., intolerance of unpredictability).

Formalizing the Function of Anterior Insula in Rapid Adaptation.

Anterior insula (aIns) is thought to play a crucial role in rapid adaptation in an ever-changing environment. Mathematically, it is known to track risk and surprise. Modern theories of learning, however, assign a dominant role to signed prediction errors (PEs), not to risk and surprise. Risk and surprise only enter to the extent that they modulate the learning rate, in an attempt to approximate Bayesian learning. Even without such modulation, adaptation is still possible, albeit slow. Here, I propose a new theory of learning, reference-model based learning (RMBL), where risk and surprise are central, and PEs play a secondary, though still crucial, role. The primary goal is to bring outcomes in line with expectations in the reference model (RM). Learning is modulated by how large the PEs are relative to model anticipation, i.e., to surprise as defined by the RM. In a target location prediction task where participants were continuously required to adapt, choices appeared to be closer with to RMBL predictions than to Bayesian learning. aIns reaction to surprise was more acute in the more difficult treatment, consistent with its hypothesized role in metacognition. I discuss links with related theories, such as Active Inference, Actor-Critic Models and Reference-Model Based Adaptive Control.

Prediction suppression and surprise enhancement in monkey inferotemporal cortex.

Exposing monkeys, over the course of days and weeks, to pairs of images presented in fixed sequence, so that each leading image becomes a predictor for the corresponding trailing image, affects neuronal visual responsiveness in area TE. At the end of the training period, neurons respond relatively weakly to a trailing image when it appears in a trained sequence and, thus, confirms prediction, whereas they respond relatively strongly to the same image when it appears in an untrained sequence and, thus, violates prediction. This effect could arise from prediction suppression (reduced firing in response to the occurrence of a probable event) or surprise enhancement (elevated firing in response to the omission of a probable event). To identify its cause, we compared firing under the prediction-confirming and prediction-violating conditions to firing under a prediction-neutral condition. The results provide strong evidence for prediction suppression and limited evidence for surprise enhancement.NEW & NOTEWORTHY In predictive coding models of the visual system, neurons carry signed prediction error signals. We show here that monkey inferotemporal neurons exhibit prediction-modulated firing, as posited by these models, but that the signal is unsigned. The response to a prediction-confirming image is suppressed, and the response to a prediction-violating image may be enhanced. These results are better explained by a model in which the visual system emphasizes unpredicted events than by a predictive coding model.

Accounting for Dynamic Fluctuations across Time when Examining fMRI Test-Retest Reliability: Analysis of a Reward Paradigm in the EMBARC Study.

Longitudinal investigation of the neural correlates of reward processing in depression may represent an important step in defining effective biomarkers for antidepressant treatment outcome prediction, but the reliability of reward-related activation is not well understood. Thirty-seven healthy control participants were scanned using fMRI while performing a reward-related guessing task on two occasions, approximately one week apart. Two main contrasts were examined: right ventral striatum (VS) activation fMRI BOLD signal related to signed prediction errors (PE) and reward expectancy (RE). We also examined bilateral visual cortex activation coupled to outcome anticipation. Significant VS PE-related activity was observed at the first testing session, but at the second testing session, VS PE-related activation was significantly reduced. Conversely, significant VS RE-related activity was observed at time 2 but not time 1. Increases in VS RE-related activity from time 1 to time 2 were significantly associated with decreases in VS PE-related activity from time 1 to time 2 across participants. Intraclass correlations (ICCs) in VS were very low. By contrast, visual cortex activation had much larger ICCs, particularly in individuals with high quality data. Dynamic changes in brain activation are widely predicted, and failure to account for these changes could lead to inaccurate evaluations of the reliability of functional MRI signals. Conventional measures of reliability cannot distinguish between changes specified by algorithmic models of neural function and noisy signal. Here, we provide evidence for the former possibility: reward-related VS activations follow the pattern predicted by temporal difference models of reward learning but have low ICCs.

Comparison of the Residual Income and the Pricing - Multiples Equity Valuation Models

The paper aims at analyzing the performance of two of the equity valuation models, the residual income (RIVM) and the pricing - multiples model. I test first how the residual income valuation model performs relative to the pricing - multiples model for a set of different value drivers and industries, second whether the performance of the different multiples increases when these are measured either with the mean, the median or the harmonic mean of the absolute prediction error and the signed prediction error. The pricing - multiples approach is in most cases a better predictor of market prices than the residual income valuation model. In addition, the harmonic mean yields to more reliable estimates of value for a set of different industries. Finally, there are some value drivers that are supposed to be more reliable than others in specific industries, but there isn’t any value driver that dominates all the industries.

Bayesian Prediction and Evaluation in the Anterior Cingulate Cortex

The dorsal anterior cingulate cortex (dACC) has been implicated in a variety of cognitive control functions, among them the monitoring of conflict, error, and volatility, error anticipation, reward learning, and reward prediction errors. In this work, we used a Bayesian ideal observer model, which predicts trial-by-trial probabilistic expectation of stop trials and response errors in the stop-signal task, to differentiate these proposed functions quantitatively. We found that dACC hemodynamic response, as measured by functional magnetic resonance imaging, encodes both the absolute prediction error between stimulus expectation and outcome, and the signed prediction error related to response outcome. After accounting for these factors, dACC has no residual correlation with conflict or error likelihood in the stop-signal task. Consistent with recent monkey neural recording studies, and in contrast with other neuroimaging studies, our work demonstrates that dACC reports at least two different types of prediction errors, and beyond contexts that are limited to reward processing.

Differential Encoding of Losses and Gains in the Human Striatum

Studies on human monetary prediction and decision making emphasize the role of the striatum in encoding prediction errors for financial reward. However, less is known about how the brain encodes financial loss. Using Pavlovian conditioning of visual cues to outcomes that simultaneously incorporate the chance of financial reward and loss, we show that striatal activation reflects positively signed prediction errors for both. Furthermore, we show functional segregation within the striatum, with more anterior regions showing relative selectivity for rewards and more posterior regions for losses. These findings mirror the anteroposterior valence-specific gradient reported in rodents and endorse the role of the striatum in aversive motivational learning about financial losses, illustrating functional and anatomical consistencies with primary aversive outcomes such as pain.