Model-based Functional Magnetic Resonance Imaging Research Articles

Individuals with problem gambling and obsessive-compulsive disorder learn through distinct reinforcement mechanisms.

Obsessive-compulsive disorder (OCD) and pathological gambling (PG) are accompanied by deficits in behavioural flexibility. In reinforcement learning, this inflexibility can reflect asymmetric learning from outcomes above and below expectations. In alternative frameworks, it reflects perseveration independent of learning. Here, we examine evidence for asymmetric reward-learning in OCD and PG by leveraging model-based functional magnetic resonance imaging (fMRI). Compared with healthy controls (HC), OCD patients exhibited a lower learning rate for worse-than-expected outcomes, which was associated with the attenuated encoding of negative reward prediction errors in the dorsomedial prefrontal cortex and the dorsal striatum. PG patients showed higher and lower learning rates for better- and worse-than-expected outcomes, respectively, accompanied by higher encoding of positive reward prediction errors in the anterior insula than HC. Perseveration did not differ considerably between the patient groups and HC. These findings elucidate the neural computations of reward-learning that are altered in OCD and PG, providing a potential account of behavioural inflexibility in those mental disorders.

Independent and interacting value systems for reward and information in the human brain.

Theories of prefrontal cortex (PFC) as optimizing reward value have been widely deployed to explain its activity in a diverse range of contexts, with substantial empirical support in neuroeconomics and decision neuroscience. Similar neural circuits, however, have also been associated with information processing. By using computational modeling, model-based functional magnetic resonance imaging analysis, and a novel experimental paradigm, we aim at establishing whether a dedicated and independent value system for information exists in the human PFC. We identify two regions in the human PFC that independently encode reward and information. Our results provide empirical evidence for PFC as an optimizer of independent information and reward signals during decision-making under realistic scenarios, with potential implications for the interpretation of PFC activity in both healthy and clinical populations.

Acute stress blunts prediction error signals in the dorsal striatum during reinforcement learning

Acute stress is pervasive in everyday modern life and is thought to affect how people make choices and learn from them. Reinforcement learning, which implicates learning from the unexpected rewarding and punishing outcomes of our choices (i.e., prediction errors), is critical for adjusted behaviour and seems to be affected by acute stress. However, the neural mechanisms by which acute stress disrupts this type of learning are still poorly understood. Here, we investigate whether and how acute stress blunts neural signalling of prediction errors during reinforcement learning using model-based functional magnetic resonance imaging. Male participants completed a well-established reinforcement-learning task involving monetary gains and losses whilst under stress and control conditions. Acute stress impaired participants’ (n = 23) behavioural performance towards obtaining monetary gains (p < 0.001), but not towards avoiding losses (p = 0.57). Importantly, acute stress blunted signalling of prediction errors during gain and loss trials in the dorsal striatum (p = 0.040) — with subsidiary analyses suggesting that acute stress preferentially blunted signalling of positive prediction errors. Our results thus reveal a neurocomputational mechanism by which acute stress may impair reward learning.

Identifying the neural dynamics of category decisions with computational model-based functional magnetic resonance imaging.

Successful categorization requires a careful coordination of attention, representation, and decision making. Comprehensive theories that span levels of analysis are key to understanding the computational and neural dynamics of categorization. Here, we build on recent work linking neural representations of category learning to computational models to investigate how category decision making is driven by neural signals across the brain. We uniquely combine functional magnetic resonance imaging with drift diffusion and exemplar-based categorization models to show that trial-by-trial fluctuations in neural activation from regions of occipital, cingulate, and lateral prefrontal cortices are linked to category decisions. Notably, only lateral prefrontal cortex activation was associated with exemplar-based model predictions of trial-by-trial category evidence. We propose that these brain regions underlie distinct functions that contribute to successful category learning.

Computational and Neurobiological Substrates of Cost-Benefit Integration in Altruistic Helping Decision

Although altruistic behaviors, e.g., sacrificing one's own interests to alleviate others' suffering, are widely observed in human society, altruism varies greatly across individuals. Such individual differences in altruistic preference have been hypothesized to arise from both individuals' dispositional empathic concern for others' welfare and context-specific cost-benefit integration processes. However, how cost-benefit integration is implemented in the brain and how it is linked to empathy remain unclear. Here, we combine a novel paradigm with the model-based fMRI approach to examine the neurocomputational basis of altruistic behaviors. Thirty-seven adults (16 females) were tested. Modeling analyses suggest that individuals are likely to integrate their own monetary costs with nonlinearly transformed recipients' benefits. Neuroimaging results demonstrate the involvement of an extended common currency system during decision-making by showing that selfish and other-regarding motives were processed in dorsal anterior cingulate cortex and right inferior parietal lobule in a domain-general manner. Importantly, a functional dissociation of adjacent but different sub-regions within anterior insular cortex (aINS) was observed for different sub-processes underlying altruistic behaviors. While dorsal aINS and inferior frontal gyrus were involved in valuation of benefactors' costs, ventral aINS and middle insular cortex (vaINS/mINS), as empathy-related regions, reflected individual variations in valuating recipients' benefits. Multivariate analyses further suggest that both vaINS/mINS and dorsolateral prefrontal cortex reflect individual variations in general altruistic preferences which account for both dispositional empathy and context-specific other-regarding tendency. Together, these findings provide valuable insights into our understanding of psychological and neurobiological basis of altruistic behaviors.SIGNIFICANCE STATEMENT:Altruistic behaviors play a crucial role in facilitating solidarity and development of human society, but the mechanisms of the cost-benefit integration underlying these behaviors are still unclear. Using model-based neuroimaging approaches, we clarify that people integrate personal costs and non-linearly transformed other's benefits during altruistic decision-making and the implementations of the integration processes are supported by an extended common currency neural network. Importantly, multivariate analyses reveal that both empathy- and cognitive control-related brain regions are involved in modulating individual variations of altruistic preference, which implicate complex psychological and computational processes. Our results provide a neurocomputational account of how people weigh between different attributes to make altruistic decisions and why altruistic preference varies to a great extent across individuals.

Neurocomputational mechanisms underpinning aberrant social learning in young adults with low self-esteem

Low self-esteem is a risk factor for a range of psychiatric disorders. From a cognitive perspective a negative self-image can be maintained through aberrant learning about self-worth derived from social feedback. We previously showed that neural teaching signals that represent the difference between expected and actual social feedback (i.e., social prediction errors) drive fluctuations in self-worth. Here, we used model-based functional magnetic resonance imaging (fMRI) to characterize learning from social prediction errors in 61 participants drawn from a population-based sample (n = 2402) who were recruited on the basis of being in the bottom or top 10% of self-esteem scores. Participants performed a social evaluation task during fMRI scanning, which entailed predicting whether other people liked them as well as the repeated provision of reported feelings of self-worth. Computational modeling results showed that low self-esteem participants had persistent expectations that others would dislike them, and a reduced propensity to update these expectations in response to social prediction errors. Low self-esteem subjects also displayed an enhanced volatility in reported feelings of self-worth, and this was linked to an increased tendency for social prediction errors to determine momentary self-worth. Canonical correlation analysis revealed that individual differences in self-esteem related to several interconnected psychiatric symptoms organized around a single dimension of interpersonal vulnerability. Such interpersonal vulnerability was associated with an attenuated social value signal in ventromedial prefrontal cortex when making predictions about being liked, and enhanced dorsal prefrontal cortex activity upon receipt of social feedback. We suggest these computational signatures of low self-esteem and their associated neural underpinnings might represent vulnerability for development of psychiatric disorder.

Neurocomputational mechanisms underlying immoral decisions benefiting self or others.

Immoral behavior often consists of weighing transgression of a moral norm against maximizing personal profits. One important question is to understand why immoral behaviors vary based on who receives specific benefits and what are the neurocomputational mechanisms underlying such moral flexibility. Here, we used model-based functional magnetic resonance imaging to investigate how immoral behaviors change when benefiting oneself or someone else. Participants were presented with offers requiring a tradeoff between a moral cost (i.e. profiting a morally bad cause) and a benefit for either oneself or a charity. Participants were more willing to obtain ill-gotten profits for themselves than for a charity, driven by a devaluation of the moral cost when deciding for their own interests. The subjective value of an immoral offer, computed as a linear summation of the weighed monetary gain and moral cost, recruited the ventromedial prefrontal cortex (PFC) regardless of beneficiaries. Moreover, paralleling the behavioral findings, this region enhanced its functional coupling with mentalizing-related regions while deciding whether to gain morally tainted profits for oneself vs charity. Finally, individual differences in moral preference differentially modulated choice-specific signals in the dorsolateral PFC according to who benefited from the decisions. These findings provide insights for understanding the neurobiological basis of moral flexibility.

Abnormal reward valuation and event-related connectivity in unmedicated major depressive disorder

Experience of emotion is closely linked to valuation. Mood can be viewed as a bias to experience positive or negative emotions and abnormally biased subjective reward valuation and cognitions are core characteristics of major depression. Thirty-four unmedicated subjects with major depressive disorder and controls estimated the probability that fractal stimuli were associated with reward, based on passive observations, so they could subsequently choose the higher of either their estimated fractal value or an explicitly presented reward probability. Using model-based functional magnetic resonance imaging, we estimated each subject's internal value estimation, with psychophysiological interaction analysis used to examine event-related connectivity, testing hypotheses of abnormal reward valuation and cingulate connectivity in depression. Reward value encoding in the hippocampus and rostral anterior cingulate was abnormal in depression. In addition, abnormal decision-making in depression was associated with increased anterior mid-cingulate activity and a signal in this region encoded the difference between the values of the two options. This localised decision-making and its impairment to the anterior mid-cingulate cortex (aMCC) consistent with theories of cognitive control. Notably, subjects with depression had significantly decreased event-related connectivity between the aMCC and rostral cingulate regions during decision-making, implying impaired communication between the neural substrates of expected value estimation and decision-making in depression. Our findings support the theory that abnormal neural reward valuation plays a central role in major depressive disorder (MDD). To the extent that emotion reflects valuation, abnormal valuation could explain abnormal emotional experience in MDD, reflect a core pathophysiological process and be a target of treatment.

Cortical encoding of speech enhances task-relevant acoustic information.

Speech is the most important signal in our auditory environment, and the processing of speech is highly dependent on context. However, it is unknown how contextual demands influence the neural encoding of speech. Here, we examine the context dependence of auditory cortical mechanisms for speech encoding at the level of the representation of fundamental acoustic features (spectrotemporal modulations) using model-based functional magnetic resonance imaging. We found that the performance of different tasks on identical speech sounds leads to neural enhancement of the acoustic features in the stimuli that are critically relevant to task performance. These task effects were observed at the earliest stages of auditory cortical processing, in line with interactive accounts of speech processing. Our work provides important insights into the mechanisms that underlie the processing of contextually relevant acoustic information within our rich and dynamic auditory environment.

Dynamic integration of conceptual information during learning

The development and application of concepts is a critical component of cognition. Although concepts can be formed on the basis of simple perceptual or semantic features, conceptual representations can also capitalize on similarities across feature relationships. By representing these types of higher-order relationships, concepts can simplify the learning problem and facilitate decisions. Despite this, little is known about the neural mechanisms that support the construction and deployment of these kinds of higher-order concepts during learning. To address this question, we combined a carefully designed associative learning task with computational model-based functional magnetic resonance imaging (fMRI). Participants were scanned as they learned and made decisions about sixteen pairs of cues and associated outcomes. Associations were structured such that individual cues shared feature relationships, operationalized as shared patterns of cue pair-outcome associations. In order to capture the large number of possible conceptual representational structures that participants might employ and to evaluate how conceptual representations are used during learning, we leveraged a well-specified Bayesian computational model of category learning [1]. Behavioral and model-based results revealed that participants who displayed a tendency to link experiences in memory benefitted from faster learning rates, suggesting that the use of the conceptual structure in the task facilitated decisions about cue pair-outcome associations. Model-based fMRI analyses revealed that trial-by-trial integration of cue information into higher-order conceptual representations was supported by an anterior temporal (AT) network of regions previously implicated in representing complex conjunctions of features and meaning-based information.

Neural mechanisms for learning self and other ownership

Sense of ownership is a ubiquitous and fundamental aspect of human cognition. Here we used model-based functional magnetic resonance imaging and a novel minimal ownership paradigm to probe the behavioural and neural mechanisms underpinning ownership acquisition for ourselves, friends and strangers. We find a self-ownership bias at multiple levels of behaviour from initial preferences to reaction times and computational learning rates. Ventromedial prefrontal cortex (vmPFC) and anterior cingulate sulcus (ACCs) responded more to self vs. stranger associations, but despite a pervasive neural bias to track self-ownership, no brain area tracked self-ownership exclusively. However, ACC gyrus (ACCg) specifically coded ownership prediction errors for strangers and ownership associative strength for friends and strangers but not for self. Core neural mechanisms for associative learning are biased to learn in reference to self but also engaged when learning in reference to others. In contrast, ACC gyrus exhibits specialization for learning about others.

S154. THE ROLE OF DOPAMINE IN PROCESSING THE MEANINGFUL INFORMATION OF OBSERVATIONS, AND IMPLICATIONS FOR THE ABERRANT SALIENCE HYPOTHESIS OF SCHIZOPHRENIA

BackgroundThe aberrant salience hypothesis of schizophrenia proposes that symptoms such as paranoia arise when behavioural salience is attributed to neutral stimuli. Mesolimbic dopamine dysfunction is thought to be central to this mechanism; building on findings that activity in this pathway conveys a (signed) reward prediction error signal. Given that many psychotic symptoms are not explicitly related to reward learning, it is relevant that recent studies in rodents have demonstrated a role for midbrain dopamine neurons in value-neutral associative learning. Direct evidence for this role in humans, however, is lacking.In this study we asked whether the mesolimbic dopamine circuit is involved in encoding the value-neutral meaningful information of observations, using a model-based functional magnetic resonance imaging (fMRI) task and dopamine positron emission tomography (PET). We define ‘meaningful information’ as the degree to which an observation results in a belief-update to an agent’s internal model of the environment (Kullback-Leibler divergence from prior to posterior beliefs; ‘Bayesian surprise’).MethodsParticipants were tasked to infer the current (hidden) state of the environment, using partially-informative observations at each trial, and then report their belief at the end of each trial. Participant beliefs were modelled using a Hidden Markov Model of the task and iterative application of Bayes’ rule, allowing us to quantify the Bayesian surprise (meaningful information content) associated with a trial observation. Crucially, our task de-correlated Bayesian surprise from both the pure sensory unexpectedness of an observation (unexpected but meaningless information) and its signed reward prediction error. 39 healthy participants (22M, mean age 26y) performed 180 task trials within an fMRI scanner. 36 participants also had a [11C]-(+)-4-propyl-9-hydroxy-naphthoxazine (PHNO) PET scan to quantify dopamine-2/3 receptor (D2/3R) availability. 17 participants additionally had a second PET scan 3hrs post 0.5mg/kg oral dexamphetamine, to quantify striatal dopamine release capacity. Neuroimaging analyses were restricted to the bilateral substantia nigra/ventral tegmental area (SN/VTA) and ventral striatum (VS).ResultsOur computational model closely predicted participant behaviour (R2= .67), and there was a negative correlation between subclinical paranoia and the degree to which participant behaviour approximated normative Bayesian performance (rho = -.60, P<0.001). Neuronal activation encoding the meaningful information content of an observation (Bayesian surprise) was present in SN/VTA and VS (both P(peak)<0.05, SVC), whereas no such encoding was present for sensory unexpectedness or reward-prediction error. Crucially, activation encoding Bayesian surprise was inversely correlated with D2/3R availability in the SN/VTA (rho = -.43, P=0.009), consistent with a tonic inhibitory role for midbrain D2/3Rs. Moreover, activation encoding Bayesian surprise was inversely related to dopamine release capacity in the VS (rho = -.66, P=0.005), indicating that subjects with high dopamine release capacity showed blunted striatal activation in response to belief-changing information, as is also found in schizophrenia.DiscussionWe provide direct evidence in humans that a mesolimbic dopamine circuit is involved in encoding the meaningful information content of observations, distinct from its involvement in processing signed reward prediction error. These results implicate dopamine in a wider range of function than reward learning, including updating a predictive associative model of the world, and are therefore relevant for the aberrant salience hypothesis of schizophrenia.

A neurocomputational investigation of reinforcement-based decision making as a candidate latent vulnerability mechanism in maltreated children.

Alterations in reinforcement-based decision making may be associated with increased psychiatric vulnerability in children who have experienced maltreatment. A probabilistic passive avoidance task and a model-based functional magnetic resonance imaging analytic approach were implemented to assess the neurocomputational components underlying decision making: (a) reinforcement expectancies (the representation of the outcomes associated with a stimulus) and (b) prediction error signaling (the ability to detect the differences between expected and actual outcomes). There were three main findings. First, the maltreated group (n = 18; mean age = 13), relative to nonmaltreated peers (n = 19; mean age = 13), showed decreased activity during expected value processing in a widespread network commonly associated with reinforcement expectancies representation, including the striatum (especially the caudate), the orbitofrontal cortex, and medial temporal structures including the hippocampus and insula. Second, consistent with previously reported hyperresponsiveness to negative cues in the context of childhood abuse, the maltreated group showed increased prediction error signaling in the middle cingulate gyrus, somatosensory cortex, superior temporal gyrus, and thalamus. Third, the maltreated group showed increased activity in frontodorsal regions and in the putamen during expected value representation. These findings suggest that early adverse environments disrupt the development of decision-making processes, which in turn may compromise psychosocial functioning in ways that increase latent vulnerability to psychiatric disorder.

The Neuro-Computational Architecture of Value-Based Selection in the Human Brain.

Current neural models of value-based decision-making consider choices as a 2-stage process, proceeding from the "valuation" of each option under consideration to the "selection" of the best option on the basis of their subjective values. However, little is known about the computational mechanisms at play at the selection stage and its implementation in the human brain. Here, we used drift-diffusion models combined with model-based functional magnetic resonance imaging, effective connectivity, and multivariate pattern analysis to characterize the neuro-computational architecture of value-based decisions. We found that 2 key drift-diffusion computations at the selection stage, namely integration and choice readout, engage distinct brain regions, with the dorsolateral prefrontal cortex integrating a decision value signal computed in the ventromedial prefrontal cortex, and the posterior parietal cortex reading out choice outcomes. Our findings suggest that this prefronto-parietal network acts as a hub implementing behavioral selection through a distributed drift-diffusion process.

Belief about Nicotine Modulates Subjective Craving and Insula Activity in Deprived Smokers.

Little is known about the specific neural mechanisms through which cognitive factors influence craving and associated brain responses, despite the initial success of cognitive therapies in treating drug addiction. In this study, we investigated how cognitive factors such as beliefs influence subjective craving and neural activities in nicotine-addicted individuals using model-based functional magnetic resonance imaging (fMRI) and neuropharmacology. Deprived smokers (N = 24) participated in a two-by-two balanced placebo design, which crossed beliefs about nicotine (told “nicotine” vs. told “no nicotine”) with the nicotine content in a cigarette (nicotine vs. placebo) which participants smoked immediately before performing a fMRI task involving reward learning. Subjects’ reported craving was measured both before smoking and after the fMRI session. We found that first, in the presence of nicotine, smokers demonstrated significantly reduced craving after smoking when told “nicotine in cigarette” but showed no change in craving when told “no nicotine.” Second, neural activity in the insular cortex related to craving was only significant when smokers were told “nicotine” but not when told “no nicotine.” Both effects were absent in the placebo condition. Third, insula activation related to computational learning signals was modulated by belief about nicotine regardless of nicotine’s presence. These results suggest that belief about nicotine has a strong impact on subjective craving and insula responses related to both craving and learning in deprived smokers, providing insights into the complex nature of belief–drug interactions.

Neural Substrates Related to Motor Memory with Multiple Timescales in Sensorimotor Adaptation.

Recent computational and behavioral studies suggest that motor adaptation results from the update of multiple memories with different timescales. Here, we designed a model-based functional magnetic resonance imaging (fMRI) experiment in which subjects adapted to two opposing visuomotor rotations. A computational model of motor adaptation with multiple memories was fitted to the behavioral data to generate time-varying regressors of brain activity. We identified regional specificity to timescales: in particular, the activity in the inferior parietal region and in the anterior-medial cerebellum was associated with memories for intermediate and long timescales, respectively. A sparse singular value decomposition analysis of variability in specificities to timescales over the brain identified four components, two fast, one middle, and one slow, each associated with different brain networks. Finally, a multivariate decoding analysis showed that activity patterns in the anterior-medial cerebellum progressively represented the two rotations. Our results support the existence of brain regions associated with multiple timescales in adaptation and a role of the cerebellum in storing multiple internal models.

Mechanisms and Model-Based Functional Magnetic Resonance Imaging

Mechanistic explanations satisfy widely held norms of explanation: the ability to manipulate and answer counterfactual questions about the explanandum phenomenon. A currently debated issue is whether any nonmechanistic explanations can satisfy these explanatory norms. Weiskopf argues that the models of object recognition and categorization, JIM, SUSTAIN, and ALCOVE, are not mechanistic yet satisfy these norms of explanation. In this article I argue that these models are mechanism sketches. My argument applies recent research using model-based functional magnetic resonance imaging, a novel neuroimaging method whose significance for current debates on psychological models and mechanistic explanation has yet to be explored.

An insula-frontostriatal network mediates flexible cognitive control by adaptively predicting changing control demands

The anterior cingulate and lateral prefrontal cortices have been implicated in implementing context-appropriate attentional control, but the learning mechanisms underlying our ability to flexibly adapt the control settings to changing environments remain poorly understood. Here we show that human adjustments to varying control demands are captured by a reinforcement learner with a flexible, volatility-driven learning rate. Using model-based functional magnetic resonance imaging, we demonstrate that volatility of control demand is estimated by the anterior insula, which in turn optimizes the prediction of forthcoming demand in the caudate nucleus. The caudate's prediction of control demand subsequently guides the implementation of proactive and reactive attentional control in dorsal anterior cingulate and dorsolateral prefrontal cortices. These data enhance our understanding of the neuro-computational mechanisms of adaptive behaviour by connecting the classic cingulate-prefrontal cognitive control network to a subcortical control-learning mechanism that infers future demands by flexibly integrating remote and recent past experiences.

Cerebral Correlates of Salient Prediction Error for Different Rewards and Punishments

Learning to predict rewarding and aversive outcomes is based on the comparison between predicted and actual outcomes (prediction error: PE). Recent electrophysiological studies reported that during a Pavlovian procedure some dopamine neurons code a classical PE signal while a larger population of dopaminergic neurons reflect a "salient" prediction error (SPE) signal, being excited both by unpredictable aversive events and by rewards. Yet, it is still unclear whether specific human brain structures receiving afferents from dopaminergic neurons code a SPE and whether this signal depends upon reinforcer type. Here, we used a model-based functional magnetic resonance imaging approach implementing a reinforcement learning model to compute the PE while subjects underwent a Pavlovian conditioning procedure with 2 types of rewards (pleasant juice and monetary gain) and 2 types of punishments (aversive juice and aversive picture). The results revealed that activity of a brain network composed of the striatum, anterior insula, and anterior cingulate cortex covaried with a SPE for appetitive and aversive juice. Moreover, amygdala activity correlated with a SPE for these 2 reinforcers and for aversive pictures. These results provide insights into the neurobiological mechanisms underlying the ability to learn stimuli-rewards and stimuli-punishments contingencies, by demonstrating that the network reflecting the SPE depends upon reinforcement's type.

Altered Neural Reward Representations in Pathological Gamblers Revealed by Delay and Probability Discounting

The neural basis of excessive delay discounting and reduced risk sensitivity of pathological gamblers with a particular focus on subjective neural reward representations has not been previously examined. To examine how pathological gamblers represent subjective reward value at a neural level and how this is affected by gambling severity. Model-based functional magnetic resonance imaging study with patients and control subjects. Department of Systems Neuroscience, University Medical Center Hamburg-Eppendorf. Participants were recruited from the local community by advertisement and through self-help groups. A sample of 16 pathological gamblers (according to the DSM-IV definition) was matched by age, sex, smoking status, income, educational level, and handedness to 16 healthy controls. Pathological gamblers showed increased discounting of delayed rewards and a trend toward decreased discounting of probabilistic rewards compared with matched controls. At the neural level, a significant group×condition interaction indicated that reward representations in the gamblers were modulated in a condition-specific manner, such that they exhibited increased (delay discounting) and decreased (probability discounting) neural value correlations in the reward system. In addition, throughout the reward system, neuronal value signals for delayed rewards were negatively correlated with gambling severity. The results extend previous reports of a generally hypoactive reward system in pathological gamblers by showing that, even when subjective reward valuation is accounted for, gamblers still show altered reward representations. Furthermore, results point toward a gradual degradation of mesolimbic reward representations for delayed rewards during the course of pathological gambling.