Reward-based Task Research Articles

Distinct dynamics of ramping activity in the frontal cortex and caudate nucleus in monkeys.

The prefronto-striatal network is involved in many cognitive functions, including perceptual decision making and reward-modulated behaviors. For well-trained subjects, neural responses frequently show similar patterns in the prefrontal cortex and striatum, making it difficult to tease apart distinct regional contributions. Here I show that, despite similar mean firing rate patterns, prefrontal and striatal responses differ in other temporal dynamics for both perceptual and reward-based tasks. Compared with simulation results, the temporal dynamics of prefrontal activity are consistent with an accumulation of sensory evidence used to solve a perceptual task but not with an accumulation of reward context-related information used for the development of a reward bias. In contrast, the dynamics of striatal activity is consistent with an accumulation of reward context-related information and with an accumulation of sensory evidence during early stimulus viewing. These results suggest that prefrontal and striatal neurons may have specialized functions for different tasks even with similar average activity.

FMRI and EEG predictors of dynamic decision parameters during human reinforcement learning.

What are the neural dynamics of choice processes during reinforcement learning? Two largely separate literatures have examined dynamics of reinforcement learning (RL) as a function of experience but assuming a static choice process, or conversely, the dynamics of choice processes in decision making but based on static decision values. Here we show that human choice processes during RL are well described by a drift diffusion model (DDM) of decision making in which the learned trial-by-trial reward values are sequentially sampled, with a choice made when the value signal crosses a decision threshold. Moreover, simultaneous fMRI and EEG recordings revealed that this decision threshold is not fixed across trials but varies as a function of activity in the subthalamic nucleus (STN) and is further modulated by trial-by-trial measures of decision conflict and activity in the dorsomedial frontal cortex (pre-SMA BOLD and mediofrontal theta in EEG). These findings provide converging multimodal evidence for a model in which decision threshold in reward-based tasks is adjusted as a function of communication from pre-SMA to STN when choices differ subtly in reward values, allowing more time to choose the statistically more rewarding option.

The effects of task difficulty, novelty and the size of the search space on intrinsically motivated exploration

Devising efficient strategies for exploration in large open-ended spaces is one of the most difficult computational problems of intelligent organisms. Because the available rewards are ambiguous or unknown during the exploratory phase, subjects must act in intrinsically motivated fashion. However, a vast majority of behavioral and neural studies to date have focused on decision making in reward-based tasks, and the rules guiding intrinsically motivated exploration remain largely unknown. To examine this question we developed a paradigm for systematically testing the choices of human observers in a free play context. Adult subjects played a series of short computer games of variable difficulty, and freely choose which game they wished to sample without external guidance or physical rewards. Subjects performed the task in three distinct conditions where they sampled from a small or a large choice set (7 vs. 64 possible levels of difficulty), and where they did or did not have the possibility to sample new games at a constant level of difficulty. We show that despite the absence of external constraints, the subjects spontaneously adopted a structured exploration strategy whereby they (1) started with easier games and progressed to more difficult games, (2) sampled the entire choice set including extremely difficult games that could not be learnt, (3) repeated moderately and high difficulty games much more frequently than was predicted by chance, and (4) had higher repetition rates and chose higher speeds if they could generate new sequences at a constant level of difficulty. The results suggest that intrinsically motivated exploration is shaped by several factors including task difficulty, novelty and the size of the choice set, and these come into play to serve two internal goals—maximize the subjects' knowledge of the available tasks (exploring the limits of the task set), and maximize their competence (performance and skills) across the task set.

Dual role of serotonin in the acquisition and extinction of reward-driven learning: Involvement of 5-HT1A, 5-HT2A and 5-HT3 receptors

Serotonin (5-HT) has been proposed as a possible encoder of reward. Nevertheless, the role of this neurotransmitter in reward-based tasks is not well understood. Given that the major serotonergic circuit in the rat brain comprises the dorsal raphe nuclei and the medial prefrontal cortex (mPFC), and because the latter structure is involved in the control of complex behaviors and expresses 1A (5-HT1A), 2A (5-HT2A), and 3 (5-HT3) receptors, the aim was to study the role of 5-HT and of these receptors in the acquisition and extinction of a reward-dependent operant conditioning task. Long Evans rats were trained in an operant conditioning task while receiving fluoxetine (serotonin reuptake inhibitor, 10mg/kg), tianeptine (serotonin reuptake enhancer, 10mg/kg), buspirone (5-HT1A partial agonist, 10mg/kg), risperidone (5-HT2A antagonist, 1mg/kg), ondansetron (5-HT3 antagonist, 2mg/kg) or vehicle. Then, animals that acquired the operant conditioning without any treatment were trained to extinct the task in the presence of the pharmacological agents. Fluoxetine impaired acquisition but improved extinction. Tianeptine administration induced the opposite effects. Buspirone induced a mild deficit in acquisition and had no effects during the extinction phase. Risperidone administration resulted in learning deficits during the acquisition phase, although it promoted improved extinction. Ondansetron treatment showed a deleterious effect in the acquisition phase and an overall improvement in the extinction phase. These data showed a differential role of 5-HT in the acquisition and extinction of an operant conditioning task, suggesting that it may have a dual function in reward encoding.

Adult age differences in frontostriatal representation of prediction error but not reward outcome.

Emerging evidence from decision neuroscience suggests that although younger and older adults show similar frontostriatal representations of reward magnitude, older adults often show deficits in feedback-driven reinforcement learning. In the present study, healthy adults completed reward-based tasks that did or did not depend on probabilistic learning, while undergoing functional neuroimaging. We observed reductions in the frontostriatal representation of prediction errors during probabilistic learning in older adults. In contrast, we found evidence for stability across adulthood in the representation of reward outcome in a task that did not require learning. Together, the results identify changes across adulthood in the dynamic coding of relational representations of feedback, in spite of preserved reward sensitivity in old age. Overall, the results suggest that the neural representation of prediction error, but not reward outcome, is reduced in old age. These findings reveal a potential dissociation between cognition and motivation with age and identify a potential mechanism for explaining changes in learning-dependent decision making in old adulthood.Electronic supplementary materialThe online version of this article (doi:10.3758/s13415-014-0297-4) contains supplementary material, which is available to authorized users.

Taking Pleasure at Another's Misfortune: The Implicit Schadenfreude of Disaster Spectators

Implicit Schadenfreude of disaster spectators, with augmentation of implicit self-esteem and higher reward responsiveness as indicators, and the influence of insecurity and masculinity was explored in two studies. Both studies were conducted under conditions without any clearly legitimizing factors. Experimental (priming with disaster video) and control (priming with neutral video) groups were compared using the Implicit Association Test in Study 1, the results of which showed augmentation of implicit self-esteem in the experimental group. In Study 2, participants who read and believed a fabricated news report of fire disaster via a local area network website showed higher reward responsiveness in a reward-based signal-detection task. Other results showed that insecurity negatively predicted the augmentation of implicit self-esteem and reward responsiveness, while masculinity only negatively predicted the augmentation of implicit self-esteem.

Dopaminergic Influences on Emotional Decision Making in Euthymic Bipolar Patients

We recently reported that the D2/D3 agonist pramipexole may have pro-cognitive effects in euthymic patients with bipolar disorder (BPD); however, the emergence of impulse-control disorders has been documented in Parkinson's disease (PD) after pramipexole treatment. Performance on reward-based tasks is altered in healthy subjects after a single dose of pramipexole, but its potential to induce abnormalities in BPD patients is unknown. We assessed reward-dependent decision making in euthymic BPD patients pre- and post 8 weeks of treatment with pramipexole or placebo by using the Iowa Gambling Task (IGT). The IGT requires subjects to choose among four card decks (two risky and two conservative) and is designed to promote learning to make advantageous (conservative) choices over time. Thirty-four BPD patients completed both assessments (18 placebo and 16 pramipexole). Baseline performance did not differ by treatment group (F=0.63; p=0.64); however, at week 8, BPD patients on pramipexole demonstrated a significantly greater tendency to make increasingly high-risk, high-reward choices across the five blocks, whereas the placebo group's pattern was similar to that reported in healthy individuals (treatment × time × block interaction, p<0.05). Analyses of choice strategy using the expectancy valence model revealed that after 8 weeks on pramipexole, BPD patients attended more readily to feedback related to gains than to losses, which could explain the impaired learning. There were no significant changes in mood symptoms over the 8 weeks, and no increased propensity toward manic-like behaviors were reported. Our results suggest that the enhancement of dopaminergic activity influences risk-associated decision-making performance in euthymic BPD. The clinical implications remain unknown.

Compositional real-time scheduling framework for periodic reward-based task model

As the size and complexity of embedded software systems increase, compositional real-time scheduling framework is widely accepted as means to build large and complex systems. A compositional real-time scheduling framework proposes to decompose a system into independent subsystems and provides ways to assemble them into a flexible hierarchical real-time scheduling system while guaranteeing the internal real-time requirements of each subsystem. In this paper, we consider the imprecise reward-based periodic task model in compositional scheduling framework. Thus, we introduce the imprecise periodic resource model to characterize the imprecise resource allocations provided by the system to a single component, and the interface model to abstract the imprecise real-time requirements of the component. The schedulability of mandatory parts is also analyzed to meet the minimum requirement of tasks. Finally, we provide a scheduling algorithm to guarantee a certain amount of reward, which makes it feasible to efficiently compose multiple imprecise components.

Dissociable Effects of Subtotal Lesions within the Macaque Orbital Prefrontal Cortex on Reward-Guided Behavior

The macaque orbital prefrontal cortex (PFo) has been implicated in a wide range of reward-guided behaviors essential for efficient foraging. The PFo, however, is not a homogeneous structure. Two major subregions, distinct by their cytoarchitecture and connections to other brain structures, compose the PFo. One subregion encompasses Walker's areas 11 and 13 and the other centers on Walker's area 14. Although it has been suggested that these subregions play dissociable roles in reward-guided behavior, direct neuropsychological evidence for this hypothesis is limited. To explore the independent contributions of PFo subregions to behavior, we studied rhesus monkeys (Macaca mulatta) with restricted excitotoxic lesions targeting either Walker's areas 11/13 or area 14. The performance of these two groups was compared to that of a group of unoperated controls on a series of reward-based tasks that has been shown to be sensitive to lesions of the PFo as a whole (Walker's areas 11, 13, and 14). Lesions of areas 11/13, but not area 14, disrupted the rapid updating of object value during selective satiation. In contrast, lesions targeting area 14, but not areas 11/13, impaired the ability of monkeys to learn to stop responding to a previously rewarded object. Somewhat surprisingly, neither lesion disrupted performance on a serial object reversal learning task, although aspiration lesions of the entire PFo produce severe deficits on this task. Our data indicate that anatomically defined subregions within macaque PFo make dissociable contributions to reward-guided behavior.

Quality of Care for Chronic Diseases in a British Cohort of Long-Term Cancer Survivors

Previous research has shown that long-term cancer survivors with other chronic diseases may receive poorer care for those diseases compared with the general population. We sought to establish the quality of care for chronic diseases among cancer survivors in the United Kingdom. From the UK General Practice Research Database, we identified 21,366 adult patients who had survived 5 or more years after a diagnosis of breast, colorectal, or prostate cancer with a diagnosis of hypertension, coronary artery disease, diabetes, or cerebrovascular disease. For each patient, an age-sex matched noncancer control patient was selected from the same general practice and with the same chronic disease. We compared the chronic disease care in cancer survivors and their matched controls. The proportion of patients meeting quality standards for chronic disease care was high in both cancer survivors and control patients. Although cancer survivors were slightly less likely to receive blood pressure monitoring and cholesterol tests, this difference was no longer apparent if patients who died during the study period were excluded. For instance, 93% of breast cancer survivors received blood pressure monitoring compared with 94% of matched control patients. Similarly, control of disease was comparable among all patients, with the exception of diabetic prostate cancer survivors, who had fewer cholesterol readings under the control limit (17% reduction, 95% CI, 7%-26%) and diabetic colorectal survivors, who had fewer calendar quarters of glycated hemoglobin control (12% reduction, 95% CI, 2%-23%). Care of comorbidities is not neglected in the United Kingdom because people have had a previous diagnosis of cancer. One explanation is that in the United Kingdom, such care is provided through a robust primary care system.

Towards a cognitive robotics methodology for reward-based decision-making: dynamical systems modelling of the Iowa Gambling Task

The somatic marker hypothesis (SMH) posits that the role of emotions and mental states in decision-making manifests through bodily responses to stimuli of import to the organism's welfare. The Iowa Gambling Task (IGT), proposed by Bechara and Damasio in the mid-1990s, has provided the major source of empirical validation to the role of somatic markers in the service of flexible and cost-effective decision-making in humans. In recent years the IGT has been the subject of much criticism concerning: (1) whether measures of somatic markers reveal that they are important for decision-making as opposed to behaviour preparation; (2) the underlying neural substrate posited as critical to decision-making of the type relevant to the task; and (3) aspects of the methodological approach used, particularly on the canonical version of the task. In this paper, a cognitive robotics methodology is proposed to explore a dynamical systems approach as it applies to the neural computation of reward-based learning and issues concerning embodiment. This approach is particularly relevant in light of a strongly emerging alternative hypothesis to the SMH, the reversal learning hypothesis, which links, behaviourally and neurocomputationally, a number of more or less complex reward-based decision-making tasks, including the ‘A-not-B’ task – already subject to dynamical systems investigations with a focus on neural activation dynamics. It is also suggested that the cognitive robotics methodology may be used to extend systematically the IGT benchmark to more naturalised, but nevertheless controlled, settings that might better explore the extent to which the SMH, and somatic states per se, impact on complex decision-making.

Parkinson's disease and dopaminergic therapy—differential effects on movement, reward and cognition

Cognitive deficits are very common in Parkinson's disease particularly for ‘executive functions’ associated with frontal cortico-striatal networks. Previous work has identified deficits in tasks that require attentional control like task-switching, and reward-based tasks like gambling or reversal learning. However, there is a complex relationship between the specific cognitive problems faced by an individual patient, their stage of disease and dopaminergic treatment. We used a bimodality continuous performance task during fMRI to examine how patients with Parkinson's disease represent the prospect of reward and switch between competing task rules accordingly. The task-switch was not separately cued but was based on the implicit reward relevance of spatial and verbal dimensions of successive compound stimuli. Nineteen patients were studied in relative ‘on’ and ‘off’ states, induced by dopaminergic medication withdrawal (Hoehn and Yahr stages 1–4). Patients were able to successfully complete the task and establish a bias to one or other dimension in order to gain reward. However the lateral prefrontal cortex and caudate nucleus showed a non-linear U-shape relationship between motor disease severity and regional brain activation. Dopaminergic treatment led to a shift in this U-shape function, supporting the hypothesis of differential neurodegeneration in separate motor and cognitive cortico–striato–thalamo–cortical circuits. In addition, anterior cingulate activation associated with reward expectation declined with more severe disease, whereas activation following actual rewards increased with more severe disease. This may facilitate a change in goal-directed behaviours from deferred predicted rewards to immediate actual rewards, particularly when on dopaminergic treatment. We discuss the implications for investigation and optimal treatment of this common condition at different stages of disease.

Focal basal ganglia lesions are associated with impairments in reward-based reversal learning

The basal ganglia (BG) are thought to play a key role in learning from feedback, with mesencephalic dopamine neurons coding errors in reward prediction, thereby mediating information processing in the BG and the prefrontal cortex. In the present study, reward-based learning was assessed in patients with focal BG lesions, by studying outcome-based acquisition and reversal of stimulus-stimulus associations with different reward magnitudes in two probabilistic learning tasks. Eleven patients with selective BG lesions (three females) and 18 healthy control subjects (six females) participated in this study. Two cognitive transfer tasks provided a measure of declarative learning strategy application. On the group level, BG patients showed deficits in reversal learning, with dorsal striatum lesion patients being most severely affected. While basic mechanisms of learning from feedback such as the processing of different reward magnitudes appeared to be intact, patients needed more trials than controls to learn a second reward-based task, suggesting reduced carry-over effects in learning. A patient with a bilateral BG lesion showed better performance than controls on most learning tasks, applying a compensatory declarative learning strategy. The results are discussed in terms of the implication of different BG subregions in different aspects of learning from feedback.