Probabilistic Reinforcement Learning Research Articles

Differential neural reward mechanisms in treatment-responsive and treatment-resistant schizophrenia.

The significant proportion of schizophrenia patients refractory to treatment, primarily directed at the dopamine system, suggests that multiple mechanisms may underlie psychotic symptoms. Reinforcement learning tasks have been employed in schizophrenia to assess dopaminergic functioning and reward processing, but these have not directly compared groups of treatment-refractory and non-refractory patients. In the current functional magnetic resonance imaging study, 21 patients with treatment-resistant schizophrenia (TRS), 21 patients with non-treatment-resistant schizophrenia (NTR), and 24 healthy controls (HC) performed a probabilistic reinforcement learning task, utilizing emotionally valenced face stimuli which elicit a social bias toward happy faces. Behavior was characterized with a reinforcement learning model. Trial-wise reward prediction error (RPE)-related neural activation and the differential impact of emotional bias on these reward signals were compared between groups. Patients showed impaired reinforcement learning relative to controls, while all groups demonstrated an emotional bias favoring happy faces. The pattern of RPE signaling was similar in the HC and TRS groups, whereas NTR patients showed significant attenuation of RPE-related activation in striatal, thalamic, precentral, parietal, and cerebellar regions. TRS patients, but not NTR patients, showed a positive relationship between emotional bias and RPE signal during negative feedback in bilateral thalamus and caudate. TRS can be dissociated from NTR on the basis of a different neural mechanism underlying reinforcement learning. The data support the hypothesis that a favorable response to antipsychotic treatment is contingent on dopaminergic dysfunction, characterized by aberrant RPE signaling, whereas treatment resistance may be characterized by an abnormality of a non-dopaminergic mechanism - a glutamatergic mechanism would be a possible candidate.

Monopolistic Models for Resource Allocation: A Probabilistic Reinforcement Learning Approach

Under the environment of cognitive radio networks, users are equipped with intelligent capabilities so that they can sense the conditions of networks and act optimally to maximize their revenues. Thus, dynamic spectrum access (DSA), which focuses on the management and distribution of the resources, is a critical problem, especially when only limited resources are available. Currently, this problem is handled by the game theory and auction theory, since this is a problem involving multiple agents. In this paper, we propose agent-based modeling method to model this multi-agent environment and probabilistic reinforcement learning to learn the optimal strategies. We focus on a simple scenario with only one primary user (PU) and multiple secondary users (SUs), and try to maximize the revenues of both sides, which may be further extended to other scenarios with different number of agents. First, we model this environment as a monopolistic market from the perspective of economics, where a PU acts as the monopoly and the SUs are passive buyers. Then, we propose probabilistic reinforcement learning methods to handle DSA so that both the PU and SUs can behave optimally by learning from the feedback of the others. Experimental results prove the flexibility and superior performance of our proposed methods.

The Habenula's Role in Adaptive Behaviors: Contributions From Neuroimaging.

The Habenula's Role in Adaptive Behaviors: Contributions From Neuroimaging.

Supplemental Material for Probabilistic Reinforcement Learning Abnormalities and Their Correlates in Adolescent Bipolar Disorders

Supplemental Material for Probabilistic Reinforcement Learning Abnormalities and Their Correlates in Adolescent Bipolar Disorders

Intact striatal dopaminergic modulation of reward learning and daily-life reward-oriented behavior in first-degree relatives of individuals with psychotic disorder.

Abnormalities in reward learning in psychotic disorders have been proposed to be linked to dysregulated subcortical dopaminergic (DA) neurotransmission, which in turn is a suspected mechanism for predisposition to psychosis. We therefore explored the striatal dopaminergic modulation of reward processing and its behavioral correlates in individuals at familial risk for psychosis. We performed a DA D2/3 receptor [18F]fallypride positron emission tomography scan during a probabilistic reinforcement learning task in 16 healthy first-degree relatives of patients with psychosis and 16 healthy volunteers, followed by a 6-day ecological momentary assessment study capturing reward-oriented behavior in the everyday life. We detected significant reward-induced DA release in bilateral caudate, putamen and ventral striatum of both groups, with no group differences in its magnitude nor spatial extent. In both groups alike, greater extent of reward-induced DA release in all regions of interest was associated with better performance in the task, as well as in greater tendency to be engaged in reward-oriented behavior in the daily life. These findings suggest intact striatal dopaminergic modulation of reinforcement learning and reward-oriented behavior in individuals with familial predisposition to psychosis. Furthermore, this study points towards a key link between striatal reward-related DA release and pursuit of ecologically relevant rewards.

Association between habenula dysfunction and motivational symptoms in unmedicated major depressive disorder.

The lateral habenula plays a central role in reward and punishment processing and has been suggested to drive the cardinal symptom of anhedonia in depression. This hypothesis is largely based on observations of habenula hypermetabolism in animal models of depression, but the activity of habenula and its relationship with clinical symptoms in patients with depression remains unclear. High-resolution functional magnetic resonance imaging (fMRI) and computational modelling were used to investigate the activity of the habenula during a probabilistic reinforcement learning task with rewarding and punishing outcomes in 21 unmedicated patients with major depression and 17 healthy participants. High-resolution anatomical scans were also acquired to assess group differences in habenula volume. Healthy individuals displayed the expected activation in the left habenula during receipt of punishment and this pattern was confirmed in the computational analysis of prediction error processing. In depressed patients, there was a trend towards attenuated left habenula activation to punishment, while greater left habenula activation was associated with more severe depressive symptoms and anhedonia. We also identified greater habenula volume in patients with depression, which was associated with anhedonic symptoms. Habenula dysfunction may contribute to abnormal response to punishment in patients with depression, and symptoms such as anhedonia.

Striatal dopaminergic modulation of reinforcement learning predicts reward—oriented behavior in daily life

Much human behavior is driven by rewards. Preclinical neurophysiological and clinical positron emission tomography (PET) studies have implicated striatal phasic dopamine (DA) release as a primary modulator of reward processing. However, the relationship between experimental reward-induced striatal DA release and responsiveness to naturalistic rewards, and therefore functional relevance of these findings, has been elusive.We therefore combined, for the first time, a DA D2/3 receptor [18F]fallypride PET during a probabilistic reinforcement learning (RL) task with a six day ecological momentary assessments (EMA) of reward-related behavior in the everyday life of 16 healthy volunteers. We detected significant reward-induced DA release in the bilateral putamen, caudate nucleus and ventral striatum, the extent of which was associated with better behavioral performance on the RL task across all regions. Furthermore, individual variability in the extent of reward-induced DA release in the right caudate nucleus and ventral striatum modulated the tendency to be actively engaged in a behavior if the active engagement was previously deemed enjoyable. This study suggests a link between striatal reward-related DA release and ecologically relevant reward-oriented behavior, suggesting an avenue for the inquiry into the DAergic basis of optimal and impaired motivational drive.

Neural mechanisms of reward processing associated with depression-related personality traits.

Although impaired reward processing in depression has been well-documented, the exact nature of that deficit remains poorly understood. To investigate the link between depression and the neural mechanisms of reward processing, we examined individual differences in personality. We recorded the electroencephalogram from healthy college students engaged in a probabilistic reinforcement learning task. Participants also completed several personality questionnaires that assessed traits related to reward sensitivity, motivation, and depression. We examined whether behavioral measures of reward learning and event-related potential components related to outcome processing and reward anticipation-namely, the cue and feedback-related reward positivity (RewP) and the stimulus preceding negativity (SPN)-would link these personality traits to depression. Participants who scored high in reward sensitivity produced a relatively larger feedback-RewP. By contrast, participants who scored high in depression learned the contingencies for infrequently rewarded cue-response combinations relatively poorly, exhibited a larger SPN, and produced a smaller feedback-RewP, especially to outcomes following cue-response combinations that were frequently rewarded. These results point to a primary deficit in reward valuation in individuals who score high in depression, with secondary consequences that impact reward learning and anticipation. Despite recent evidence arguing for an anticipatory deficit in depression, impaired reward valuation as a primary deficit should be further examined in clinical samples.

Reversal learning reveals cognitive deficits and altered prediction error encoding in the ventral striatum in Huntington's disease.

Huntington's disease (HD) is an autosomal dominant neurodegenerative condition characterized by a triad of movement disorder, neuropsychiatric symptoms and cognitive deficits. The striatum is particularly vulnerable to the effects of mutant huntingtin, and cell loss can already be found in presymptomatic stages. Since the striatum is well known for its role in reinforcement learning, we hypothesized to find altered behavioral and neural responses in HD patients in a probabilistic reinforcement learning task performed during functional magnetic resonance imaging. We studied 24 HD patients without central nervous system (CNS)-active medication and 25 healthy controls. Twenty HD patients and 24 healthy controls were able to complete the task. Computational modeling was used to calculate prediction error values and estimate individual parameters. We observed that gray matter density and prediction error signals during the learning task were related to disease stage. HD patients in advanced disease stages appear to use a less complex strategy in the reversal learning task. In contrast, HD patients in early disease stages show intact encoding of learning signals in the degenerating left ventral striatum. This effect appears to be lost with disease progression.

An Upside to Reward Sensitivity: The Hippocampus Supports Enhanced Reinforcement Learning in Adolescence

Adolescents are notorious for engaging in reward-seeking behaviors, a tendency attributed to heightened activity in the brain's reward systems during adolescence. It has been suggested that reward sensitivity in adolescence might be adaptive, but evidence of an adaptive role has been scarce. Using a probabilistic reinforcement learning task combined with reinforcement learning models and fMRI, we found that adolescents showed better reinforcement learning and a stronger link between reinforcement learning and episodic memory for rewarding outcomes. This behavioral benefit was related to heightened prediction error-related BOLD activity in the hippocampus and to stronger functional connectivity between the hippocampus and the striatum at the time of reinforcement. These findings reveal an important role for the hippocampus in reinforcement learning in adolescence and suggest that reward sensitivity in adolescence is related to adaptive differences in how adolescents learn from experience.

Memory-reliant Post-error Slowing Is Associated with Successful Learning and Fronto-occipital Activity.

Negative feedback after an action in a cognitive task can lead to devaluing that action on future trials as well as to more cautious responding when encountering that same choice again. These phenomena have been explored in the past by reinforcement learning theories and cognitive control accounts, respectively. Yet, how cognitive control interacts with value updating to give rise to adequate adaptations under uncertainty is less clear. In this fMRI study, we investigated cognitive control-based behavioral adjustments during a probabilistic reinforcement learning task and studied their influence on performance in a later test phase in which the learned value of items is tested. We provide support for the idea that functionally relevant and memory-reliant behavioral adjustments in the form of post-error slowing during reinforcement learning are associated with test performance. Adjusting response speed after negative feedback was correlated with BOLD activity in right inferior frontal gyrus and bilateral middle occipital cortex during the event of receiving the feedback. Bilateral middle occipital cortex activity overlapped partly with activity reflecting feedback deviance from expectations as measured by unsigned prediction error. These results suggest that cognitive control and feature processing cortical regions interact to implement feedback-congruent adaptations beneficial to learning.

A competence-performance based model to develop a syntactic language for artificial agents

The hypothesis of language use is an attractive theory in order to explain how natural languages evolve and develop in social populations. In this paper we present a model partially based on the idea of language games, so that a group of artificial agents are able to produce and share a symbolic language with syntactic structure. Grammatical structure is induced by grammatical evolution of stochastic regular grammars with learning capabilities, while language development is refined by means of language games where the agents apply on-line probabilistic reinforcement learning. Within this framework, the model adapts the concepts of competence and performance in language, as they have been proposed in some linguistic theories. The first experiments in this article have been organized around the linguistic description of visual scenes with the possibility of changing the referential situations. A second and more complicated experimental setting is also analyzed, where linguistic descriptions are enforced to keep word order constraints.

Adaptive behaviour and feedback processing integrate experience and instruction in reinforcement learning

In any non-deterministic environment, unexpected events can indicate true changes in the world (and require behavioural adaptation) or reflect chance occurrence (and must be discounted). Adaptive behaviour requires distinguishing these possibilities. We investigated how humans achieve this by integrating high-level information from instruction and experience. In a series of EEG experiments, instructions modulated the perceived informativeness of feedback: Participants performed a novel probabilistic reinforcement learning task, receiving instructions about reliability of feedback or volatility of the environment. Importantly, our designs de-confound informativeness from surprise, which typically co-vary. Behavioural results indicate that participants used instructions to adapt their behaviour faster to changes in the environment when instructions indicated that negative feedback was more informative, even if it was simultaneously less surprising. This study is the first to show that neural markers of feedback anticipation (stimulus-preceding negativity) and of feedback processing (feedback-related negativity; FRN) reflect informativeness of unexpected feedback. Meanwhile, changes in P3 amplitude indicated imminent adjustments in behaviour. Collectively, our findings provide new evidence that high-level information interacts with experience-driven learning in a flexible manner, enabling human learners to make informed decisions about whether to persevere or explore new options, a pivotal ability in our complex environment.

Probabilistic Reinforcement Learning in Patients With Schizophrenia: Relationships to Anhedonia and Avolition

Anhedonia (a reduced experience of pleasure) and avolition (a reduction in goal-directed activity) are common features of schizophrenia that have substantial effects on functional outcome, but are poorly understood and treated. Here, we examined whether alterations in reinforcement learning may contribute to these symptoms in schizophrenia by impairing the translation of reward information into goal-directed action. 38 stable outpatients with schizophrenia or schizoaffective disorder and 37 healthy controls underwent fMRI during a probabilistic stimulus selection reinforcement learning task with dissociated choice- and feedback-related activation, followed by a behavioral transfer task allowing separate assessment of learning from positive versus negative outcomes. A Q-learning algorithm was used to examine functional activation relating to prediction error at the time of feedback and to expected value at the time of choice. Behavioral results suggested a reduction in learning from positive feedback in patients; however, this reduction was unrelated to anhedonia/avolition severity. On fMRI analysis, prediction error-related activation at the time of feedback was highly similar between patients and controls. During early learning, patients activated regions in the cognitive control network to a lesser extent than controls. Correlation analyses revealed reduced responses to positive feedback in dorsolateral prefrontal cortex and caudate among those patients higher in anhedonia/avolition. Together, these results suggest that anhedonia/avolition are as strongly related to cortical learning or higher-level processes involved in goal-directed behavior such as effort computation and planning as to striatally mediated learning mechanisms.

Mild Reinforcement Learning Deficits in Patients With First-Episode Psychosis.

Numerous studies have identified reinforcement learning (RL) deficits in schizophrenia. Most have focused on chronic patients with longstanding antipsychotic treatment, however, and studies of RL in early-illness patients have produced mixed results, particularly regarding gradual/procedural learning. No study has directly contrasted both rapid and gradual RL in first-episode psychosis (FEP) samples. We examined probabilistic RL in 34 FEP patients and 36 controls, using Go/NoGo (GNG) and Gain vs Loss-Avoidance (GLA) paradigms. Our results were mixed, with FEP patients exhibiting greater impairment in the ability to use positive, as opposed to negative, feedback to drive rapid RL on the GLA, but not the GNG. By contrast, patients and controls showed similar improvement across the acquisition. Finally, we found no significant between-group differences in the postacquisition expression of value-based preference in both tasks. Negative symptoms were modestly associated with RL measures, while the overall bias to engage in Go-responding correlated significantly with psychosis severity in FEP patients, consistent with striatal hyperdopaminergia. Taken together, FEP patients demonstrated more circumscribed RL impairments than previous studies have documented in chronic samples, possibly reflecting differential symptom profiles between first-episode and chronic samples. Our finding of relatively preserved gradual/procedural RL, in briefly medicated FEP patients, might suggest spared or restored basal ganglia function. Our findings of preserved abilities to use representations of expected value to guide decision making, and our mixed results regarding rapid RL, may reflect a lesser degree of prefrontal cortical functional impairment in FEP than in chronic samples. Further longitudinal research, in larger samples, is required.

Does Feedback-Related Brain Response during Reinforcement Learning Predict Socio-motivational (In-)dependence in Adolescence?

This multi-methodological study applied functional magnetic resonance imaging to investigate neural activation in a group of adolescent students (N = 88) during a probabilistic reinforcement learning task. We related patterns of emerging brain activity and individual learning rates to socio-motivational (in-)dependence manifested in four different motivation types (MTs): (1) peer-dependent MT, (2) teacher-dependent MT, (3) peer-and-teacher-dependent MT, (4) peer-and-teacher-independent MT. A multinomial regression analysis revealed that the individual learning rate predicts students’ membership to the independent MT, or the peer-and-teacher-dependent MT. Additionally, the striatum, a brain region associated with behavioral adaptation and flexibility, showed increased learning-related activation in students with motivational independence. Moreover, the prefrontal cortex, which is involved in behavioral control, was more active in students of the peer-and-teacher-dependent MT. Overall, this study offers new insights into the interplay of motivation and learning with (1) a focus on inter-individual differences in the role of peers and teachers as source of students’ individual motivation and (2) its potential neurobiological basis.

Multifaceted Contributions by Different Regions of the Orbitofrontal and Medial Prefrontal Cortex to Probabilistic Reversal Learning.

The ability to adjust behavior in response to changes involving uncertain or probabilistic reward contingencies is an essential survival skill that is impaired in a variety of psychiatric disorders. It is well established that different forms of cognitive flexibility are mediated by anatomically distinct regions of the frontal lobes when reinforcement contingencies are assured, however, less is known about the contribution of these regions to probabilistic reinforcement learning. Here we show that different regions of the orbitofrontal and medial prefrontal cortex make distinct contributions to probabilistic reversal learning. These findings provide novel information about the complex interplay between frontal lobe regions in mediating these processes and accordingly provide insight into possible pathophysiology that underlies impairments in cognitive flexibility observed in mental illnesses.

Feedback-related negativity is enhanced in adolescence during a gambling task with and without probabilistic reinforcement learning.

Feedback-related negativity (FRN) is a negative deflection that appears around 250 ms after the gain or loss of feedback to chosen alternatives in a gambling task in frontocentral regions following outcomes. Few studies have reported FRN enhancement in adolescents compared with adults in a gambling task without probabilistic reinforcement learning, despite the fact that learning from positive or negative consequences is crucial for decision-making during adolescence. Therefore, the aim of the present research was to identify differences in FRN amplitude and latency between adolescents and adults on a gambling task with favorable and unfavorable probabilistic reinforcement learning conditions, in addition to a nonlearning condition with monetary gains and losses. Higher rate scores of high-magnitude choices during the final 30 trials compared with the first 30 trials were observed during the favorable condition, whereas lower rates were observed during the unfavorable condition in both groups. Higher FRN amplitude in all conditions and longer latency in the nonlearning condition were observed in adolescents compared with adults and in relation to losses. Results indicate that both the adolescents and the adults improved their performance in relation to positive and negative feedback. However, the FRN findings suggest an increased sensitivity to external feedback to losses in adolescents compared with adults, irrespective of the presence or absence of probabilistic reinforcement learning. These results reflect processing differences on the neural monitoring system and provide new perspectives on the dynamic development of an adolescent's brain.

Feedback-driven trial-by-trial learning in autism spectrum disorders.

Impairments in learning are central to autism spectrum disorders. The authors investigated the cognitive and neural basis of these deficits in young adults with autism spectrum disorders using a well-characterized probabilistic reinforcement learning paradigm. The probabilistic selection task was implemented among matched participants with autism spectrum disorders (N=22) and with typical development (N=25), aged 18-40 years, using rapid event-related functional MRI. Participants were trained to choose the correct stimulus in high-probability (AB), medium-probability (CD), and low-probability (EF) pairs, presented with valid feedback 80%, 70%, and 60% of the time, respectively. Whole-brain voxel-wise and parametric modulator analyses examined early and late learning during the stimulus and feedback epochs of the task. The groups exhibited comparable performance on medium- and low-probability pairs. Typically developing persons showed higher accuracy on the high-probability pair, better win-stay performance (selection of the previously rewarded stimulus on the next trial of that type), and more robust recruitment of the anterior and medial prefrontal cortex during the stimulus epoch, suggesting development of an intact reward-based working memory for recent stimulus values. Throughout the feedback epoch, individuals with autism spectrum disorders exhibited greater recruitment of the anterior cingulate and orbito-frontal cortices compared with individuals with typical development, indicating continuing trial-by-trial activity related to feedback processing. Individuals with autism spectrum disorders exhibit learning deficits reflecting impaired ability to develop an effective reward-based working memory to guide stimulus selection. Instead, they continue to rely on trial-by-trial feedback processing to support learning dependent upon engagement of the anterior cingulate and orbito-frontal cortices.

Alignment in vision-based syntactic language games for teams of robots using stochastic regular grammars and reinforcement learning: The fully autonomous case and the human supervised case

This paper approaches the syntactic alignment of a robot team by means of dialogic language games and by applying online probabilistic reinforcement learning algorithms. The syntactic alignment is studied under two different configurations of the robot team: (a) when the team is formed exclusively by robots and (b) when a human is included in the team, in which case the human is endowed with a natural language to communicate with the other members of the team. For the two above mentioned cases, we are interested in the analysis of the convergence of the team to an optimal common language. The main contribution of the paper is the application of stochastic regular grammars, with learning capability, to generate the robots team’s language. Apart from the analysis of the convergence to a common language in the case of a fully autonomous robot team without human intervention we are also particularly interested in analyzing how the syntactic alignment of the robot team can be influenced or mediated by humans. The paper is organized as follows: first, we describe the syntactic language games, in particular the type of grammar and syntactic rules of the robots team’s language and the dynamic process of the language games which are based on dialogic communicative acts and a reinforcement learning policy that allows the robot team to converge to a common language. Afterwards, the experimental results are presented and discussed. The experimental work has been organized around the linguistic description of visual scenes of the blocks world type. The general conclusion of our experiments can be briefly stated in this way: “for the fully autonomous case (only robots) the final emergent grammar is arbitrary, while in the second case of including a human in the team the final emergent grammar is the one used by the human”.