Cognitive Reinforcement Learning Research Articles

Attention-Based Mechanisms for Cognitive Reinforcement Learning

In this paper, we propose a cognitive reinforcement learning method based on an attention mechanism (CRL-CBAM) to address the problems of complex interactive communication, limited range, and time-varying communication topology in multi-intelligence collaborative work. The method not only combines the efficient decision-making capability of reinforcement learning, the representational capability of deep learning, and the self-learning capability of cognitive learning but also inserts a convolutional block attention module to increase the representational capability by using the attention mechanism to focus on important features and suppress unnecessary ones. The use of two modules, channel and spatial axis, to emphasize meaningful features in the two main dimensions can effectively aid the flow of information in the network. Results from simulation experiments show that the method has more rewards and is more efficient than other methods in formation control, which means a greater advantage when dealing with scenarios with a large number of agents. In group containment, the agents learn to sacrifice individual rewards to maximize group rewards. All tasks are successfully completed, even if the simulation scenario changes from the training scenario. The method can therefore be applied to new environments with effectiveness and robustness.

Measurement of Striatal Dopamine Release Induced by Neuropsychological Stimulation in Positron Emission Tomography With Dual Injections of [11C]Raclopride.

ObjectivesPositron emission tomography (PET) with [11C]raclopride has been applied to measure changes in the concentration of endogenous dopamine induced by pharmacological challenge or neuropsychological stimulation by evaluating the binding potential (BP) between the baseline and activated state. Recently, to reliably estimate BP in the activated state, a new approach with dual-bolus injections in a single PET scan was developed. In this study, we investigated the feasibility of applying this dual-bolus injection approach to measure changes in endogenous dopamine levels induced by cognitive tasks in humans.MethodsFirst, the reproducibility of BP estimation using the dual-bolus injection approach was evaluated using PET scans without stimulation in nine healthy volunteers. A 90-min scan was performed with bolus injections of [11C]raclopride administered at the beginning of the scan and 45 min after the first injection. BPs in the striatum for the first injection (BP1) and second injection (BP2) were estimated using an extended simplified reference tissue model, and the mean absolute difference (MAD) between the two BPs was calculated. The MAD was also compared with the conventional bolus-plus-continuous infusion approach. Next, PET studies with a cognitive reinforcement learning task were performed on 10 healthy volunteers using the dual-bolus injection approach. The BP1 at baseline and BP2 at the activated state were estimated, and the reduction in BP was evaluated.ResultsIn the PET scans without stimulation, the dual-bolus injection approach showed a smaller MAD (<2%) between BP1 and BP2 than the bolus-plus-continuous infusion approach, demonstrating good reproducibility of this approach. In the PET scans with the cognitive task performance, the reduction in BP was not observed in the striatum by either approach, showing that the changes in dopamine level induced by the cognitive tasks performed in this study were not sufficient to be detected by PET.ConclusionOur results indicate that the cognitive task-induced changes in dopamine-related systems may be complex and difficult to measure accurately using PET scans. However, the proposed dual-bolus injection approach provided reliable BP estimates with high reproducibility, suggesting that it has the potential to improve the accuracy of PET scans for measuring changes in dopamine concentrations.

Null effects of levodopa on reward- and error-based motor adaptation, savings, and anterograde interference.

Dopamine signaling is thought to mediate reward-based learning. We tested for a role of dopamine in motor adaptation by administering the dopamine precursor levodopa to healthy participants in two experiments involving reaching movements. Levodopa has been shown to impair reward-based learning in cognitive tasks. Thus, we hypothesized that levodopa would selectively impair aspects of motor adaptation that depend on the reinforcement of rewarding actions. In the first experiment, participants performed two separate tasks in which adaptation was driven either by visual error-based feedback of the hand position or binary reward feedback. We used EEG to measure event-related potentials evoked by task feedback. We hypothesized that levodopa would specifically diminish adaptation and the neural responses to feedback in the reward learning task. However, levodopa did not affect motor adaptation in either task nor did it diminish event-related potentials elicited by reward outcomes. In the second experiment, participants learned to compensate for mechanical force field perturbations applied to the hand during reaching. Previous exposure to a particular force field can result in savings during subsequent adaptation to the same force field or interference during adaptation to an opposite force field. We hypothesized that levodopa would diminish savings and anterograde interference, as previous work suggests that these phenomena result from a reinforcement learning process. However, we found no reliable effects of levodopa. These results suggest that reward-based motor adaptation, savings, and interference may not depend on the same dopaminergic mechanisms that have been shown to be disrupted by levodopa during various cognitive tasks.NEW & NOTEWORTHY Motor adaptation relies on multiple processes including reinforcement of successful actions. Cognitive reinforcement learning is impaired by levodopa-induced disruption of dopamine function. We administered levodopa to healthy adults who participated in multiple motor adaptation tasks. We found no effects of levodopa on any component of motor adaptation. This suggests that motor adaptation may not depend on the same dopaminergic mechanisms as cognitive forms or reinforcement learning that have been shown to be impaired by levodopa.

A Role for Dopamine-Mediated Learning in the Pathophysiology and Treatment of Parkinson’s Disease

Dopamine contributes to corticostriatal plasticity and motor learning. Dopamine denervation profoundly alters motor performance, as in Parkinson's disease (PD); however, the extent to which these symptoms reflect impaired motor learning is unknown. Here, we demonstrate a D2 receptor blockade-induced aberrant learning that impedes future motor performance when dopamine signaling is restored, an effect diminished by coadministration of adenosine antagonists during blockade. We hypothesize that an inappropriate corticostriatal potentiation in striatopallidal cells of the indirect pathway underlies aberrant learning. We demonstrate synaptic potentiation in striatopallidal neurons induced by D2 blockade and diminished by application of an adenosine antagonist, consistent with behavioral observations. A neurocomputational model of the basal ganglia recapitulates the behavioral pattern and further links aberrant learning to plasticity in the indirect pathway. Thus, D2-mediated aberrant learning may contribute to motor deficits in PD, suggesting new avenues for the development of therapeutics.

Mechanisms Underlying Dopamine-Mediated Reward Bias in Compulsive Behaviors

Pathological behaviors such as problem gambling or shopping are characterized by compulsive choice despite alternative options and negative costs. Reinforcement learning algorithms allow a computation of prediction error, a comparison of actual and expected outcomes, which updates our predictions and influences our subsequent choices. Using a reinforcement learning model, we show data consistent with the idea that dopamine agonists in susceptible individuals with Parkinson's disease increase the rate of learning from gain outcomes. Dopamine agonists also increase striatal prediction error activity, thus signifying a "better than expected" outcome. Thus, our findings are consistent with a model whereby a distorted estimation of the gain cue underpins a choice bias toward gains.

Feedback-based versus observational classification learning in healthy aging and Parkinson's disease

Previous studies underline the role of dopamine in cognitive reinforcement learning. This has been demonstrated by a striatal involvement in feedback-based probabilistic classification learning. In order to determine to which extent the dopaminergic loss of Parkinson's disease and aging affects the feedback aspect in classification learning, we applied two versions of the same visual classification task. One version had to be learnt by trial-by-trial feedback, the other by observing the correct assignment of stimulus and category. Performance was evaluated in test blocks that were identical under the feedback and the observational conditions. There were 31 patients with Parkinson's disease (PD), 30 older controls and 20 younger controls tested. The results show that younger healthy participants perform better than older participants in the classification task and this difference significantly interacts with the learning condition: both groups show nearly the same level of performance under the observational condition but younger participants show a better performance than older ones under the feedback condition. In contrast, PD patients and older controls did not differ in their performance in the classification task; both groups performed better under the observational than under the feedback condition. These results demonstrate that healthy aging affects feedback-based learning but does not affect learning by observation. The fact that PD patients showed no additional deficit in feedback-based learning is an indication that the loss of dopamine does not play the key role under the feedback condition of our classification task. This finding questions the general role of the striatum in feedback-based learning and demonstrates that healthy aging particularly affects feedback-based learning.

By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism

To what extent do we learn from the positive versus negative outcomes of our decisions? The neuromodulator dopamine plays a key role in these reinforcement learning processes. Patients with Parkinson's disease, who have depleted dopamine in the basal ganglia, are impaired in tasks that require learning from trial and error. Here, we show, using two cognitive procedural learning tasks, that Parkinson's patients off medication are better at learning to avoid choices that lead to negative outcomes than they are at learning from positive outcomes. Dopamine medication reverses this bias, making patients more sensitive to positive than negative outcomes. This pattern was predicted by our biologically based computational model of basal ganglia-dopamine interactions in cognition, which has separate pathways for "Go" and "NoGo" responses that are differentially modulated by positive and negative reinforcement.