Reward Learning Research Articles

Intelligent robotic sonographer: Mutual information-based disentangled reward learning from few demonstrations

Ultrasound (US) imaging is widely used for biometric measurement and diagnosis of internal organs due to the advantages of being real-time and radiation-free. However, due to inter-operator variations, resulting images highly depend on the experience of sonographers. This work proposes an intelligent robotic sonographer to autonomously “explore” target anatomies and navigate a US probe to standard planes by learning from the expert. The underlying high-level physiological knowledge from experts is inferred by a neural reward function, using a ranked pairwise image comparison approach in a self-supervised fashion. This process can be referred to as understanding the “language of sonography.” Considering the generalization capability to overcome inter-patient variations, mutual information is estimated by a network to explicitly disentangle the task-related and domain features in latent space. The robotic localization is carried out in coarse-to-fine mode based on the predicted reward associated with B-mode images. To validate the effectiveness of the proposed reward inference network, representative experiments were performed on vascular phantoms (“line” target), two types of ex vivo animal organ phantoms (chicken heart and lamb kidney representing “point” target), and in vivo human carotids. To further validate the performance of the autonomous acquisition framework, physical robotic acquisitions were performed on three phantoms (vascular, chicken heart, and lamb kidney). The results demonstrated that the proposed advanced framework can robustly work on a variety of seen and unseen phantoms as well as in vivo human carotid data. Code: https://github.com/yuan-12138/MI-GPSR. Video: https://youtu.be/u4ThAA9onE0.

Morphometric alterations of the mesocorticolimbic network in Parkinson's disease with impulse control disorders.

Impulse control disorders (ICDs) are a group of non-motor symptoms of Parkinson disease (PD) leading to significant psychosocial detrimental outcome. The mesocorticolimbic network plays a distinctive role in reward learning and executive decision making and has been suggested to be involved in ICDs in PD. To study morphometric changes of the mesocorticolimbic network in PD with ICD. A total of 18 patients of PD with ICD (PD + ICD), 19 patients of PD without ICD (PD - ICD) and 19 healthy controls (HC) were included in the study. ICDs were diagnosed using Questionnaire for Impulsive-Compulsive Disorders in PD-Rating Scale (QUIP-RS). MRI was done using a 3T scanner and assessment of cortical thickness and subcortical volumes were done using FreeSurfer. Brain regions known to be part of the mesocorticolimbic network were extracted and included for statistical analysis. There was no difference between PD + ICD and PD - ICD with regard to duration of illness or total dopaminergic medication. In comparison to HC, patients with PD + ICD demonstrated atrophy of the left frontal pole, and this atrophy neared significance in comparison to PD - ICD. The QUIP-RS had a negative correlation with left caudate volume in PD + ICD. The PD + ICD group showed distinct morphometric changes in regions involved in the mesocorticolimbic system which may contribute to the presence of ICD.

Memory consolidation in honey bees is enhanced by down-regulation of Down syndrome cell adhesion molecule and changes its alternative splicing.

Down syndrome cell adhesion molecule (Dscam) gene encodes a cell adhesion molecule required for neuronal wiring. A remarkable feature of arthropod Dscam is massive alternative splicing generating thousands of different isoforms from three variable clusters of alternative exons. Dscam expression and diversity arising from alternative splicing have been studied during development, but whether they exert functions in adult brains has not been determined. Here, using honey bees, we find that Dscam expression is critically linked to memory retention as reducing expression by RNAi enhances memory after reward learning in adult worker honey bees. Moreover, alternative splicing of Dscam is altered in all three variable clusters after learning. Since identical Dscam isoforms engage in homophilic interactions, these results suggest a mechanism to alter inclusion of variable exons during memory consolidation to modify neuronal connections for memory retention.

FedAVE: Adaptive data value evaluation framework for collaborative fairness in federated learning

Collaborative fairness in federated learning rewards high-contribution clients with high-performance models when multiple clients train a machine learning model cooperatively. Existing approaches ignore the information on data distribution when evaluating the clients’ data quality, resulting in a mismatch between the reward allocation and the real data quality of clients under different data heterogeneity settings. To address this problem, we propose a novel Federated learning framework with Adaptive data Value Evaluation mechanism (FedAVE) to ensure collaborative fairness without affecting the predictive performance of models. First, an adaptive reputation calculation module is designed to generate reputations that match the clients’ contributions based on the information of their data distribution, respectively. Second, a dynamic gradient reward distribution module is devised to allocate a certain number of aggregated model parameter updates/gradients as the rewards corresponding to the reputations and the data distribution information. Extensive experiments on three public benchmarks show that the proposed FedAVE outperforms all baseline methods in terms of fairness, and achieves comparable performance to the state-of-the-art methods in terms of accuracy. Code available at https://github.com/wangzihuixmu/FedAVE.

An adaptive intelligent routing algorithm based on deep reinforcement learning

A mobile ad hoc network (MANET) is a self-organizing network composed of multiple mobile nodes. These nodes achieve distributed communication and wireless resource sharing through autonomous collaboration. However, typical routing protocols for MANETs suffer from uneven network load, low network throughput, poor anti-destruction performance, and low transmission reliability. In order to meet the routing protocol characteristics of distributed adaptability, reliability and anti-destruction, balance of load and energy consumption and guarantee of high QoS in MANETs, this paper proposes an adaptive intelligent routing algorithm based on the PER-D3QN model of deep reinforcement learning. To achieve better routing strategy selection, the routing problem is first modeled, a distributed multi-agent reinforcement learning routing framework is designed, and a deep reinforcement learning reward function considering multiple factors is used. Next, the implementation process of the adaptive intelligent routing algorithm using the PER-D3QN model is described in detail. Finally, a mobile self-organizing network environment was constructed based on Networkx, and corresponding data was collected to simulate the algorithm. The simulation results show that the adaptive intelligent routing algorithm proposed in this paper can balance the energy consumption of the network and the load of nodes, its average remaining energy has increased by more than 10% and PL is reduced by at least 10% compared to other algorithms, while guaranteeing high QoS performance of routing, that is, reducing the average end-to-end delay of data packet transmission by over 22% compared to other algorithms and maintaining a high packet delivery ratio.

Longitudinal data and a semantic similarity reward for chest X-ray report generation

Radiologists face high burnout rates, partially due to the increasing volume of Chest X-rays (CXRs) requiring interpretation and reporting. Automated CXR report generation holds promise for reducing this burden and improving patient care. While current models show potential, their diagnostic accuracy is limited. Our proposed CXR report generator integrates elements of the radiologist workflow and introduces a novel reward for reinforcement learning. Our approach leverages longitudinal data from a patient’s prior CXR study and effectively handles cases where no prior study exists, thus mirroring the radiologist’s workflow. In contrast, existing models typically lack this flexibility, often requiring prior studies for the model to function optimally. Our approach also incorporates all CXRs from a patient’s study and distinguishes between report sections through section embeddings. Our reward for reinforcement learning leverages CXR-BERT, which forces our model to learn the clinical semantics of radiology reporting. We conduct experiments on publicly available datasets — MIMIC-CXR and Open-i IU X-ray — with metrics shown to more closely correlate with radiologists’ assessment of reporting. Results from our study demonstrate that the proposed model generates reports that are more aligned with radiologists’ reports than state-of-the-art models, such as those utilising large language models, reinforcement learning, and multi-task learning. The proposed model improves the diagnostic accuracy of CXR report generation, which could one day reduce radiologists’ workload and enhance patient care. Our Hugging Face checkpoint (https://huggingface.co/aehrc/cxrmate) and code (https://github.com/aehrc/cxrmate) are publicly available.

КУЛЬТУРА БЕЗПЕРЕРВНОГО НАВЧАННЯ ТА СТРАТЕГІЇ ЇЇ РОЗВИТКУ

In the conditions of rapid technological changes and intensified competition, the ability of organizations to quickly adapt and innovate becomes a critical success factor. The formation of a culture of continuous learning, which encourages the continuous development of employee competencies, knowledge sharing and readiness for change, changes the key in ensuring the adaptability and competitiveness of the role of organizations. The article examines the importance of a culture of continuous learning and its impact on the effectiveness of teams and organizations. The authors address the key elements of such a culture, including leadership that encourages open dialogue and experimentation, psychological safety, flexible work processes, investment in staff development, structures for expanding knowledge, and systems of recognition and reward for learning. The article provides practical recommendations for creating an environment conducive to continuous learning and talent development. which emphasizes the importance of leadership that demonstrates commitment to the values of learning, radical openness and respect for diversity of opinion. It also highlights the role of psychological safety in encouraging employees to share ideas, experiment and recognize mistakes as opportunities for growth. Empirical studies confirm the positive impact of learning culture on innovativeness, productivity and adaptability of teams and organizations. The authors present an example of a questionnaire for measuring the level of psychological safety and interpreting the results, which allows organizations to monitor progress and identify areas for improvement. A comprehensive strategy for the development of a culture of lifelong learning, which covers both structural and cultural aspects of the organization, is also proposed. At the end, the direction of further research is outlined, including the empirical validation of approaches in various fields, the study of the impact of digital technologies on educational processes, the analysis of the relationship between the culture of learning and indicators of organizational effectiveness, as well as strategies for adaptation to the specifics of different types of organizations.

Learning Individual Potential-Based Rewards in Multi-Agent Reinforcement Learning

Learning Individual Potential-Based Rewards in Multi-Agent Reinforcement Learning

Research on University Students' Perceptions of Effort and Rewards in Online Learning

Research on University Students' Perceptions of Effort and Rewards in Online Learning

Creating false rewarding memories guides novel decision making.

When memories of past rewarding experiences are distorted, are relevant decision-making preferences impacted? Although recent research has demonstrated the important role of episodic memory in value-based decision making, very few have examined the role of false memory in guiding novel decision making. The current study combined the pictorial Deese/Roediger-McDermott false memory paradigm with a reward learning task, where participants learned that items from some related lists gained reward and items from other lists led to no reward. Later, participants' memories and decision-making preferences were tested. With three experiments conducted in three countries, we successfully created false memories of rewarding experiences in which participants falsely remembered seeing a nonpresented lure picture bring them reward thereby confirming our constructive association hypothesis. Such false memories led participants to prefer the lure pictures and respond faster in a follow-up decision-making task, and the more false memories they formed, the higher preferences for the lure items they displayed (Experiment 2). Finally, results were replicated with or without a memory test before the decision-making task, showing that the impact of false memory on decision making was not cued by a memory test (Experiment 3). Our data suggest that the reconstructive nature of memory enables individuals to create new memory episodes to guide decision making in novel situations. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Impaired flexible reward learning in ADHD patients is associated with blunted reinforcement sensitivity and neural signals in ventral striatum and parietal cortex

Reward-based learning and decision-making are prime candidates to understand symptoms of attention deficit hyperactivity disorder (ADHD). However, only limited evidence is available regarding the neurocomputational underpinnings of the alterations seen in ADHD. This concerns flexible behavioral adaption in dynamically changing environments, which is challenging for individuals with ADHD. One previous study points to elevated choice switching in adolescent ADHD, which was accompanied by disrupted learning signals in medial prefrontal cortex.Here, we investigated young adults with ADHD (n = 17) as compared to age- and sex-matched controls (n = 17) using a probabilistic reversal learning experiment during functional magnetic resonance imaging (fMRI). The task requires continuous learning to guide flexible behavioral adaptation to changing reward contingencies. To disentangle the neurocomputational underpinnings of the behavioral data, we used reinforcement learning (RL) models, which informed the analysis of fMRI data.ADHD patients performed worse than controls particularly in trials before reversals, i.e., when reward contingencies were stable. This pattern resulted from ‘noisy’ choice switching regardless of previous feedback. RL modelling showed decreased reinforcement sensitivity and enhanced learning rates for negative feedback in ADHD patients. At the neural level, this was reflected in a diminished representation of choice probability in the left posterior parietal cortex in ADHD. Moreover, modelling showed a marginal reduction of learning about the unchosen option, which was paralleled by a marginal reduction in learning signals incorporating the unchosen option in the left ventral striatum.Taken together, we show that impaired flexible behavior in ADHD is due to excessive choice switching (‘hyper-flexibility’), which can be detrimental or beneficial depending on the learning environment. Computationally, this resulted from blunted sensitivity to reinforcement of which we detected neural correlates in the attention-control network, specifically in the parietal cortex. These neurocomputational findings remain preliminary due to the relatively small sample size.

The effect of reward and punishment on student learning outcome through student learning activity at SMA Terpadu Wira Bhakti Gorontalo, Indonesia

This research aims to analyze the effect of reward and punishment on student learning outcomes through student learning activity in the economic subject for social science classes at SMA Terpadu Wira Bhakti Gorontalo. It applied a survey-type quantitative method. Primary data were collected by distributing questionnaires to 94 students as respondents and statistically analyzed using SEMPLS. Results indicated reward significantly impacted student learning outcome, punishment significantly impacted student learning outcome, student learning activity significantly impacted student learning outcome, reward significantly impacted student learning outcome mediated by student learning activity, and punishment significantly impacted student learning outcome mediated by student learning activity in the economic subject for social science classes at SMA Terpadu Wira Bhakti Gorontalo. The goodness of fit showed a Q-Square of 0.501 or 50.1%. It explained reward and punishment impacted learning outcomes mediated by student learning activity by 50.1%, whereas the other 49.9% was explicated by other unexamined factors here.

Anticipating the main theme: A model for understanding prospective memory and reward learning in sonata-form listening

A number of studies have established a connection between musical anticipation and reward processing, yet the interrelated roles of the musical themes, their anticipatory cues, and the large-scale design of musical forms have not been adequately explored. To fill this gap, I present in this article a psychological model that focuses on listeners’ anticipation of musical themes while listening to familiar pieces written in sonata form. Active listening may engage prospective memory, which refers to the ability to recall an intended action in the future. While listening to a piece in sonata form, an intended action is to monitor the recapitulation of the main theme in the home key. The retransition preceding the recapitulation may prompt listeners to engage in anticipatory imagery of the imminent theme. I identify four types of musical cues present in retransitions: dissonance-related, attention-related, goal-related, and time-based. These cues can foster listeners’ attention allocation toward predicting the theme through sequence processing and subvocal humming along with the music. The confirmation of this prediction and the harmonic resolution at the beginning of the recapitulation can serve as positive reinforcers, strengthening the listening habit of anticipatory imagery of main themes. Drawing on recent brain-imaging studies, the proposed model suggests that sensitization to anticipatory musical cues and precise prediction/imagery of themes are essential for reward-based learning of music. Moreover, this model clarifies both the presence of themes in various musical forms and a critical attribute of effective musical themes, namely, their ease of memorization and imaginability.

Reward and Punishment Learning as Predictors of Cognitive Behavioral Therapy Response in Parkinson's Disease Comorbid with Clinical Depression.

Depression is highly comorbid among individuals with Parkinson's Disease (PD), who often experience unique challenges to accessing and benefitting from empirically supported interventions like Cognitive Behavioral Therapy (CBT). Given the role of reward processing in both depression and PD, this study analyzed a subset (N = 25) of participants who participated in a pilot telemedicine intervention of PD-informed CBT, and also completed a Reward- and Punishment-Learning Task (RPLT) at baseline. At the conclusion of CBT, participants were categorized into treatment responders (n = 14) and non-responders (n = 11). Responders learned more optimally from negative rather than positive feedback on the RPLT, while this pattern was reversed in non-responders. Computational modeling suggested group differences in learning rate to negative feedback may drive the observed differences. Overall, the results suggest that a within-subject bias for punishment-based learning might help to predict response to CBT intervention for depression in those with PD.Plain Language Summary Performance on a Computerized Task may predict which Parkinson's Disease Patients benefit from Cognitive Behavioral Treatment of Clinical DepressionWhy was the study done? Clinical depression regularly arises in individuals with Parkinson's Disease (PD) due to the neurobiological changes with the onset and progression of the disease as well as the unique psychosocial difficulties associated with living with a chronic condition. Nonetheless, psychiatric disorders among individuals with PD are often underdiagnosed and likewise undertreated for a variety of reasons. The results of our study have implications about how to improve the accuracy and specificity of mental health treatment recommendations in the future to maximize benefits for individuals with PD, who often face additional barriers to accessing quality mental health treatment.What did the researchers do? We explored whether performance on a computerized task called the Reward- and Punishment-Learning Task (RPLT) helped to predict response to Cognitive Behavioral Therapy (CBT) for depression better than other predictors identified in previous studies. Twenty-five individuals with PD and clinical depression that completed a 10-week telehealth CBT program were assessed for: Demographics (Age, gender, etc.); Clinical information (PD duration, mental health diagnoses, levels of anxiety/depression, etc.); Neurocognitive performance (Memory, processing speed, impulse control, etc.); and RPLT performance.What did the researchers find? A total of 14 participants significantly benefitted from CBT treatment while 11 did not significantly benefit from treatment.There were no differences before treatment in the demographics, clinical information, and neurocognitive performance of those participants who ended up benefitting from the treatment versus those who did not.There were, however, differences before treatment in RPLT performance so that those individuals that benefitted from CBT seemed to learn better from negative feedback.What do the findings mean? Our results suggest that the CBT program benefitted those PD patients with clinical depression that seemed to overall learn best from avoiding punishment rather than obtaining reward which was targeted in CBT by focusing on increasing engagement in rewarding activities. The Reward- and Punishment-Learning Task hence may be a useful tool to help predict treatment response and provide more individualized recommendations on how to best maximize the benefits of psychotherapy for individuals with PD that may struggle to connect to mental health care. Caution is recommended about interpretating these results beyond this study as the overall number of participants was small and the data for this study were collected as part of a previous study so there was no opportunity to include additional measurements of interest.

Eye understand: Physiological measures as novel predictors of adaptive learning in horses

Striatal dopamine is a neurotransmitter that marks reward and mediates reward learning. Spontaneous Eye Blink Rate (SEBR) reflects striatal dopamine activity and could offer a novel, proxy measure of learning performance. Additionally, arousal affects performance in a range of cognitive tasks, but there is less evidence for the relationship between baseline arousal and learning. This study investigated whether SEBR predicts learning performance in an equine model. Further, it investigated novel physiological predictors of learning performance (eye temperature and heart rate variability). Nineteen horses completed a two-choice discrimination reversal learning (RL) task. Performance was measured using a Performance Index. SEBR and Heart Rate Variability (HRV) were measured at rest (home stable) and during RL. Infrared Thermography of eye temperature (IRT) was measured immediately before and after RL. SEBR did not change in response to RL despite being thought to reflect striatal dopamine activity. HRV was higher during RL than at rest though was not statistically significant (p = 0.06). Additionally, eye temperature decreased significantly during trials (left eye: p = 0.002; right eye: p = 0.05). These results indicate lower arousal in response to training, possibly similar to a ‘flow state’ in humans. Results of a negative binomial GLM revealed that SEBR was not associated with performance, however, positive predictors of learning performance included resting HRV (p = 0.009), HRV during the task (p = 0.002), and left eye temperature change (p < 0.0001). The association between left eye temperature and learning performance is consistent with lateralised blood flow to the left-brain hemisphere (responsible for learning targeted responses) and thought to be the first observation of this phenomenon during learning. Lower arousal may facilitate left-hemisphere dominance, creating the cognitive space to respond adaptively in RL. Left hemispheric dominance controls parasympathetic activity, potentially explaining the interactions observed here. Results suggest arousal both in the home and training environment may have important impacts on learning.

Supplemental Material for Reward Learning Capacity in a Community Sample of Individuals Who Use Cannabis

Supplemental Material for Reward Learning Capacity in a Community Sample of Individuals Who Use Cannabis

Value-Based Search Efficiency Is Encoded in the Substantia Nigra Reticulata Firing Rate, Spiking Irregularity and Local Field Potential.

Recent results show that valuable objects can pop out in visual search, yet its neural mechanisms remain unexplored. Given the role of substantia nigra reticulata (SNr) in object value memory and control of gaze, we recorded its single-unit activity while male macaque monkeys engaged in efficient or inefficient search for a valuable target object among low-value objects. The results showed that efficient search was concurrent with stronger inhibition and higher spiking irregularity in the target-present (TP) compared with the target-absent (TA) trials in SNr. Importantly, the firing rate differentiation of TP and TA trials happened within ∼100 ms of display onset, and its magnitude was significantly correlated with the search times and slopes (search efficiency). Time-frequency analyses of local field potential (LFP) after display onset revealed significant modulations of the gamma band power with search efficiency. The greater reduction of SNr firing in TP trials in efficient search can create a stronger disinhibition of downstream superior colliculus, which in turn can facilitate saccade to obtain valuable targets in competitive environments.

Interpretable Imitation Learning with Symbolic Rewards

Sample inefficiency of deep reinforcement learning methods is a major obstacle for their use in real-world tasks as they naturally feature sparse rewards. In fact, this from-scratch approach is often impractical in environments where extreme negative outcomes are possible. Recent advances in imitation learning have improved sample efficiency by leveraging expert demonstrations. Most work along this line of research employs neural network-based approaches to recover an expert cost function. However, the complexity and lack of transparency make neural networks difficult to trust and deploy in the real world. In contrast, we present a method for extracting interpretable symbolic reward functions from expert data, which offers several advantages. First, the learned reward function can be parsed by a human to understand, verify and predict the behavior of the agent. Second, the reward function can be improved and modified by an expert. Finally, the structure of the reward function can be leveraged to extract explanations that encode richer domain knowledge than standard scalar rewards. To this end, we use an autoregressive recurrent neural network that generates hierarchical symbolic rewards represented by simple symbolic trees. The recurrent neural network is trained via risk-seeking policy gradients. We test our method in MuJoCo environments as well as a chemical plant simulator. We show that the discovered rewards can significantly accelerate the training process and achieve similar or better performance than neural network-based algorithms.

The Role of the Cerebellum in Learning to Predict Reward: Evidence from Cerebellar Ataxia.

Recent findings in animals have challenged the traditional view of the cerebellum solely as the site of motor control, suggesting that the cerebellum may also be important for learning to predict reward from trial-and-error feedback. Yet, evidence for the role of the cerebellum in reward learning in humans is lacking. Moreover, open questions remain about which specific aspects of reward learning the cerebellum may contribute to. Here we address this gap through an investigation of multiple forms of reward learning in individuals with cerebellum dysfunction, represented by cerebellar ataxia cases. Nineteen participants with cerebellar ataxia and 57 age- and sex-matched healthy controls completed two separate tasks that required learning about reward contingencies from trial-and-error. To probe the selectivity of reward learning processes, the tasks differed in their underlying structure: while one task measured incremental reward learning ability alone, the other allowed participants to use an alternative learning strategy based on episodic memory alongside incremental reward learning. We found that individuals with cerebellar ataxia were profoundly impaired at reward learning from trial-and-error feedback on both tasks, but retained the ability to learn to predict reward based on episodic memory. These findings provide evidence from humans for a specific and necessary role for the cerebellum in incremental learning of reward associations based on reinforcement. More broadly, the findings suggest that alongside its role in motor learning, the cerebellum likely operates in concert with the basal ganglia to support reinforcement learning from reward.

Transdiagnostic inflexible learning dynamics explain deficits in depression and schizophrenia.

Deficits in reward learning are core symptoms across many mental disorders. Recent work suggests that such learning impairments arise by a diminished ability to use reward history to guide behaviour, but the neuro-computational mechanisms through which these impairments emerge remain unclear. Moreover, limited work has taken a transdiagnostic approach to investigate whether the psychological and neural mechanisms that give rise to learning deficits are shared across forms of psychopathology. To provide insight into this issue, we explored probabilistic reward learning in patients diagnosed with major depressive disorder (n = 33) or schizophrenia (n = 24) and 33 matched healthy controls by combining computational modelling and single-trial EEG regression. In our task, participants had to integrate the reward history of a stimulus to decide whether it is worthwhile to gamble on it. Adaptive learning in this task is achieved through dynamic learning rates that are maximal on the first encounters with a given stimulus and decay with increasing stimulus repetitions. Hence, over the course of learning, choice preferences would ideally stabilize and be less susceptible to misleading information. We show evidence of reduced learning dynamics, whereby both patient groups demonstrated hypersensitive learning (i.e. less decaying learning rates), rendering their choices more susceptible to misleading feedback. Moreover, there was a schizophrenia-specific approach bias and a depression-specific heightened sensitivity to disconfirmational feedback (factual losses and counterfactual wins). The inflexible learning in both patient groups was accompanied by altered neural processing, including no tracking of expected values in either patient group. Taken together, our results thus provide evidence that reduced trial-by-trial learning dynamics reflect a convergent deficit across depression and schizophrenia. Moreover, we identified disorder distinct learning deficits.