Credit Assignment Research Articles

A neural mechanism for learning from delayed postingestive feedback.

Animals learn the value of foods based on their postingestive effects and thereby develop aversions to foods that are toxic1-6 and preferences to those that are nutritious7-14. However, it remains unclear how the brain is able to assign credit to flavors experienced during a meal with postingestive feedback signals that can arise after a substantial delay. Here, we reveal an unexpected role for postingestive reactivation of neural flavor representations in this temporal credit assignment process. To begin, we leverage the fact that mice learn to associate novel15-18, but not familiar, flavors with delayed gastric malaise signals to investigate how the brain represents flavors that support aversive postingestive learning. Surveying cellular resolution brainwide activation patterns reveals that a network of amygdala regions is unique in being preferentially activated by novel flavors across every stage of the learning process: the initial meal, delayed malaise, and memory retrieval. By combining high-density recordings in the amygdala with optogenetic stimulation of genetically defined hindbrain malaise cells, we find that postingestive malaise signals potently and specifically reactivate amygdalar novel flavor representations from a recent meal. The degree of malaise-driven reactivation of individual neurons predicts strengthening of flavor responses upon memory retrieval, leading to stabilization of the population-level representation of the recently consumed flavor. In contrast, meals without postingestive consequences degrade neural flavor representations as flavors become familiar and safe. Thus, our findings demonstrate that interoceptive reactivation of amygdalar flavor representations provides a neural mechanism to resolve the temporal credit assignment problem inherent to postingestive learning.

Neural mechanisms of credit assignment for delayed outcomes during contingent learning.

Adaptive behavior in complex environments critically relies on the ability to appropriately link specific choices or actions to their outcomes. However, the neural mechanisms that support the ability to credit only those past choices believed to have caused the observed outcomes remain unclear. Here, we leverage multivariate pattern analyses of functional magnetic resonance imaging (fMRI) data and an adaptive learning task to shed light on the underlying neural mechanisms of such specific credit assignment. We find that the lateral orbitofrontal cortex (lOFC) and hippocampus (HC) code for the causal choice identity when credit needs to be assigned for choices that are separated from outcomes by a long delay, even when this delayed transition is punctuated by interim decisions. Further, we show when interim decisions must be made, learning is additionally supported by lateral frontopolar cortex (FPl). Our results indicate that FPl holds previous causal choices in a "pending" state until a relevant outcome is observed, and the fidelity of these representations predicts the fidelity of subsequent causal choice representations in lOFC and HC during credit assignment. Together, these results highlight the importance of the timely reinstatement of specific causes in lOFC and HC in learning choice-outcome relationships when delays and choices intervene, a critical component of real-world learning and decision making.

Attention-Based Deep Spiking Neural Networks for Temporal Credit Assignment Problems.

The temporal credit assignment (TCA) problem, which aims to detect predictive features hidden in distracting background streams, remains a core challenge in biological and machine learning. Aggregate-label (AL) learning is proposed by researchers to resolve this problem by matching spikes with delayed feedback. However, the existing AL learning algorithms only consider the information of a single timestep, which is inconsistent with the real situation. Meanwhile, there is no quantitative evaluation method for TCA problems. To address these limitations, we propose a novel attention-based TCA (ATCA) algorithm and a minimum editing distance (MED)-based quantitative evaluation method. Specifically, we define a loss function based on the attention mechanism to deal with the information contained within the spike clusters and use MED to evaluate the similarity between the spike train and the target clue flow. Experimental results on musical instrument recognition (MedleyDB), speech recognition (TIDIGITS), and gesture recognition (DVS128-Gesture) show that the ATCA algorithm can reach the state-of-the-art (SOTA) level compared with other AL learning algorithms.

Online Surveillance of IoT Agents in Smart Cities Using Deep Reinforcement Learning

In the context of today's smart cities, the effective operation of online surveillance of IoT agents is crucial for maintaining public safety and security. To achieve this, collaboration and cooperation among these autonomous IoT agents are indispensable. While the existing research has focused on collaboration amongst the neighboring agents or implicit cooperation, real-world scenarios often necessitate broader forms of collaboration. In response to this need, we introduce a novel framework that leverages visual signals and observations to facilitate collaboration among online surveillance. Our proposed framework incorporates the Multi-Agent POsthumous Credit Assignment (MA-POCA) algorithm as a training mechanism. The empirical results demonstrate that our framework consistently outperforms the base model in various performance metrics. Specifically, it exhibits superior performance in group cumulative reward, cumulative reward, and episode length. Furthermore, our proposed model excels in policy loss performance measures when compared to base model.

Understanding mechanisms of generalization following locomotor adaptation

Our nervous system has the remarkable ability to adapt our gait to accommodate changes in our body or surroundings. However, our adapted walking patterns often generalize only partially (or not at all) between different contexts. Here, we sought to understand how the nervous system generalizes adapted gait patterns from one context to another. Through a series of split-belt treadmill walking experiments, we evaluated different mechanistic hypotheses to explain the partial generalization of adapted gait patterns from split-belt treadmill to overground walking. In support of the credit assignment hypothesis, our experiments revealed the central finding that adaptation involves recalibration of two distinct forward models. Recalibration of the first model generalizes to overground walking, suggesting that the model represents the general movement dynamics of our body. On the other hand, recalibration of the second model does not generalize to overground walking, suggesting the model represents dynamics specific to treadmill walking. These findings reveal that there is a predefined portion of forward model recalibration that generalizes across context, leading to overall partial generalization of walking adaptation.

Towards reconciling usability and usefulness of policy explanations for sequential decision-making systems.

Safefy-critical domains often employ autonomous agents which follow a sequential decision-making setup, whereby the agent follows a policy to dictate the appropriate action at each step. AI-practitioners often employ reinforcement learning algorithms to allow an agent to find the best policy. However, sequential systems often lack clear and immediate signs of wrong actions, with consequences visible only in hindsight, making it difficult to humans to understand system failure. In reinforcement learning, this is referred to as the credit assignment problem. To effectively collaborate with an autonomous system, particularly in a safety-critical setting, explanations should enable a user to better understand the policy of the agent and predict system behavior so that users are cognizant of potential failures and these failures can be diagnosed and mitigated. However, humans are diverse and have innate biases or preferences which may enhance or impair the utility of a policy explanation of a sequential agent. Therefore, in this paper, we designed and conducted human-subjects experiment to identify the factors which influence the perceived usability with the objective usefulness of policy explanations for reinforcement learning agents in a sequential setting. Our study had two factors: the modality of policy explanation shown to the user (Tree, Text, Modified Text, and Programs) and the "first impression" of the agent, i.e., whether the user saw the agent succeed or fail in the introductory calibration video. Our findings characterize a preference-performance tradeoff wherein participants perceived language-based policy explanations to be significantly more useable; however, participants were better able to objectively predict the agent's behavior when provided an explanation in the form of a decision tree. Our results demonstrate that user-specific factors, such as computer science experience (p 0.05), and situational factors, such as watching agent crash (p 0.05), can significantly impact the perception and usefulness of the explanation. This research provides key insights to alleviate prevalent issues regarding innapropriate compliance and reliance, which are exponentially more detrimental in safety-critical settings, providing a path forward for XAI developers for future work on policy-explanations.

The challenge of learning adaptive mental behavior.

Many psychotherapies aim to help people replace maladaptive mental behaviors (such as those leading to unproductive worry) with more adaptive ones (such as those leading to active problem solving). Yet, little is known empirically about how challenging it is to learn adaptive mental behaviors. Mental behaviors entail taking mental operations and thus may be more challenging to perform than motor actions; this challenge may enhance or impair learning. In particular, challenge when learning is often desirable because it improves retention. Yet, it is also plausible that the necessity of carrying out mental operations interferes with learning the expected values of mental actions by impeding credit assignment: the process of updating an action's value after reinforcement. Then, it may be more challenging not only to perform-but also to learn the consequences of-mental (vs. motor) behaviors. We designed a task to assess learning to take adaptive mental versus motor actions via matched probabilistic feedback. In two experiments (N = 300), most participants found it more difficult to learn to select optimal mental (vs. motor) actions, as evident in worse accuracy not only in a learning but also test (retention) phase. Computational modeling traced this impairment to an indicator of worse credit assignment (impaired construction and maintenance of expected values) when learning mental actions, accounting for worse accuracy in the learning and retention phases. The results suggest that people have particular difficulty learning adaptive mental behavior and pave the way for novel interventions to scaffold credit assignment and promote adaptive thinking. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

On the emission-path dependency of the efficiency of ocean alkalinity enhancement

Ocean alkalinity enhancement (OAE) deliberately modifies the chemistry of the surface ocean to enhance the uptake of atmospheric CO2. The chemical efficiency of OAE (the amount of CO2 sequestered per unit of alkalinity added) depends, among other factors, on the background state of the surface ocean, which will significantly change until the end of this century and beyond. Here, we investigate the consequences of such changes for the long-term efficiency of OAE. We show, using idealized and scenario simulations with an Earth system model, that under doubling (quadrupling) of pre-industrial atmospheric CO2 concentrations, the simulated mean efficiency of OAE increases by about 18% (29%) from 0.76 to 0.90 (0.98). We find that only half of this effect can be explained by changes in the sensitivity of CO2 sequestration to alkalinity addition itself. The remainder is due to the larger portion of anthropogenic emissions taken up by a high-alkalinity ocean. Importantly, both effects are reversed if atmospheric CO2 concentrations were to decline due to large-scale deployment of land-based (or alternative ocean-based) carbon dioxide removal (CDR) methods. By considering an overshoot pathway that relies on large amounts of land-based CDR, we demonstrate that OAE efficiency indeed shows a strong decline after atmospheric CO2 concentrations have peaked. Our results suggest that the assumption of a constant, present-day chemical efficiency of OAE in integrated assessment modeling and carbon credit assignments could lead to economically inefficient OAE implementation pathways.

Anterior Cingulate Cortex Causally Supports Meta-Learning.

In dynamic environments with volatile rewards the anterior cingulate cortex (ACC) is believed to determine whether a visual object is relevant and should be chosen. The ACC may achieve this by integrating reward information over time to estimate which objects are worth to explore and which objects should be avoided. Such a higher-order meta-awareness about which objects should be explored predicts that the ACC causally contributes to choices when the reward values of objects are unknown and must be inferred from ongoing exploration. We tested this suggestion in nonhuman primates using a learning task that varied the number of object features that could be relevant, and by controlling the motivational value of choosing objects. During learning the ACC was transiently micro-stimulated when subjects foveated the to-be-chosen stimulus. We found that stimulation selectively impaired learning when feature uncertainty and motivational value of choices were high, which was linked to a deficit in using reward outcomes for feature-specific credit assignment. Application of an adaptive reinforcement learning model confirmed a primary deficit in weighting prediction errors that led to a meta-learning impairment to adaptively increase exploration during learning and to an impaired use of working memory to support learning. These findings provide causal evidence that the reward history traces in ACC are essential for meta-adjusting the exploration-exploitation balance and the strength of working memory of object values during adaptive behavior.

Attention-Based Intrinsic Reward Mixing Network for Credit Assignment in Multiagent Reinforcement Learning

Credit assignment is a critical problem in cooperative Multi-Agent Reinforcement Learning (MARL). To address this problem, current studies mainly rely on the intrinsic reward, which is directly summed with the global reward to generate a total reward. However, such kinds of intrinsic reward functions ignore the dependence among agents and inevitably limit the adaptivity and effectiveness of MARL methods. In this paper, we propose a novel method, Attention-based Intrinsic Reward Mixing Network (AIRMN), for credit assignment in MARL. Specifically, we design a new intrinsic reward network on the basis of the attention mechanism, in order to enhance the effectiveness of teamwork. Besides, we devise a new mixing network that combines the intrinsic and extrinsic rewards in a nonlinear and dynamic manner, so as to adapt the total reward to the variation of the environment. Experimental results on the battle games of StarCraft II demonstrate that AIRMN outperforms the state-of-the-art methods in terms of the average test win rate, and also validate that AIRMN can dynamically return the precise intrinsic reward to each agent based on their contributions to the team cooperation, thereby better dealing with the credit assignment problem.

Surprise-minimization as a solution to the structural credit assignment problem.

The structural credit assignment problem arises when the causal structure between actions and subsequent outcomes is hidden from direct observation. To solve this problem and enable goal-directed behavior, an agent has to infer structure and form a representation thereof. In the scope of this study, we investigate a possible solution in the human brain. We recorded behavioral and electrophysiological data from human participants in a novel variant of the bandit task, where multiple actions lead to multiple outcomes. Crucially, the mapping between actions and outcomes was hidden and not instructed to the participants. Human choice behavior revealed clear hallmarks of credit assignment and learning. Moreover, a computational model which formalizes action selection as the competition between multiple representations of the hidden structure was fit to account for participants data. Starting in a state of uncertainty about the correct representation, the central mechanism of this model is the arbitration of action control towards the representation which minimizes surprise about outcomes. Crucially, single-trial latent-variable analysis reveals that the neural patterns clearly support central quantitative predictions of this surprise minimization model. The results suggest that neural activity is not only related to reinforcement learning under correct as well as incorrect task representations but also reflects central mechanisms of credit assignment and behavioral arbitration.

Credit Assignment: Challenges and Opportunities in Developing Human-like Learning Agents

Temporal credit assignment is the process of distributing delayed outcomes to each action in a sequence, which is essential for learning to adapt and make decisions in dynamic environments. While computational methods in reinforcement learning, such as temporal difference (TD), have shown success in tackling this issue, it remains unclear whether these mechanisms accurately reflect how humans handle feedback delays. Furthermore, cognitive science research has not fully explored the credit assignment problem in humans and cognitive models. Our study uses a cognitive model based on Instance-Based Learning Theory (IBLT) to investigate various credit assignment mechanisms, including equal credit, exponential credit, and TD credit, using the IBL decision mechanism in a goal-seeking navigation task with feedback delays and varying levels of decision complexity. We compare the performance and process measures of the different models with human decision-making in two experiments. Our findings indicate that the human learning process cannot be fully explained by any of the mechanisms. We also observe that decision complexity affects human behavior but not model behavior. By examining the similarities and differences between human and model behavior, we summarize the challenges and opportunities for developing learning agents that emulate human decisions in dynamic environments.

Toward Collaborative Multitarget Search and Navigation with Attention‐Enhanced Local Observation

Collaborative multitarget search and navigation (CMTSN) is highly demanded in complex missions such as rescue and warehouse management. Traditional centralized and decentralized approaches fall short in terms of scalability and adaptability to real‐world complexities such as unknown targets and large‐scale missions. This article addresses this challenging CMTSN problem in three‐dimensional spaces, specifically for agents with local visual observation operating in obstacle‐rich environments. To overcome these challenges, this work presents the POsthumous Mix‐credit assignment with Attention (POMA) framework. POMA integrates adaptive curriculum learning and mixed individual‐group credit assignments to efficiently balance individual and group contributions in a sparse reward environment. It also leverages an attention mechanism to manage variable local observations, enhancing the framework's scalability. Extensive simulations demonstrate that POMA outperforms a variety of baseline methods. Furthermore, the trained model is deployed over a physical visual drone swarm, demonstrating the effectiveness and generalization of our approach in real‐world autonomous flight.

Therapeutic doses of ketamine acutely attenuate the aversive effect of losses during decision-making.

The discovery of rapid-acting antidepressant, ketamine has opened a pathway to a new generation of treatments for depression, and inspired neuroscientific investigation based on a new perspective that non-adaptive changes in the intrinsic excitatory and inhibitory circuitry might underlie the pathophysiology of depression. Nevertheless, it still remains largely unknown how the hypothesized molecular and synaptic levels of changes in the circuitry might mediate behavioral and neuropsychological changes underlying depression, and how ketamine might restore adaptive behavior. Here, we used computational models to analyze behavioral changes induced by therapeutic doses of ketamine, while rhesus macaques were iteratively making decisions based on gains and losses of tokens. When administered intramuscularly or intranasally, ketamine reduced the aversiveness of undesirable outcomes such as losses of tokens without significantly affecting the evaluation of gains, behavioral perseveration, motivation, and other cognitive aspects of learning such as temporal credit assignment and time scales of choice and outcome memory. Ketamine's potentially antidepressant effect was separable from other side effects such as fixation errors, which unlike outcome evaluation, was readily countered with strong motivation to avoid errors. We discuss how the acute effect of ketamine to reduce the initial impact of negative events could potentially mediate longer-term antidepressant effects through mitigating the cumulative effect of those events produced by slowly decaying memory, and how the disruption-resistant affective memory might pose challenges in treating depression. Our study also invites future investigations on ketamine's antidepressant action over diverse mood states and with affective events exerting their impacts at diverse time scales.

Therapeutic doses of ketamine acutely attenuate the aversive effect of losses during decision-making

The discovery of rapid-acting antidepressant, ketamine has opened a pathway to a new generation of treatments for depression, and inspired neuroscientific investigation based on a new perspective that non-adaptive changes in the intrinsic excitatory and inhibitory circuitry might underlie the pathophysiology of depression. Nevertheless, it still remains largely unknown how the hypothesized molecular and synaptic levels of changes in the circuitry might mediate behavioral and neuropsychological changes underlying depression, and how ketamine might restore adaptive behavior. Here, we used computational models to analyze behavioral changes induced by therapeutic doses of ketamine, while rhesus macaques were iteratively making decisions based on gains and losses of tokens. When administered intramuscularly or intranasally, ketamine reduced the aversiveness of undesirable outcomes such as losses of tokens without significantly affecting the evaluation of gains, behavioral perseveration, motivation, and other cognitive aspects of learning such as temporal credit assignment and time scales of choice and outcome memory. Ketamine’s potentially antidepressant effect was separable from other side effects such as fixation errors, which unlike outcome evaluation, was readily countered with strong motivation to avoid errors. We discuss how the acute effect of ketamine to reduce the initial impact of negative events could potentially mediate longer-term antidepressant effects through mitigating the cumulative effect of those events produced by slowly decaying memory, and how the disruption-resistant affective memory might pose challenges in treating depression. Our study also invites future investigations on ketamine’s antidepressant action over diverse mood states and with affective events exerting their impacts at diverse time scales.

Neuromorphic one-shot learning utilizing a phase-transition material

Design of hardware based on biological principles of neuronal computation and plasticity in the brain is a leading approach to realizing energy- and sample-efficient AI and learning machines. An important factor in selection of the hardware building blocks is the identification of candidate materials with physical properties suitable to emulate the large dynamic ranges and varied timescales of neuronal signaling. Previous work has shown that the all-or-none spiking behavior of neurons can be mimicked by threshold switches utilizing material phase transitions. Here, we demonstrate that devices based on a prototypical metal-insulator-transition material, vanadium dioxide (VO2), can be dynamically controlled to access a continuum of intermediate resistance states. Furthermore, the timescale of their intrinsic relaxation can be configured to match a range of biologically relevant timescales from milliseconds to seconds. We exploit these device properties to emulate three aspects of neuronal analog computation: fast (~1 ms) spiking in a neuronal soma compartment, slow (~100 ms) spiking in a dendritic compartment, and ultraslow (~1 s) biochemical signaling involved in temporal credit assignment for a recently discovered biological mechanism of one-shot learning. Simulations show that an artificial neural network using properties of VO2 devices to control an agent navigating a spatial environment can learn an efficient path to a reward in up to fourfold fewer trials than standard methods. The phase relaxations described in our study may be engineered in a variety of materials and can be controlled by thermal, electrical, or optical stimuli, suggesting further opportunities to emulate biological learning in neuromorphic hardware.

Meta predictive learning model of languages in neural circuits.

Large language models based on self-attention mechanisms have achieved astonishing performances, not only in natural language itself, but also in a variety of tasks of different nature. However, regarding processing language, our human brain may not operate using the same principle. Then, a debate is established on the connection between brain computation and artificial self-supervision adopted in large language models. One of most influential hypotheses in brain computation is the predictive coding framework, which proposes to minimize the prediction error by local learning. However, the role of predictive coding and the associated credit assignment in language processing remains unknown. Here, we propose a mean-field learning model within the predictive coding framework, assuming that the synaptic weight of each connection follows a spike and slab distribution, and only the distribution, rather than specific weights, is trained. This meta predictive learning is successfully validated on classifying handwritten digits where pixels are input to the network in sequence, and moreover, on the toy and real language corpus. Our model reveals that most of the connections become deterministic after learning, while the output connections have a higher level of variability. The performance of the resulting network ensemble changes continuously with data load, further improving with more training data, in analogy with the emergent behavior of large language models. Therefore, our model provides a starting point to investigate the connection among brain computation, next-token prediction, and general intelligence.

A Multiagent Cooperative Learning System With Evolution of Social Roles

Recent developments in reinforcement learning have been able to derive optimal policies for sophisticated and capable agents, and shown to achieve human-level performance on a number of challenging tasks. Unfortunately, when it comes to multi-agent systems (MASs), complexities such as non-stationarity and partial observability bring new challenges to the field. Building a flexible and efficient multi-agent reinforcement learning algorithm capable of handling complex tasks has to date remained an open challenge. This paper presents an Multi-Agent learning system with the evolution of Social Roles (eSRMA). The main interest is placed on solving the key issues in the definition and evolution of suitable roles, and optimizing the policies accompanied by social roles in MAS efficiently. Specifically, eSRMA incorporates and cultivates role division awareness of agents to improve the ability to deal with complex cooperative tasks. Each agent is assigned a role module, which can dynamically generate roles based on the individuals’ local observations. A novel multi-agent reinforcement learning algorithm (MARL) is designed as the principal driving force that governs the role-policy learning process by a role-attention credit assignment mechanism. Moreover, a role evolution process is developed to help agents dynamically choose appropriate roles in decision-making. Comprehensive experiments on the StarCraft II micromanagement benchmark have demonstrated that eSRMA exhibits superiority in achieving higher learning capability and efficiency for multiple agents compared to the state-of-the-art MARL methods.

Análisis exploratorio de la asignatura Trabajo Fin de Grado en las universidades españolas

Introduction and objectiveThe Bachelor Thesis (BT) is a compulsory subject in the Physiotherapy Degree. The objective of this work was to elaborate an exploratory analysis of the BT in the Spanish Universities 12years after its implementation. Material and methodsCross-sectional descriptive study using a structured search on the websites public and private Spanish universities offering Physiotherapy Degrees. The variables studied were number of credits, timing, type of work, teaching guide, language, distribution of hours, supervisor profile, evaluation system and defense format. Results57 degrees in 53 universities were analysed. The number of credits most frequently assigned was 6. Observed distribution of hours was face-to-face hours (17.5±19.5); hours of supervised work (18.8±21.3); hours of autonomous work (144.4±46.9). The most frequently used methodological design was the bibliographic review (n=24). In the evaluation criteria, the supervisor's rating ranged between 0 and 90% of the final mark (41.7±23.7) compared to 10-100% corresponding to the thesis evaluation panel (58.1±24.4). Only 21 degrees report the profile of the tutor teachers. ConclusionsThe assignment of credits, methodologies and evaluation systems are very heterogeneous among Spanish universities in the BT of the Physiotherapy Degree. After 15years since its implantation into the degree, it is necessary to establish a consensus to rationalize the subject.

Intrinsic Action Tendency Consistency for Cooperative Multi-Agent Reinforcement Learning

Efficient collaboration in the centralized training with decentralized execution (CTDE) paradigm remains a challenge in cooperative multi-agent systems. We identify divergent action tendencies among agents as a significant obstacle to CTDE's training efficiency, requiring a large number of training samples to achieve a unified consensus on agents' policies. This divergence stems from the lack of adequate team consensus-related guidance signals during credit assignment in CTDE. To address this, we propose Intrinsic Action Tendency Consistency, a novel approach for cooperative multi-agent reinforcement learning. It integrates intrinsic rewards, obtained through an action model, into a reward-additive CTDE (RA-CTDE) framework. We formulate an action model that enables surrounding agents to predict the central agent's action tendency. Leveraging these predictions, we compute a cooperative intrinsic reward that encourages agents to align their actions with their neighbors' predictions. We establish the equivalence between RA-CTDE and CTDE through theoretical analyses, demonstrating that CTDE's training process can be achieved using N individual targets. Building on this insight, we introduce a novel method to combine intrinsic rewards and RA-CTDE. Extensive experiments on challenging tasks in SMAC, MPE, and GRF benchmarks showcase the improved performance of our method.