Credit Assignment Research Articles

Multilevel development of cognitive abilities in an artificial neural network

Several neuronal mechanisms have been proposed to account for the formation of cognitive abilities through postnatal interactions with the physical and sociocultural environment. Here, we introduce a three-level computational model of information processing and acquisition of cognitive abilities. We propose minimal architectural requirements to build these levels, and how the parameters affect their performance and relationships. The first sensorimotor level handles local nonconscious processing, here during a visual classification task. The second level or cognitive level globally integrates the information from multiple local processors via long-ranged connections and synthesizes it in a global, but still nonconscious, manner. The third and cognitively highest level handles the information globally and consciously. It is based on the global neuronal workspace (GNW) theory and is referred to as the conscious level. We use the trace and delay conditioning tasks to, respectively, challenge the second and third levels. Results first highlight the necessity of epigenesis through the selection and stabilization of synapses at both local and global scales to allow the network to solve the first two tasks. At the global scale, dopamine appears necessary to properly provide credit assignment despite the temporal delay between perception and reward. At the third level, the presence of interneurons becomes necessary to maintain a self-sustained representation within the GNW in the absence of sensory input. Finally, while balanced spontaneous intrinsic activity facilitates epigenesis at both local and global scales, the balanced excitatory/inhibitory ratio increases performance. We discuss the plausibility of the model in both neurodevelopmental and artificial intelligence terms.

Edge-Assisted Spectrum Sharing for Freshness-Aware Industrial Wireless Networks: A Learning-Based Approach

Information freshness is essential to industrial wireless networks (IWNs) and can be quantified by the age-of-information (AoI) metric. This paper addresses an AoI-aware spectrum sharing (AgeS) problem in IWNs, where multiple device-to-device (D2D) links opportunistically access the spectrum to satisfy their AoI constraints while maximizing primal links’ throughput. Particularly, we orchestrate the access of D2D links in a distributed manner. Since distributed scheduling results in incomplete observation, D2D links share the spectrum with uncertainty on the transmission environment. Therefore, we propose a distributed scheduling scheme, called D-age, to deal with the transmission uncertainty in the AgeS problem, where an adaptation of actor-critic method is adopted with AoI constraints tackled in the dual domain. To address the non-stationary environment and multi-agent credit assignment issue, cooperative multi-agent reinforcement learning (MARL) approach is developed, where multiple local actors are designed to guide D2D links to make real-time decisions via distributed scheduling policies, which are evaluated by an edge-assisted global critic with action-aware advantage functions. Integrated with graph attention networks (GATs), the critic selectively learns contextual information by assigning different importances to neighboring links, which enables the evaluation of scheduling policies in a scalable and computation-efficient manner. Theoretical guarantee of the time-averaged AoI constraints is provided and the effectiveness of D-age in terms of both AoI violation ratio and the capacity of primal links is demonstrated by simulation.

Two-level fusion of multi-sensor information for compressor blade crack detection based on self-attention mechanism

Nowadays, acoustic and vibration signals have been widely used to detect crack and its degree on compressor blades for non-destructive evaluation and structural health monitoring. Due to complex working conditions and interference of strong noise, single signal-based methods reach unsatisfactory accuracy. To improve the reliability and accuracy of crack detection, the two-level fusion method based on Mahalanobis distance (MD) and self-attention mechanism is proposed based on multi-acoustic, vibration, and acoustic emission information. First, the MD is calculated to measure the similarity between raw samples and group, which can fuse the same type of samples in data level. Then, the processed raw and data-level fusion samples are inputted to one-dimensional convolutional neural network, and the preliminary decisions are obtained. Finally, the decision-level fusion based on self-attention mechanism and credit assignment is proposed to modify and correct the preliminary decisions. The compressor experiments are implemented to test the proposed method under single and mixed working conditions with strong noise intensity. Further, the results illustrate that the proposed method can accurately detect crack for compressor blades. By comparing with other data fusion approaches, the advantage of the proposed method is validated under complex conditions.

Multi-Level Credit Assignment for Cooperative Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) has become more and more popular over recent decades, and the need for high-level cooperation is increasing every day because of the complexity of the real-world environment. However, the multi-agent credit assignment problem that serves as the main obstacle to high-level coordination is still not addressed properly. Though lots of methods have been proposed, none of them have thought to perform credit assignments across multi-levels. In this paper, we aim to propose an approach to learning a better credit assignment scheme by credit assignment across multi-levels. First, we propose a hierarchical model that consists of the manager level and the worker level. The manager level incorporates the dilated Gated Recurrent Unit (GRU) to focus on high-level plans and the worker level uses GRU to execute primitive actions conditioned on high-level plans. Then, one centralized critic is designed for each level to learn each level’s credit assignment scheme. To this end, we construct a novel hierarchical MARL algorithm, named MLCA, which can achieve multi-level credit assignment. We also conduct experiments on three classical and challenging tasks to demonstrate the performance of the proposed algorithm against three baseline methods. The results show that our method gains great performance improvement across all maps that require high-level cooperation.

Hindsight Network Credit Assignment: Efficient Credit Assignment in Networks of Discrete Stochastic Units

Training neural networks with discrete stochastic variables presents a unique challenge. Backpropagation is not directly applicable, nor are the reparameterization tricks used in networks with continuous stochastic variables. To address this challenge, we present Hindsight Network Credit Assignment (HNCA), a novel gradient estimation algorithm for networks of discrete stochastic units. HNCA works by assigning credit to each unit based on the degree to which its output influences its immediate children in the network. We prove that HNCA produces unbiased gradient estimates with reduced variance compared to the REINFORCE estimator, while the computational cost is similar to that of backpropagation. We first apply HNCA in a contextual bandit setting to optimize a reward function that is unknown to the agent. In this setting, we empirically demonstrate that HNCA significantly outperforms REINFORCE, indicating that the variance reduction implied by our theoretical analysis is significant and impactful. We then show how HNCA can be extended to optimize a more general function of the outputs of a network of stochastic units, where the function is known to the agent. We apply this extended version of HNCA to train a discrete variational auto-encoder and empirically show it compares favourably to other strong methods. We believe that the ideas underlying HNCA can help stimulate new ways of thinking about efficient credit assignment in stochastic compute graphs.

Locality Matters: A Scalable Value Decomposition Approach for Cooperative Multi-Agent Reinforcement Learning

Cooperative multi-agent reinforcement learning (MARL) faces significant scalability issues due to state and action spaces that are exponentially large in the number of agents. As environments grow in size, effective credit assignment becomes increasingly harder and often results in infeasible learning times. Still, in many real-world settings, there exist simplified underlying dynamics that can be leveraged for more scalable solutions. In this work, we exploit such locality structures effectively whilst maintaining global cooperation. We propose a novel, value-based multi-agent algorithm called LOMAQ, which incorporates local rewards in the Centralized Training Decentralized Execution paradigm. Additionally, we provide a direct reward decomposition method for finding these local rewards when only a global signal is provided. We test our method empirically, showing it scales well compared to other methods, significantly improving performance and convergence speed.

Adaptive Pairwise Weights for Temporal Credit Assignment

How much credit (or blame) should an action taken in a state get for a future reward? This is the fundamental temporal credit assignment problem in Reinforcement Learning (RL). One of the earliest and still most widely used heuristics is to assign this credit based on a scalar coefficient, lambda (treated as a hyperparameter), raised to the power of the time interval between the state-action and the reward. In this empirical paper, we explore heuristics based on more general pairwise weightings that are functions of the state in which the action was taken, the state at the time of the reward, as well as the time interval between the two. Of course it isn't clear what these pairwise weight functions should be, and because they are too complex to be treated as hyperparameters we develop a metagradient procedure for learning these weight functions during the usual RL training of a policy. Our empirical work shows that it is often possible to learn these pairwise weight functions during learning of the policy to achieve better performance than competing approaches.

Learning by Competition of Self-Interested Reinforcement Learning Agents

An artificial neural network can be trained by uniformly broadcasting a reward signal to units that implement a REINFORCE learning rule. Though this presents a biologically plausible alternative to backpropagation in training a network, the high variance associated with it renders it impractical to train deep networks. The high variance arises from the inefficient structural credit assignment since a single reward signal is used to evaluate the collective action of all units. To facilitate structural credit assignment, we propose replacing the reward signal to hidden units with the change in the L2 norm of the unit's outgoing weight. As such, each hidden unit in the network is trying to maximize the norm of its outgoing weight instead of the global reward, and thus we call this learning method Weight Maximization. We prove that Weight Maximization is approximately following the gradient of rewards in expectation. In contrast to backpropagation, Weight Maximization can be used to train both continuous-valued and discrete-valued units. Moreover, Weight Maximization solves several major issues of backpropagation relating to biological plausibility. Our experiments show that a network trained with Weight Maximization can learn significantly faster than REINFORCE and slightly slower than backpropagation. Weight Maximization illustrates an example of cooperative behavior automatically arising from a population of self-interested agents in a competitive game without any central coordination.

OAM: An Option-Action Reinforcement Learning Framework for Universal Multi-Intersection Control

Efficient traffic signal control is an important means to alleviate urban traffic congestion. Reinforcement learning (RL) has shown great potentials in devising optimal signal plans that can adapt to dynamic traffic congestion. However, several challenges still need to be overcome. Firstly, a paradigm of state, action, and reward design is needed, especially for an optimality-guaranteed reward function. Secondly, the generalization of the RL algorithms is hindered by the varied topologies and physical properties of intersections. Lastly, enhancing the cooperation between intersections is needed for large network applications. To address these issues, the Option-Action RL framework for universal Multi-intersection control (OAM) is proposed. Based on the well-known cell transmission model, we first define a lane-cell-level state to better model the traffic flow propagation. Based on this physical queuing dynamics, we propose a regularized delay as the reward to facilitate temporal credit assignment while maintaining the equivalence with minimizing the average travel time. We then recapitulate the phase actions as the constrained combinations of lane options and design a universal neural network structure to realize model generalization to any intersection with any phase definition. The multiple-intersection cooperation is then rigorously discussed using the potential game theory. We test the OAM algorithm under four networks with different settings, including a city-level scenario with 2,048 intersections using synthetic and real-world datasets. The results show that the OAM can outperform the state-of-the-art controllers in reducing the average travel time.

Neural mechanisms of credit assignment for inferred relationships in a structured world

Animals abstract compact representations of a task's structure, which supports accelerated learning and flexible behavior. Whether and how such abstracted representations may be used to assign credit for inferred, but unobserved, relationships in structured environments are unknown. We develop a hierarchical reversal-learning task and Bayesian learning model to assess the computational and neural mechanisms underlying how humans infer specific choice-outcome associations via structured knowledge. We find that the medial prefrontal cortex (mPFC) efficiently represents hierarchically related choice-outcome associations governed by the same latent cause, using a generalized code to assign credit for both experienced and inferred outcomes. Furthermore, the mPFC and lateral orbitofrontal cortex track the current "position" within a latent association space that generalizes over stimuli. Collectively, these findings demonstrate the importance of both tracking the current position in an abstracted task space and efficient, generalizable representations in the prefrontal cortex for supporting flexible learning and inference in structured environments.

How Do Chemistry Faculty and Graduate Students Engage in Decision Making on Issues Related to Ethical and Responsible Conduct of Research Including Authorship?

In the United States National Science Foundation and the National Institutes of Health have mandated training STEM doctoral students in the ethical and responsible conduct of research to improve doctoral students' ethical decision-making skills; however, little is known about the process and factors that STEM faculty and graduate students use in their decision-making. This exploratory case study examined how four triads of chemistry faculty and their doctoral students recruited from three research universities in the eastern United States engaged in ethical decision-making on issues of authorship, assignment of credit, and plagiarism. A mixed-methods approach involving the administration of an online survey consisting of three open-ended case studies followed by a think-aloud interview was utilized. Participants were found to use analogical reasoning and base their decision-making on a common core set of considerations including fundamental principles, social contracts, consequences, and discussion with an advisor, often using prior personal experiences as sources. Co-authorship did not appear to impact the doctoral students' ethical decision-making. Gender may play a role in graduate students' decision-making; female doctoral students appeared to be less likely to consider prior experiences when evaluating the vignettes. Graduate students' lack of knowledge of the core issues in the responsible conduct of research, coupled with their lack of research experience, and inability to identify the core considerations may lead them to make bad judgments in specific situations. Our findings help explain the minimal impact that the current responsible conduct of research training methods has had on graduate students' ethical decision-making and should lead to the development of more effective approaches.

Choice-selective sequences dominate in cortical relative to thalamic inputs to NAc to support reinforcement learning.

SUMMARYHow are actions linked with subsequent outcomes to guide choices? The nucleus accumbens, which is implicated in this process, receives glutamatergic inputs from the prelimbic cortex and midline regions of the thalamus. However, little is known about whether and how representations differ across these input pathways. By comparing these inputs during a reinforcement learning task in mice, we discovered that prelimbic cortical inputs preferentially represent actions and choices, whereas midline thalamic inputs preferentially represent cues. Choice-selective activity in the prelimbic cortical inputs is organized in sequences that persist beyond the outcome. Through computational modeling, we demonstrate that these sequences can support the neural implementation of reinforcement-learning algorithms, in both a circuit model based on synaptic plasticity and one based on neural dynamics. Finally, we test and confirm a prediction of our circuit models by direct manipulation of nucleus accumbens input neurons.

Rethinking the division of tax room and revenue in fiscal federalism

The division of tax room for shared tax bases, such as income taxes in Canada, the United States, Australia, and Switzerland, is a frequent cause of political conflict between national and sub-national governments. Economists and legal scholars have developed a theory that sets out an optimal division of tax room, but federal nations often substantially depart from this prescription by allocating too much or too little tax room to national governments. This article argues that the divergence between theory and actual practice can be partially explained by a ‘credit assignment problem’ that affects the political economy of government decision making. To illustrate, the article develops a model to identify the incentives governments face when dividing tax room. The model’s central observations are that (a) the optimal division of tax room requires robust intergovernmental contracting, but (b) the necessary contracts are difficult or impossible to perform due to problems in allocating political credit – electoral rewards and punishments – between the two levels of government. The result is political conflict and, frequently, a sub-optimal division of tax room and the revenue that results. This article argues that law can perform two functions in responding to the credit assignment problem: (a) it can facilitate credit assignment, so that the necessary contracts over tax room and revenue are politically feasible, and (b), failing perfect credit assignment, it can mitigate the welfare effects of a sub-optimal division of tax room. The article shows how a credit assignment perspective should lead to reconsideration of constitutional law doctrines, such as the federal spending power and various doctrines that enable or limit concurrent expenditure jurisdiction. While these doctrines are the subject of long-running debates in legal scholarship, the credit assignment perspective offers new insights and new doctrinal prescriptions.

Learning Attentional and Gated Communication via Curiosity.

Due to the partial observability in decentralized multi-agent systems, communication is critical for cooperation. Furthermore, the ability to decide when and whom to communicate is important to achieve efficient communication. However, the existing methods are typically driven by extrinsic rewards. Hence, when the reward from environment is sparse, delayed, or noisy, the communication performance of these methods would be restricted. Furthermore, it would introduce additional difficulty named credit assignment when using extrinsic reward to train communication and sample policies together. To tackle these difficulties, we introduce the mechanism of intrinsic motivation from psychology to multi-agent communication. We hold the view that the observations with more uncertainty and curiosity are more valuable for communication. It can help agent find useful information from observations. It is a good complement to existing extrinsic driven methods. Concretely, at sending end, we learn a curiosity from local observations to model the communication importance. Then, we design a heuristic mechanism to prune unnecessary messages. It can solve the problem of when to communicate. Then, the ability to gate unnecessary message can reduce the cost and improve the efficiency of communication, which is important to apply to real-world scenarios. Furthermore, at receiving end, we utilize the intrinsic importance to differentiate information, which can be helpful for local decisions. It could solve the problem of whom to communicate. The ability to pay attention to useful information can efficiently improve the performance of communication behaviors. At last, we evaluate our method on a variety of multi-agent scenarios. The experiments of full communication demonstrate that the curiosity is capable to model the communication importance, and the results of gated communication further prove the conclusion.

Learning Cooperative Multi-Agent Policies With Partial Reward Decoupling

One of the preeminent obstacles to scaling multi-agent reinforcement learning to large numbers of agents is assigning credit to individual agents' actions. In this paper, we address this credit assignment problem with an approach that we call \textit{partial reward decoupling} (PRD), which attempts to decompose large cooperative multi-agent RL problems into decoupled subproblems involving subsets of agents, thereby simplifying credit assignment. We empirically demonstrate that decomposing the RL problem using PRD in an actor-critic algorithm results in lower variance policy gradient estimates, which improves data efficiency, learning stability, and asymptotic performance across a wide array of multi-agent RL tasks, compared to various other actor-critic approaches. Additionally, we relate our approach to counterfactual multi-agent policy gradient (COMA), a state-of-the-art MARL algorithm, and empirically show that our approach outperforms COMA by making better use of information in agents' reward streams, and by enabling recent advances in advantage estimation to be used.

Adaptive operator selection with test-and-apply structure for decomposition-based multi-objective optimization

In this paper, we propose a test-and-apply based adaptive operator selection strategy (TAOS) towards significant improvement of decomposition-based multi-objective optimization. In this approach, the entire evolutionary process is structured into several continuous sections, each of which is designed to run a test-and-apply procedure, in order to pave the way for adaptive selection of the best possible operators. In the phase of test, the adopted operators are tested by running sequentially on individuals and then the operators credits are assigned by their successful update of counts to replace the parent solutions. In the phase of apply, the best operator is selected to run on the remaining time of this section. In comparison with the existing state of the arts, our introduced two phases with the test-and-apply strategy not only achieve a better fairness, but also a more appropriate balance between exploration and exploitation for the decomposition-based multi-objective evolutionary algorithms, where the decomposition approach is convenient for credit assignment in the test phase. To evaluate our proposed strategy, we have carried out extensive experiments and the results support that our proposed outperforms the existing state of the arts on three sets of multi-objective optimization problems.

Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents

How do societies learn and maintain social norms? Here we use multiagent reinforcement learning to investigate the learning dynamics of enforcement and compliance behaviors. Artificial agents populate a foraging environment and need to learn to avoid a poisonous berry. Agents learn to avoid eating poisonous berries better when doing so is taboo, meaning the behavior is punished by other agents. The taboo helps overcome a credit assignment problem in discovering delayed health effects. Critically, introducing an additional taboo, which results in punishment for eating a harmless berry, further improves overall returns. This "silly rule" counterintuitively has a positive effect because it gives agents more practice in learning rule enforcement. By probing what individual agents have learned, we demonstrate that normative behavior relies on a sequence of learned skills. Learning rule compliance builds upon prior learning of rule enforcement by other agents. Our results highlight the benefit of employing a multiagent reinforcement learning computational model focused on learning to implement complex actions.

Video Content Placement At the Network Edge: Centralized and Distributed Algorithms

In the traditional video streaming service provisioning paradigm, viewers typically request video content through a central Content Delivery Network (CDN) server. However, because of the uncertain wide area network delays, the (remote) viewers usually suffer from long video streaming delay, which affects the quality of experience. Multi-Access Edge Computing (MEC) offers a way to shorten the video streaming delay by building small-scale cloud infrastructures at the network edge, which are in close proximity to the viewers. In this paper, we present novel centralized and distributed algorithms for the video content placement problem in MEC. In the proposed centralized video content placement algorithm, we leverage the Lyapunov optimization technique to formulate the video content placement problem as a series of one-time-slot optimization problems and apply an Alternating Direction Method of Multipliers (ADMM)-based method to solve each of them. We further devise a distributed Multi-Agent Reinforcement Learning (MARL)-based method with value decomposition mechanism and parallelization policy update method to solve the video content placement problem. The value Decomposition mechanism deals with the credit assignment among multiple agents, which promotes the cooperative optimization of the global target and reduces the frequency of information exchange. The parallelization of policy network can speed up the convergence process. Simulation results verify the effectiveness and superiority of our proposed centralized and distributed algorithms in terms of performance.

How to Fool Your Supervisor

Reinforcement learning is being broadly studied as a solution for many real-world problems. Reinforcement learning algorithms outperform humans in games such as Go and chess. On the other hand, reinforcement learning still can not achieve human-level in most real-world tasks. When reinforcement learning agents perform similar to humans, they will replace humans in many areas. Many real-world tasks need cooperation, and multi-agent reinforcement learning appears as a solution. However, multi-agent reinforcement learning algorithms are not yet being applied because of the challenge of high-dimensional data, computational complexity, and credit assignment. Meanwhile, people can manage to adapt to a new group without understanding the global goal of the group he or she belongs to. This is because people moderately imitate senior workers. Our work aims to test if the imitation strategy can be applied to a reinforcement learning agent. To validate the proposed methodology, we design a problem named Convoy problem and evaluate our new algorithm on the convoy problem.

Assigning Credits to Multiple Contributors of a Scholarly Output Using Arithmetic Series

As the academic world is focusing deeply on quantification of contributions brought about by researchers, it becomes imperative to assign true and deserving credits to the individuals. Modern-day databases do not account for the number of contributors or sequence number of a contributor in the contributors’ list while assigning credit of a scholarly output. As such, every contributor of a researcher-studded output gets the same credit as does the sole contributor of a single-author output. The current work presents a mathematical method to assign credits based on the number of contributors, the sequence of the contributor in the contributors list (if so required), and the magnitude of commendation achieved per unit time by the scholarly output. The concept of arithmetic series is utilized to materialize the idea of true assignment of credits. A case study is also provided to elaborate working of the mathematical method.