Computational Learning Model Research Articles

How humans learn and represent networks

Humans receive information from the world around them in sequences of discrete items—from words in language or notes in music to abstract concepts in books and websites on the Internet. To model their environment, from a young age people are tasked with learning the network structures formed by these items (nodes) and the connections between them (edges). But how do humans uncover the large-scale structures of networks when they experience only sequences of individual items? Moreover, what do people’s internal maps and models of these networks look like? Here, we introduce graph learning, a growing and interdisciplinary field studying how humans learn and represent networks in the world around them. Specifically, we review progress toward understanding how people uncover the complex webs of relationships underlying sequences of items. We begin by describing established results showing that humans can detect fine-scale network structure, such as variations in the probabilities of transitions between items. We next present recent experiments that directly control for differences in transition probabilities, demonstrating that human behavior depends critically on the mesoscale and macroscale properties of networks. Finally, we introduce computational models of human graph learning that make testable predictions about the impact of network structure on people’s behavior and cognition. Throughout, we highlight open questions in the study of graph learning that will require creative insights from cognitive scientists and network scientists alike.

Domain-General Tutor Authoring with Apprentice Learner Models

Intelligent tutoring systems are effective for improving students’ learning outcomes (Pane et al. 2013; Koedinger and Anderson, International Journal of Artificial Intelligence in Education, 8, 1–14, 1997; Bowen et al. Journal of Policy Analysis and Management, 1, 94–111 2013). However, constructing tutoring systems that are pedagogically effective has been widely recognized as a challenging problem (Murray 2003; Murray, International Journal of Artificial Intelligence in Education, 10, 98–129, 1999). In this paper, we explore the use of computational models of apprentice learning, or computer models that learn interactively from examples and feedback, for authoring expert-models via demonstrations and feedback (Matsuda et al. International Journal of Artificial Intelligence in Education, 25(1), 1–34 2014) across a wide range of domains. To support these investigations, we present the Apprentice Learner Architecture, which posits the types of knowledge, performance, and learning components needed for apprentice learning. We use this architecture to create two models: the Decision Tree model, which non-incrementally learns skills, and the Trestle model, which instead learns incrementally. Both models draw on the same small set of prior knowledge (six operators and three types of relational knowledge) to support expert model authoring. Despite their limited prior knowledge, we demonstrate their use for efficiently authoring a novel experimental design tutor and show that they are capable of learning an expert model for seven additional tutoring systems that teach a wide range of knowledge types (associations, categories, and skills) across multiple domains (language, math, engineering, and science). This work shows that apprentice learner models are efficient for authoring tutors that would be difficult to build with existing non-programmer authoring approaches (e.g., experimental design or stoichiometry tutors). Further, we show that these models can be applied to author tutors across eight tutor domains even though they only have a small, fixed set of prior knowledge. This work lays the foundation for new interactive machine-learning based authoring paradigms that empower teachers and other non-programmers to build pedagogically effective educational technologies at scale.

What Constitutes a Science of Reading Instruction?

ABSTRACTRecently, the term science of reading has been used in public debate to promote policies and instructional practices based on research on the basic cognitive mechanisms of reading, the neural processes involved in reading, computational models of learning to read, and the like. According to those views, such data provide convincing evidence that explicit decoding instruction (e.g., phonological awareness, phonics) should be beneficial to reading success. Nevertheless, there has been pushback against such policies, the use of the term science of reading by “phonics‐centric people”, and their lack of instructional knowledge and experience. In this article, although the author supports pedagogical decision making on the basis of a confluence of evidence from a variety of sources, he cautions against instructional overgeneralizations based on various kinds of basic research without an adequate consideration of instructional experiments. The author provides several examples of the premature translation of basic research findings into wide‐scale pedagogical application.

A Hierarchical Bayesian Implementation of the Experience-Weighted Attraction Model.

Social and decision-making deficits are often the first symptoms of neuropsychiatric disorders. In recent years, economic games, together with computational models of strategic learning, have been increasingly applied to the characterization of individual differences in social behavior, as well as their changes across time due to disease progression, treatment, or other factors. At the same time, the high dimensionality of these data poses an important challenge to statistical estimation of these models, potentially limiting the adoption of such approaches in patients and special populations. We introduce a hierarchical Bayesian implementation of a class of strategic learning models, experience-weighted attraction (EWA), that is widely used in behavioral game theory. Importantly, this approach provides a unified framework for capturing between- and within-participant variation, including changes associated with disease progression, comorbidity, and treatment status. We show using simulated data that our hierarchical Bayesian approach outperforms representative agent and individual-level estimation methods that are commonly used in extant literature, with respect to parameter estimation and uncertainty quantification. Furthermore, using an empirical dataset, we demonstrate the value of our approach over competing methods with respect to balancing model fit and complexity. Consistent with the success of hierarchical Bayesian approaches in other areas of behavioral science, our hierarchical Bayesian EWA model represents a powerful and flexible tool to apply to a wide range of behavioral paradigms for studying the interplay between complex human behavior and biological factors.

A biologically-inspired reinforcement learning based intelligent distributed flocking control for Multi-Agent Systems in presence of uncertain system and dynamic environment

In this paper, we investigate the real-time flocking control of Multi-Agent Systems (MAS) in the presence of system uncertainties and dynamic environment. To handle the impacts from system uncertainties and dynamic environment, a novel reinforcement learning technique, which is appropriate for real-time implementation, has been integrated with multi-agent flocking control in this paper. The Brain Emotional Learning Based Intelligent Controller (BELBIC) is a biologically-inspired reinforcement learning-based controller relying on a computational model of emotional learning in the mammalian limbic system. The learning capabilities, multi-objective properties, and low computational complexity of BELBIC make it a very promising learning technique for implementation in real-time applications. Firstly, a novel brain emotional learning-based flocking control structure is proposed. Then, the real-time update laws are developed to tune the emotional signals based on real-time operational data. It is important to note that this data-driven reinforcement learning approach relaxes the requirement for system dynamics and effectively handle the uncertain impacts of the environment. Using the tuned emotional signals, the optimal flocking control can be obtained. The Lyapunov analysis has been used to prove the convergence of the proposed design. The effectiveness of the proposed design is also demonstrated through numerical and experimental results based on the coordination of multiple Unmanned Aerial Vehicles (UAVs).

The Object Space Task reveals increased expression of cumulative memory in a mouse model of Kleefstra syndrome

Kleefstra syndrome is a disorder caused by a mutation in the EHMT1 gene characterized in humans by general developmental delay, mild to severe intellectual disability and autism. Here, we characterized cumulative memory in the Ehmt1+/− mouse model using the Object Space Task. We combined conventional behavioral analysis with automated analysis by deep-learning networks, a session-based computational learning model, and a trial-based classifier. Ehmt1+/− mice showed more anxiety-like features and generally explored objects less, but the difference decreased over time. Interestingly, when analyzing memory-specific exploration, Ehmt1+/− show increased expression of cumulative memory, but a deficit in a more simple, control memory condition. Using our automatic classifier to differentiate between genotypes, we found that cumulative memory features are better suited for classification than general exploration differences. Thus, detailed behavioral classification with the Object Space Task produced a more detailed behavioral phenotype of the Ehmt1+/− mouse model.

Characterizing the Dynamics of Learning in Repeated Reference Games.

The language we use over the course of conversation changes as we establish common ground and learn what our partner finds meaningful. Here we draw upon recent advances in natural language processing to provide a finer-grained characterization of the dynamics of this learning process. We release an open corpus (>15,000 utterances) of extended dyadic interactions in a classic repeated reference game task where pairs of participants had to coordinate on how to refer to initially difficult-to-describe tangram stimuli. We find that different pairs discover a wide variety of idiosyncratic but efficient and stable solutions to the problem of reference. Furthermore, these conventions are shaped by the communicative context: words that are more discriminative in the initial context (i.e., that are used for one target more than others) are more likely to persist through the final repetition. Finally, we find systematic structure in how a speaker's referring expressions become more efficient over time: Syntactic units drop out in clusters following positive feedback from the listener, eventually leaving short labels containing open-class parts of speech. These findings provide a higher resolution look at the quantitative dynamics of ad hoc convention formation and support further development of computational models of learning in communication.

Unsupervised experience with temporal continuity of the visual environment is causally involved in the development of V1 complex cells.

Unsupervised adaptation to the spatiotemporal statistics of visual experience is a key computational principle that has long been assumed to govern postnatal development of visual cortical tuning, including orientation selectivity of simple cells and position tolerance of complex cells in primary visual cortex (V1). Yet, causal empirical evidence supporting this hypothesis is scant. Here, we show that degrading the temporal continuity of visual experience during early postnatal life leads to a sizable reduction of the number of complex cells and to an impairment of their functional properties while fully sparing the development of simple cells. This causally implicates adaptation to the temporal structure of the visual input in the development of transformation tolerance but not of shape tuning, thus tightly constraining computational models of unsupervised cortical learning.

Predicting Regioselectivity in Radical C−H Functionalization of Heterocycles through Machine Learning

AbstractRadical C−H bond functionalization provides a versatile approach for elaborating heterocyclic compounds. The synthetic design of this transformation relies heavily on the knowledge of regioselectivity, while a quantified and efficient regioselectivity prediction approach is still elusive. Herein, we report the feasibility of using a machine learning model to predict the transition state barrier from the computed properties of isolated reactants. This enables rapid and reliable regioselectivity prediction for radical C−H bond functionalization of heterocycles. The Random Forest model with physical organic features achieved 94.2 % site accuracy and 89.9 % selectivity accuracy in the out‐of‐sample test set. The prediction performance was further validated by comparing the machine learning results with additional substituents, heteroarene scaffolds and experimental observations. This work revealed that the combination of mechanism‐based computational statistics and machine learning model can serve as a useful strategy for selectivity prediction of organic transformations.

Predicting Regioselectivity in Radical C-H Functionalization of Heterocycles through Machine Learning.

Radical C-H bond functionalization provides a versatile approach for elaborating heterocyclic compounds. The synthetic design of this transformation relies heavily on the knowledge of regioselectivity, while a quantified and efficient regioselectivity prediction approach is still elusive. Herein, we report the feasibility of using a machine learning model to predict the transition state barrier from the computed properties of isolated reactants. This enables rapid and reliable regioselectivity prediction for radical C-H bond functionalization of heterocycles. The Random Forest model with physical organic features achieved 94.2 % site accuracy and 89.9 % selectivity accuracy in the out-of-sample test set. The prediction performance was further validated by comparing the machine learning results with additional substituents, heteroarene scaffolds and experimental observations. This work revealed that the combination of mechanism-based computational statistics and machine learning model can serve as a useful strategy for selectivity prediction of organic transformations.

Beyond the average: The role of variable reward sensitivity in eating disorders

Eating disorders are often characterized by episodes of overeating and undereating. To date, most theories have explained the liability for such episodes by differences in traits such as reward sensitivity or cognitive control. Here, we review the evidence for a more parsimonious account of the waxing and waning in food intake by linking it to state-like variability of alleged traits such as reward sensitivity. To formally demonstrate that our variability model of eating disorders could explain a wide range of observed reward-related behavior, we conducted simulations of value-based choices and learning. These simulations based on well-established computational models of reinforcement learning and Bayesian sequential updating show how variability may arise and manifest in eating behavior. We argue that by reconceptualizing stable traits as distributions over likely states promoting adaptation, our proposed model integrates disparate findings and leads to novel predictions in a quantitative framework. Collectively, these emerging results call for a stronger emphasis on within-person variability to improve mechanistic insights into eating disorders.

Stimulation of the vagus nerve reduces learning in a go/no-go reinforcement learning task

When facing decisions to approach rewards or to avoid punishments, we often figuratively go with our gut, and the impact of metabolic states such as hunger on motivation are well documented. However, whether and how vagal feedback signals from the gut influence instrumental actions is unknown. Here, we investigated the effect of non-invasive transcutaneous auricular vagus nerve stimulation (taVNS) vs. sham (randomized cross-over design) on approach and avoidance behavior using an established go/no-go reinforcement learning paradigm in 39 healthy human participants (23 female) after an overnight fast. First, mixed-effects logistic regression analysis of choice accuracy showed that taVNS acutely impaired decision-making, p = .041. Computational reinforcement learning models identified the cause of this as a reduction in the learning rate through taVNS (∆α = −0.092, pboot = .002), particularly after punishment (∆αPun = −0.081, pboot = .012 vs. ∆αRew =−0.031, pboot = .22). However, taVNS had no effect on go biases, Pavlovian response biases or response time. Hence, taVNS appeared to influence learning rather than action execution. These results highlight a novel role of vagal afferent input in modulating reinforcement learning by tuning the learning rate according to homeostatic needs.

Altered Reinforcement Learning From Reward and Punishment in Anorexia Nervosa: Evidence From Computational Modeling

OBJECTIVES Anorexia nervosa (AN) is associated with altered sensitivity to reward and punishment. Few studies have investigated whether this results in aberrant learning. The ability to learn from rewarding and aversive experiences is essential for flexibly adapting to changing environments, yet individuals with AN tend to demonstrate cognitive inflexibility, difficulty set-shifting and altered decision-making. Deficient reinforcement learning may contribute to repeated engagement in maladaptive behavior. METHODS This study investigated learning in AN using a probabilistic associative learning task that separated learning of stimuli via reward from learning via punishment. Forty-two individuals with Diagnostic and Statistical Manual of Mental Disorders (DSM)-5 restricting-type AN were compared to 38 healthy controls (HCs). We applied computational models of reinforcement learning to assess group differences in learning, thought to be driven by violations in expectations, or prediction errors (PEs). Linear regression analyses examined whether learning parameters predicted BMI at discharge. RESULTS AN had lower learning rates than HC following both positive and negative PE (p < .02), and were less likely to exploit what they had learned. Negative PE on punishment trials predicted lower discharge BMI (p < .001), suggesting individuals with more negative expectancies about avoiding punishment had the poorest outcome. CONCLUSIONS This is the first study to show lower rates of learning in AN following both positive and negative outcomes, with worse punishment learning predicting less weight gain. An inability to modify expectations about avoiding punishment might explain persistence of restricted eating despite negative consequences, and suggests that treatments that modify negative expectancy might be effective in reducing food avoidance in AN.

Arbitrating Computational Models of Observational Learning

How do we learn in the absence of direct experience? In this issue of Neuron, Charpentier etal. (2020) proposes a new computational account of observational learning, which arbitrates between choice imitation and goal emulation.

Bayesian modelling captures inter-individual differences in social belief computations in the putamen and insula

Computational models of social learning and decision-making provide mechanistic tools to investigate the neural mechanisms that are involved in understanding other people. While most studies employ explicit instructions to learn from social cues, everyday life is characterized by the spontaneous use of such signals (e.g., the gaze of others) to infer on internal states such as intentions. To investigate the neural mechanisms of the impact of gaze cues on learning and decision-making, we acquired behavioural and fMRI data from 50 participants performing a probabilistic task, in which cards with varying winning probabilities had to be chosen. In addition, the task included a computer-generated face that gazed towards one of these cards providing implicit advice. Participants’ individual belief trajectories were inferred using a hierarchical Gaussian filter (HGF) and used as predictors in a linear model of neuronal activation. During learning, social prediction errors were correlated with activity in inferior frontal gyrus and insula. During decision-making, the belief about the accuracy of the social cue was correlated with activity in inferior temporal gyrus, putamen and pallidum while the putamen and insula showed activity as a function of individual differences in weighting the social cue during decision-making. Our findings demonstrate that model-based fMRI can give insight into the behavioural and neural aspects of spontaneous social cue integration in learning and decision-making. They provide evidence for a mechanistic involvement of specific components of the basal ganglia in subserving these processes.

Cognitive inflexibility specificity for individuals with high levels of obsessive-compulsive symptoms

Abstract Obsessive and compulsive symptoms are only partially explained by current cognitive-behavioral models. A complementary proposal, behavioral dysregulation, considers cognitive inflexibility (a malfunction of the capacity of switching between mental processes in order to generate appropriate behavioral responses) as an endophenotype of OCD. However, it is unclear whether deficits in cognitive flexibility are specific to individuals with OCD, or whether cognitive underperformance in OCD should instead be explained by nonspecific factors. Here, we probed cognitive flexibility by using symptom-related stimuli in a probabilistic reversal learning task. We compared performance of individuals with two closely related OCD-related phenomena: OC symptoms and “Not Just Right Experiences.” Data were analyzed through computational models of reinforcement learning. Academic achievement was used for comparison as an unrelated learning task. Our results support the inflexibility specificity hypothesis, but only for the processing of symptom-related triggers. Clinical implications are discussed in relation to contemporary CBT models.

Machine learning identifies the dynamics and influencing factors in an auditory category learning experiment

Human learning is one of the main topics in psychology and cognitive neuroscience. The analysis of experimental data, e.g. from category learning experiments, is a major challenge due to confounding factors related to perceptual processing, feedback value, response selection, as well as inter-individual differences in learning progress due to differing strategies or skills. We use machine learning to investigate (Q1) how participants of an auditory category-learning experiment evolve towards learning, (Q2) how participant performance saturates and (Q3) how early we can differentiate whether a participant has learned the categories or not. We found that a Gaussian Mixture Model describes well the evolution of participant performance and serves as basis for identifying influencing factors of task configuration (Q1). We found early saturation trends (Q2) and that CatBoost, an advanced classification algorithm, can separate between participants who learned the categories and those who did not, well before the end of the learning session, without much degradation of separation quality (Q3). Our results show that machine learning can model participant dynamics, identify influencing factors of task design and performance trends. This will help to improve computational models of auditory category learning and define suitable time points for interventions into learning, e.g. by tutorial systems.

A Computational Role for Top-Down Modulation from Frontal Cortex in Infancy.

Recent findings have shown that full-term infants engage in top-down sensory prediction, and these predictions are impaired as a result of premature birth. Here, we use an associative learning model to uncover the neuroanatomical origins and computational nature of this top-down signal. Infants were exposed to a probabilistic audiovisual association. We find that both groups (full term, preterm) have a comparable stimulus-related response in sensory and frontal lobes and track prediction error in their frontal lobes. However, preterm infants differ from their full-term peers in weaker tracking of prediction error in sensory regions. We infer that top-down signals from the frontal lobe to the sensory regions carry information about prediction error. Using computational learning models and comparing neuroimaging results from full-term and preterm infants, we have uncovered the computational content of top-down signals in young infants when they are engaged in a probabilistic associative learning.

Doubting what you already know: Uncertainty regarding state transitions is associated with obsessive compulsive symptoms.

Obsessive compulsive (OC) symptoms involve excessive information gathering (e.g., checking, reassurance-seeking), and uncertainty about possible, often catastrophic, future events. Here we propose that these phenomena are the result of excessive uncertainty regarding state transitions (transition uncertainty): a computational impairment in Bayesian inference leading to a reduced ability to use the past to predict the present and future, and to oversensitivity to feedback (i.e. prediction errors). Using a computational model of Bayesian learning under uncertainty in a reversal learning task, we investigate the relationship between OC symptoms and transition uncertainty. Individuals high and low in OC symptoms performed a task in which they had to detect shifts (i.e. transitions) in cue-outcome contingencies. Modeling subjects' choices was used to estimate each individual participant's transition uncertainty and associated responses to feedback. We examined both an optimal observer model and an approximate Bayesian model in which participants were assumed to attend (and learn about) only one of several cues on each trial. Results suggested the participants were more likely to distribute attention across cues, in accordance with the optimal observer model. As hypothesized, participants with higher OC symptoms exhibited increased transition uncertainty, as well as a pattern of behavior potentially indicative of a difficulty in relying on learned contingencies, with no evidence for perseverative behavior. Increased transition uncertainty compromised these individuals' ability to predict ensuing feedback, rendering them more surprised by expected outcomes. However, no evidence for excessive belief updating was found. These results highlight a potential computational basis for OC symptoms and obsessive compulsive disorder (OCD). The fact the OC symptoms predicted a decreased reliance on the past rather than perseveration challenges preconceptions of OCD as a disorder of inflexibility. Our results have implications for the understanding of the neurocognitive processes leading to excessive uncertainty and distrust of past experiences in OCD.

Social structure learning in human anterior insula.

Humans form social coalitions in every society, yet we know little about how we learn and represent social group boundaries. Here we derive predictions from a computational model of latent structure learning to move beyond explicit category labels and interpersonal, or dyadic, similarity as the sole inputs to social group representations. Using a model-based analysis of functional neuroimaging data, we find that separate areas correlate with dyadic similarity and latent structure learning. Trial-by-trial estimates of 'allyship' based on dyadic similarity between participants and each agent recruited medial prefrontal cortex/pregenual anterior cingulate (pgACC). Latent social group structure-based allyship estimates, in contrast, recruited right anterior insula (rAI). Variability in the brain signal from rAI improved prediction of variability in ally-choice behavior, whereas variability from the pgACC did not. These results provide novel insights into the psychological and neural mechanisms by which people learn to distinguish 'us' from 'them.'