Learned Behavior Research Articles

Compressed sensing based approach identifies modular neural circuitry driving learned pathogen avoidance.

An animal's survival hinges on its ability to integrate past information to modify future behavior. The nematode C. elegans adapts its behavior based on prior experiences with pathogen exposure, transitioning from attraction to avoidance of the pathogen. A systematic screen for the neural circuits that integrate the information of previous pathogen exposure to modify behavior has not been feasible because of the lack of tools for neuron type specific perturbations. We overcame this challenge using methods based on compressed sensing to efficiently determine the roles of individual neuron types in learned avoidance behavior. Our screen revealed that distinct sets of neurons drive exit from lawns of pathogenic bacteria and prevent lawn re-entry. Using calcium imaging of freely behaving animals and optogenetic perturbations, we determined the neural dynamics that regulate one key behavioral transition after infection: stalled re-entry into bacterial lawns. We find that key neuron types govern pathogen lawn specific stalling but allow the animal to enter nonpathogenic E. coli lawns. Our study shows that learned pathogen avoidance requires coordinated transitions in discrete neural circuits and reveals the modular structure of this complex adaptive behavioral response to infection.

Song-like activation of syringeal and respiratory muscles during sleep in canaries.

Sleep replay activity involves the reactivation of brain structures with patterns similar to those observed during waking behavior. In this study, we demonstrate that adult male canaries exhibit spontaneous, song-like peripheral reactivation during night sleep. Our findings include: (1) the presence of activity in respiratory muscles, leading to song-like air sac pressure patterns of low amplitude, (2) the simultaneous occurrence of respiratory replay events and reactivation of syringeal muscles, and (3) the reactivation of syringeal muscles without concurrent respiratory system activity. This song-like reactivation of peripheral motor systems enables the identification of specific motor patterns, with replay events preserving individual morphological and temporal properties. The activation of peripheral motor systems in songbirds and the differences in activation patterns between species give unique insights into the fictive behavioral output of activation of a complex learned motor behavior during sleep, shedding light on the neural control mechanisms and potential functions.

PUMA: Deep Metric Imitation Learning for Stable Motion Primitives

Imitation learning (IL) facilitates intuitive robotic programming. However, ensuring the reliability of learned behaviors remains a challenge. In the context of reaching motions, a robot should consistently reach its goal, regardless of its initial conditions. To meet this requirement, IL methods often employ specialized function approximators that guarantee this property by construction. Although effective, these approaches come with some limitations: 1) they are typically restricted in the range of motions they can model, resulting in suboptimal IL capabilities, and 2) they require explicit extensions to account for the geometry of motions that consider orientations. To address these challenges, we introduce a novel stability loss function that does not constrain the function approximator's architecture and enables learning policies that yield accurate results. Furthermore, it is not restricted to a specific state space geometry; therefore, it can easily incorporate the geometry of the robot's state space. Proof of the stability properties induced by this loss is provided and the method is empirically validated in various settings. These settings include Euclidean and non‐Euclidean state spaces, as well as first‐order and second‐order motions, both in simulation and with real robots. More details about the experimental results can be found at https://youtu.be/ZWKLGntCI6w.

How does anthropogenic food influence the trophic ecology of Rocky Mountain Red Fox?

Abstract Providing novel food subsidies to wildlife can influence behavior, resource use, disease dynamics, and food web interactions. Studies examining impacts of anthropogenic food have primarily occurred in urban landscapes where anthropogenic resources are consistently accessible. National parks, however, experience similar issues regarding wildlife exploiting anthropogenic food, but access to anthropogenic food is contingent on human visitation, which can fluctuate dramatically between seasons. We evaluated how fluctuating availability of anthropogenic food and demographic factors influenced the isotopic niche of Rocky Mountain Red Fox (Vulpes vulpes macroura) at the population and individual level in Grand Teton National Park, Wyoming. From 2016 to 2021, we assessed diet-related questions by collecting 55 whisker samples and 52 hair samples from 40 unique individuals and analyzed them using stable isotope analysis (δ13C and δ15N). We discovered that Rocky Mountain red foxes have a broad isotopic niche, with seasonal differences in their consumption of anthropogenic food. Contrary to our expectations, foxes demonstrated increased reliance on anthropogenic food (higher δ13C values) during winter months, whereas natural food consumption or dietary mixing was more prominent in summer months. Natural food scarcity could be driving foxes to seek out more anthropogenic food in the winter, whereas the increased abundance of natural food items in the summer could result in a lower need for foxes to supplement their diet with anthropogenic food. Adults had consistently higher δ13C values relative to juveniles, indicating either intraspecific variation in boldness or dominance leading to increased exploitation of anthropogenic food, which could be a learned behavior coinciding with habituation. Finally, we observed that increasing densities of human features within the home range of a fox was a strong predictor of anthropogenic food reliance at the individual level. Our work provided a deeper understanding of the trophic ecology of Rocky Mountain Red Fox in relation to human activity in a nature preserve, and equipped managers with insight into potential patterns associated with food-conditioned foxes.

From Skill Development to Skill Duration: Three Strategies for Cultivating a Culture of Learning in Your Organization

This article discusses strategies for leaders to foster a culture of continuous learning within their organizations. It establishes that learning is a learned behavior that must be developed over time through reinforcement and environment. Left to chance, companies will not cultivate an effective learning culture. The article then presents three key steps leaders can take: make learning part of daily work through reflection sessions, identifying teachable moments, and recognizing achievements; support ongoing skill development with resources, peer learning, and skills gap assessments; and lead by example through participation in learning activities, promoting experimentation, and developing adaptability in hiring and coaching. It provides an example of implementing these strategies at a financial services firm and the positive outcomes it achieved. The conclusion emphasizes the integral role of leaders in socializing learning as a habit and competitive advantage through intentional culture design. A culture of continuous learning will help organizations adapt to future challenges.

Toward the confident deployment of real‐world reinforcement learning agents

AbstractIntelligent learning agents must be able to learn from experience so as to accomplish tasks that require more ability than could be initially programmed. Reinforcement learning (RL) has emerged as a potentially powerful class of solution methods to create agents that learn from trial‐and‐error interaction with the world. Despite many prominent success stories, a number of challenges often stand between the use of RL in real‐world problems. As part of the AAAI New Faculty Highlight Program, in this article, I will describe the work that my group is doing at the University of Wisconsin—Madison with the intent to remove barriers to the use of RL in practice. Specifically, I will describe recent work that aims to give practitioners confidence in learned behaviors, methods to increase the data efficiency of RL, and work on “challenge” domains that stress RL algorithms beyond current testbeds.

Discovering and Exploiting Sparse Rewards in a Learned Behavior Space.

Learning optimal policies in sparse rewards settings is difficult as the learning agent has little to no feedback on the quality of its actions. In these situations, a good strategy is to focus on exploration, hopefully leading to the discovery of a reward signal to improve on. A learning algorithm capable of dealing with this kind of setting has to be able to (1) explore possible agent behaviors and (2) exploit any possible discovered reward. Exploration algorithms have been proposed that require the definition of a low-dimension behavior space, in which the behavior generated by the agent's policy can be represented. The need to design a priori this space such that it is worth exploring is a major limitation of these algorithms. In this work, we introduce STAX, an algorithm designed to learn a behavior space on-the-fly and to explore it while optimizing any reward discovered (see Figure 1). It does so by separating the exploration and learning of the behavior space from the exploitation of the reward through an alternating two-step process. In the first step, STAX builds a repertoire of diverse policies while learning a low-dimensional representation of the high-dimensional observations generated during the policies evaluation. In the exploitation step, emitters optimize the performance of the discovered rewarding solutions. Experiments conducted on three different sparse reward environments show that STAX performs comparably to existing baselines while requiring much less prior information about the task as it autonomously builds the behavior space it explores.

Spatial and seasonal foraging patterns drive diet differences among north Pacific resident killer whale populations.

Highly social top marine predators, including many cetaceans, exhibit culturally learned ecological behaviours such as diet preference and foraging strategy that can affect their resilience to competition or anthropogenic impacts. When these species are also endangered, conservation efforts require management strategies based on a comprehensive understanding of the variability in these behaviours. In the northeast Pacific Ocean, three partially sympatric populations of resident killer whales occupy coastal ecosystems from California to Alaska. One population (southern resident killer whales) is endangered, while another (southern Alaska resident killer whales) has exhibited positive abundance trends for the last several decades. Using 185 faecal samples collected from both populations between 2011 and 2021, we compare variability in diet preference to provide insight into differences in foraging patterns that may be linked with the relative success and decline of these populations. We find broad similarities in the diet of the two populations, with differences arising from spatiotemporal and social variability in resource use patterns, especially in the timing of shifts between target prey species. The results described here highlight the importance of comprehensive longitudinal monitoring of foraging ecology to inform management strategies for endangered, highly social top marine predators.

Independent operations of appetitive and aversive conditioning systems lead to simultaneous production of conflicting memories in an insect.

Pavlovian conditioning is a ubiquitous form of associative learning that enables animals to remember appetitive and aversive experiences. Animals possess appetitive and aversive conditioning systems that memorize and retrieve appetitive and aversive experiences. Here, we addressed a question of whether integration of competing appetitive and aversive information takes place during the encoding of the experience or during memory retrieval. We developed novel experimental procedures to address this question using crickets (Gryllus bimaculatus), which allowed selective blockade of the expression of appetitive and aversive memories by injecting octopamine and dopamine receptor antagonists. We conditioned an odour (conditioned stimulus 1, CS1) with water and then with sodium chloride solution. At 24 h after conditioning, crickets retained both appetitive and aversive memories, and the memories were integrated to produce a conditioned response (CR). Importantly, when a visual pattern (CS2) was conditioned with CS1, appetitive and aversive memories formed simultaneously. This indicates that appetitive and aversive second-order conditionings are achieved at the same time. The memories were integrated for producing a conditioned response. We conclude that appetitive and aversive conditioning systems operate independently to form parallel appetitive and aversive memories, which compete to produce learned behaviour in crickets.

Resilience of A Learned Motor Behavior After Chronic Disruption of Inhibitory Circuits.

Maintaining motor behaviors throughout life is crucial for an individual's survival and reproductive success. The neuronal mechanisms that preserve behavior are poorly understood. To address this question, we focused on the zebra finch, a bird that produces a highly stereotypical song after learning it as a juvenile. Using cell-specific viral vectors, we chronically silenced inhibitory neurons in the pre-motor song nucleus called the high vocal center (HVC), which caused drastic song degradation. However, after producing severely degraded vocalizations for around 2 months, the song rapidly improved, and animals could sing songs that highly resembled the original. In adult birds, single-cell RNA sequencing of HVC revealed that silencing interneurons elevated markers for microglia and increased expression of the Major Histocompatibility Complex I (MHC I), mirroring changes observed in juveniles during song learning. Interestingly, adults could restore their songs despite lesioning the lateral magnocellular nucleus of the anterior neostriatum (LMAN), a brain nucleus crucial for juvenile song learning. This suggests that while molecular mechanisms may overlap, adults utilize different neuronal mechanisms for song recovery. Chronic and acute electrophysiological recordings within HVC and its downstream target, the robust nucleus of the archistriatum (RA), revealed that neuronal activity in the circuit permanently altered with higher spontaneous firing in RA and lower in HVC compared to control even after the song had fully recovered. Together, our findings show that a complex learned behavior can recover despite extended periods of perturbed behavior and permanently altered neuronal dynamics. These results show that loss of inhibitory tone can be compensated for by recovery mechanisms partly local to the perturbed nucleus and do not require circuits necessary for learning.

Mapping functional traces of opioid memories in the rat brain.

Addiction to psychoactive substances is a maladaptive learned behaviour. Contexts surrounding drug use integrate this aberrant mnemonic process and hold strong relapse-triggering ability. Here, we asked where context and salience might be concurrently represented in the brain during retrieval of drug-context paired associations. For this, we developed a morphine-conditioned place preference protocol that allows contextual stimuli presentation inside a magnetic resonance imaging scanner and investigated differences in activity and connectivity at context recall. We found context-specific responses to stimulus onset in multiple brain regions, namely, limbic, sensory and striatal. Differences in functional interconnectivity were found among amygdala, lateral habenula, and lateral septum. We also investigated alterations to resting-state functional connectivity and found increased centrality of the lateral septum in a proposed limbic network, as well as increased functional connectivity of the lateral habenula and hippocampal 'cornu ammonis' 1 region, after a protocol of associative drug-context. Finally, we found that pre- conditioned place preference resting-state connectivity of the lateral habenula and amygdala was predictive of inter-individual conditioned place preference score differences. Overall, our findings show that drug and saline-paired contexts establish distinct memory traces in overlapping functional brain microcircuits and that intrinsic connectivity of the habenula, septum, and amygdala likely underlies the individual maladaptive contextual learning to opioid exposure. We have identified functional maps of acquisition and retrieval of drug-related memory that may support the relapse-triggering ability of opioid-associated sensory and contextual cues. These findings may clarify the inter-individual sensitivity and vulnerability seen in addiction to opioids found in humans.

Demonstration-enhanced policy search for space multi-arm robot collaborative skill learning

The increasing complexity of on-orbit tasks imposes great demands on the flexible operation of space robotic arms, prompting the development of space robots from single-arm manipulation to multi-arm collaboration. In this paper, a combined approach of Learning from Demonstration (LfD) and Reinforcement Learning (RL) is proposed for space multi-arm collaborative skill learning. The combination effectively resolves the trade-off between learning efficiency and feasible solution in LfD, as well as the time-consuming pursuit of the optimal solution in RL. With the prior knowledge of LfD, space robotic arms can achieve efficient guided learning in high-dimensional state-action space. Specifically, an LfD approach with Probabilistic Movement Primitives (ProMP) is firstly utilized to encode and reproduce the demonstration actions, generating a distribution as the initialization of policy. Then in the RL stage, a Relative Entropy Policy Search (REPS) algorithm modified in continuous state-action space is employed for further policy improvement. More importantly, the learned behaviors can maintain and reflect the characteristics of demonstrations. In addition, a series of supplementary policy search mechanisms are designed to accelerate the exploration process. The effectiveness of the proposed method has been verified both theoretically and experimentally. Moreover, comparisons with state-of-the-art methods have confirmed the outperformance of the approach.

Population-level coding of avoidance learning in medial prefrontal cortex

The medial prefrontal cortex (mPFC) has been proposed to link sensory inputs and behavioral outputs to mediate the execution of learned behaviors. However, how such a link is implemented has remained unclear. To measure prefrontal neural correlates of sensory stimuli and learned behaviors, we performed population calcium imaging during a new tone-signaled active avoidance paradigm in mice. We developed an analysis approach based on dimensionality reduction and decoding that allowed us to identify interpretable task-related population activity patterns. While a large fraction of tone-evoked activity was not informative about behavior execution, we identified an activity pattern that was predictive of tone-induced avoidance actions and did not occur for spontaneous actions with similar motion kinematics. Moreover, this avoidance-specific activity differed between distinct avoidance actions learned in two consecutive tasks. Overall, our results are consistent with a model in which mPFC contributes to the selection of goal-directed actions by transforming sensory inputs into specific behavioral outputs through distributed population-level computations.

Viscerosensory signalling to the nucleus accumbens via the solitary tract nucleus.

The nucleus of the solitary tract (NTS) receives direct viscerosensory vagal afferent input that drives autonomic reflexes, neuroendocrine function and modulates behaviour. A subpopulation of NTS neurons project to the nucleus accumbens (NAc); however, the function of this NTS-NAc pathway remains unknown. A combination of neuroanatomical tracing, slice electrophysiology and fibre photometry was used in mice and/or rats to determine how NTS-NAc neurons fit within the viscerosensory network. NTS-NAc projection neurons are predominantly located in the medial and caudal portions of the NTS with 54 ± 7% (mice) and 65 ± 3% (rat) being TH-positive, representing the A2 NTS cell group. In horizontal brainstem slices, solitary tract (ST) stimulation evoked excitatory post-synaptic currents (EPSCs) in NTS-NAc projection neurons. The majority (75%) received low-jitter, zero-failure EPSCs characteristic of monosynaptic ST afferent input that identifies them as second order to primary sensory neurons. We then examined whether NTS-NAc neurons respond to cholecystokinin (CCK, 20 μg/kg ip) invivo in both mice and rats. Surprisingly, there was no difference in the number of activated NTS-NAc cells between CCK and saline-treated mice. In rats, just 6% of NTS-NAc cells were recruited by CCK. As NTS TH neurons are the primary source for NAc noradrenaline, we measured noradrenaline release in the NAc and showed that NAc noradrenaline levels declined in response to cue-induced reward retrieval but not foot shock. Combined, these findings suggest that high-fidelity afferent information from viscerosensory afferents reaches the NAc. These signals are likely unrelated to CCK-sensitive vagal afferents but could interact with other sensory and higher order inputs to modulate learned appetitive behaviours.

What factors affect teacher professional learning: a qualitative meta-analysis based on the Onion model

Purpose Current academic research on teacher learning is increasing in number and deepening in quality, but at the same time, the heterogeneity is growing significantly. Similar work is challenging to cross-check regarding conclusions due to the different research foci. This paper aims to provide a reliable theoretical framework and offers solid insights based on the existing research. Design/methodology/approach Based on the Onion model, 67 core literature in English and Chinese were coded through qualitative meta-analysis methods to explore environmental, behavioral, competence, belief, identity, mission and other factors that may impact teacher learning. Findings It was found that the quantitative structure of the current relevant studies was in an inverted triangular shape with three levels of steps, respectively, which can be summarized as structural environment, core behaviors and dominant mission. The heterogeneity between the findings mainly originated from two situations, oppositional and complementary and some structural adjustments were made to the Onion model according to the coding results to better represent the interaction of influences between the levels. It also analyzes current research trends and the centrality of learned behaviors based on the coding results. Social implications The design of teacher learning activities should combine theoretical, practical and inquiry learning to ensure that teachers are kept fresh and motivated by sustained and varied stimuli. Originality/value To the best of the authors’ knowledge, this study is the first to analyze teacher learning influences through qualitative meta-analysis and create node saturation to analyze the results, resulting in highly credible and valuable research findings.

To Cache or Not to Cache

Unlike conventional CPU caches, non-datapath caches, such as host-side flash caches which are extensively used as storage caches, have distinct requirements. While every cache miss results in a cache update in a conventional cache, non-datapath caches allow for the flexibility of selective caching, i.e., the option of not having to update the cache on each miss. We propose a new, generalized, bimodal caching algorithm, Fear Of Missing Out (FOMO), for managing non-datapath caches. Being generalized has the benefit of allowing any datapath cache replacement policy, such as LRU, ARC, or LIRS, to be augmented by FOMO to make these datapath caching algorithms better suited for non-datapath caches. Operating in two states, FOMO is selective—it selectively disables cache insertion and replacement depending on the learned behavior of the workload. FOMO is lightweight and tracks inexpensive metrics in order to identify these workload behaviors effectively. FOMO is evaluated using three different cache replacement policies against the current state-of-the-art non-datapath caching algorithms, using five different storage system workload repositories (totaling 176 workloads) for six different cache size configurations, each sized as a percentage of each workload’s footprint. Our extensive experimental analysis reveals that FOMO can improve upon other non-datapath caching algorithms across a range of production storage workloads, while also reducing the write rate.

Perirhinal cortex learns a predictive map of the task environment

Goal-directed tasks involve acquiring an internal model, known as a predictive map, of relevant stimuli and associated outcomes to guide behavior. Here, we identified neural signatures of a predictive map of task behavior in perirhinal cortex (Prh). Mice learned to perform a tactile working memory task by classifying sequential whisker stimuli over multiple training stages. Chronic two-photon calcium imaging, population analysis, and computational modeling revealed that Prh encodes stimulus features as sensory prediction errors. Prh forms stable stimulus-outcome associations that can progressively be decoded earlier in the trial as training advances and that generalize as animals learn new contingencies. Stimulus-outcome associations are linked to prospective network activity encoding possible expected outcomes. This link is mediated by cholinergic signaling to guide task performance, demonstrated by acetylcholine imaging and systemic pharmacological perturbation. We propose that Prh combines error-driven and map-like properties to acquire a predictive map of learned task behavior.

Modelling two-dimensional driving behaviours at unsignalised intersection using multi-agent imitation learning

Traffic at unsignalised intersection usually involves complex interactions. It is critical to explicitly model the two-dimensional driving behaviours and capture the interactions among vehicles. However, little effort has been made to develop microscopic traffic simulation models at unsignalised intersections, which can generate realistic vehicle trajectories. To fill this gap, this study aims to develop a two-dimensional data-driven simulation model at an unsignalised intersection based on multi-agent imitation learning for generating realistic trajectories of vehicles crossing the intersection and macroscopic traffic characteristics. We propose a multi-agent adversarial inverse reinforcement learning model with four policies (MA-AIRL-4) to separately learn the driving behaviours with different directions (East-bound, West-bound, South-bound, and North–bound) by capturing the interactions between the vehicles. Using the vehicle trajectories data extracted from an unsignalised intersection of the open-source Interaction Dataset, we evaluate the performance of the proposed model by comparing it with several bench-marking models, i.e., MA-AIRL-2 model with two policies (i.e., one policy for West-East and North-South direction, respectively), MA-AIRL-1 model with one policy for all vehicles, three corresponding multi-agent generative adversarial imitation learning (MA-GAIL) models with four policies, two policies and one policy, respectively, and long short-term memory (LSTM) model. The results demonstrate the superiority of the proposed MA-AIRL-4 model in generating accurate vehicle trajectories and reproducing realistic speed distributions. The interpretation of the recovered reward function also indicates the effective learning of the proposed model and provides insights into the learned behaviours.

Rats Lacking the Dopamine Transporter Display Inflexibility in Innate and Learned Behavior.

Playing a key role in the organization of striatal motor output, the dopamine (DA)-ergic system regulates both innate and complex learned behaviors. Growing evidence clearly indicates the involvement of the DA-ergic system in different forms of repetitive (perseverative) behavior. Some of these behaviors accompany such disorders as obsessive-compulsive disorder (OCD), Tourette's syndrome, schizophrenia, and addiction. In this study, we have traced how the inflexibility of repetitive reactions in the recently developed animal model of hyper-DA-ergia, dopamine transporter knockout rats (DAT-KO rats), affects the realization of innate behavior (grooming) and the learning of spatial (learning and reversal learning in T-maze) and non-spatial (extinction of operant reaction) tasks. We found that the microstructure of grooming in DAT-KO rats significantly differed in comparison to control rats. DAT-KO rats more often demonstrated a fixed syntactic chain, making fewer errors and very rarely missing the chain steps in comparison to control rats. DAT-KO rats' behavior during inter-grooming intervals was completely different to the control animals. During learning and reversal learning in the T-maze, DAT-KO rats displayed pronounced patterns of hyperactivity and perseverative (stereotypical) activity, which led to worse learning and a worse performance of the task. Most of the DAT-KO rats could not properly learn the behavioral task in question. During re-learning, DAT-KO rats demonstrated rigid perseverative activity even in the absence of any reinforcement. In operant tasks, the mutant rats demonstrated poor extinction of operant lever pressing: they continued to perform lever presses despite no there being reinforcement. Our results suggest that abnormally elevated DA levels may be responsible for behavioral rigidity. It is conceivable that this phenomenon in DAT-KO rats reflects some of the behavioral traits observed in clinical conditions associated with endogenous or exogenous hyper-DA-ergia, such as schizophrenia, substance abuse, OCD, patients with Parkinson disease treated with DA mimetics, etc. Thus, DAT-KO rats may be a valuable behavioral model in the search for new pharmacological approaches to treat such illnesses.

A field experiment on ISP training designs for enhancing employee information security compliance

ABSTRACT Information security policy (ISP) training plays an important role in enhancing organisational resilience against cyber threats by providing employees with the necessary knowledge and skills to effectively identify, prevent, and respond to security breaches. This research aims to explore how the use of deterrence arguments and threat arguments can enhance the effectiveness of ISP training. We theorise how ISP training affects employees’ ISP compliance behaviour by arguing for a transfer of training lens to study the effectiveness of ISP training. The results of our field experiment with triangulated data suggest that the effect of argumentative-enhanced ISP training is twofold. First, employees who participated in enhanced training sessions with deterrence and threat arguments demonstrated superior training outputs after the training, which, in turn, translated into a sustained training outcome three weeks after the training. Second, we also find evidence that threat arguments can reinforce the application of training outputs in the maintenance stage of learned behaviours. With this applied research study, we contribute to the research and practice by providing empirical evidence of the effectiveness of ISP training designs.