Cross-modal Associative Learning Research Articles

Visual–verbal paired-associate learning: An investigation of the role of verbal and crossmodal associative learning in reading skills in French first- and second-grade children.

Visual–verbal paired-associate learning: An investigation of the role of verbal and crossmodal associative learning in reading skills in French first- and second-grade children.

Supplemental Material for Visual–Verbal Paired-Associate Learning: An Investigation of the Role of Verbal and Crossmodal Associative Learning in Reading Skills in French First- and Second-Grade Children

Supplemental Material for Visual–Verbal Paired-Associate Learning: An Investigation of the Role of Verbal and Crossmodal Associative Learning in Reading Skills in French First- and Second-Grade Children

Representational Similarity Analysis for Tracking Neural Correlates of Haptic Learning on a Multimodal Device.

A goal of wearable haptic devices has been to enable haptic communication, where individuals learn to map information typically processed visually or aurally to haptic cues via a process of cross-modal associative learning. Neural correlates have been used to evaluate haptic perception and may provide a more objective approach to assess association performance than more commonly used behavioral measures of performance. In this article, we examine Representational Similarity Analysis (RSA) of electroencephalography (EEG) as a framework to evaluate how the neural representation of multifeatured haptic cues changes with association training. We focus on the first phase of cross-modal associative learning, perception of multimodal cues. A participant learned to map phonemes to multimodal haptic cues, and EEG data were acquired before and after training to create neural representational spaces that were compared to theoretical models. Our perceptual model showed better correlations to the neural representational space before training, while the feature-based model showed better correlations with the post-training data. These results suggest that training may lead to a sharpening of the sensory response to haptic cues. Our results show promise that an EEG-RSA approach can capture a shift in the representational space of cues, as a means to track haptic learning.

Gender congruence and emotion effects in cross-modal associative learning: Insights from ERPs and pupillary responses.

Social and emotional cues from faces and voices are highly relevant and have been reliably demonstrated to attract attention involuntarily. However, there are mixed findings as to which degree associating emotional valence to faces occurs automatically. In the present study, we tested whether inherently neutral faces gain additional relevance by being conditioned with either positive, negative, or neutral vocal affect bursts. During learning, participants performed a gender-matching task on face-voice pairs without explicit emotion judgments of the voices. In the test session on a subsequent day, only the previously associated faces were presented and had to be categorized regarding gender. We analyzed event-related potentials (ERPs), pupil diameter, and response times (RTs) of N = 32 subjects. Emotion effects were found in auditory ERPs and RTs during the learning session, suggesting that task-irrelevant emotion was automatically processed. However, ERPs time-locked to the conditioned faces were mainly modulated by the task-relevant information, that is, the gender congruence of the face and voice, but not by emotion. Importantly, these ERP and RT effects of learned congruence were not limited to learning but extended to the test session, that is, after removing the auditory stimuli. These findings indicate successful associative learning in our paradigm, but it did not extend to the task-irrelevant dimension of emotional relevance. Therefore, cross-modal associations of emotional relevance may not be completely automatic, even though the emotion was processed in the voice.

The Associations of Emotion Coping Appraisal With Both the Cue-Outcome Contingency and Perceived Verbal Abuse Exposure.

Previous studies have reported an association between verbal abuse in early childhood and structural and functional alterations in the young adult brain, supporting the existence of critical periods in human brain development. In addition, exposure to verbal abuse in early childhood is strongly correlated with lifetime psychiatric illness. Resilience is defined as the ability to avoid the negative psychological, biological, and social consequences of stress that impair psychological and physical homeostasis and is used to cope with these psychiatric diseases. We attempted to explain the mediatable present function of resilience and its associations with several psychiatric disorders, with verbal abuse exposure in early childhood and with the present value of the readily measurable and conceptually connected generative Bayesian model parameter. Thirty-six subjects performed a cross-modal associative learning task requiring them to learn the predictive strength of auditory cues and predict a subsequent visual stimulus. The probability of the association changed across each trial block. Subjects’ responses were modeled as a hierarchical Bayesian belief-updating process using a hierarchical Gaussian filter (HGF) with three levels, a Sutton K1 model, and a Rescorla–Wagner model. Subjects completed the Korean version of the Verbal Abuse Questionnaire (VAQ) for segmented periods (aged 0 to 6, 7 to 12, and 13 to 18 years), and their positive self-appraisal was estimated using the Resilience Appraisal Scale (RAS). Random-effects Bayesian model selection identified HGF as the best model. Of the VAQ values for specific periods, only preschool VAQ scores were negatively correlated with RAS scores. The tonic volatility parameter, ω2, of HGF showed a negative relationship with RAS emotion coping scores. The linear regression model explained 18.3% of the variance of emotion coping appraisal with ω2 and preschool VAQ scores. Based on the results obtained from young adults, decrease in emotion coping appraisal can be explained by an increase in the number of experiences of verbal abuse in early childhood and the increased tendency to update beliefs about the cue–outcome associative probability in a volatile environment.

Neural processing of odor-associated words: an fMRI study in patients with acquired olfactory loss.

Perception of olfactory information is mediated by both bottom-up (from molecules to perception) and top-down (e.g. cross-modal associative learning) processes. Acquired olfactory loss is a frequent disorder which is typically due to alterations in the bottom-up pathway. However, it is unclear how the top-down modulation of olfactory processing is affected by olfactory impairment. Our study aimed to investigate the top-down olfactory processing in patients with acquired olfactory loss and participants with normal olfaction. Using functional MRI, brain responses from 14 patients and 16 healthy controls were assessed during a task of expectation and reading of words with strong olfactory associations (OW) (e.g. "Rose") and control words with little to no olfactory associations (CW) (e.g. "Door"). The expectation but not reading of the OW was associated with stronger neural activation in the angular gyrus and the inferior frontal gyrus extending to insula in the group of patients. During OW reading, the brain activation in the left OFC and right putamen was negatively correlated with odor identification score in patient and control groups, respectively. In addition, the duration of olfactory loss among patients was positively associated with activation in the left putamen during OW expectation. Taken together, these findings suggest an enhanced top-down olfactory modulation in patients with olfactory loss.

The Neural Correlates of Hierarchical Predictions for Perceptual Decisions.

Sensory information is inherently noisy, sparse, and ambiguous. In contrast, visual experience is usually clear, detailed, and stable. Bayesian theories of perception resolve this discrepancy by assuming that prior knowledge about the causes underlying sensory stimulation actively shapes perceptual decisions. The CNS is believed to entertain a generative model aligned to dynamic changes in the hierarchical states of our volatile sensory environment. Here, we used model-based fMRI to study the neural correlates of the dynamic updating of hierarchically structured predictions in male and female human observers. We devised a crossmodal associative learning task with covertly interspersed ambiguous trials in which participants engaged in hierarchical learning based on changing contingencies between auditory cues and visual targets. By inverting a Bayesian model of perceptual inference, we estimated individual hierarchical predictions, which significantly biased perceptual decisions under ambiguity. Although "high-level" predictions about the cue-target contingency correlated with activity in supramodal regions such as orbitofrontal cortex and hippocampus, dynamic "low-level" predictions about the conditional target probabilities were associated with activity in retinotopic visual cortex. Our results suggest that our CNS updates distinct representations of hierarchical predictions that continuously affect perceptual decisions in a dynamically changing environment.SIGNIFICANCE STATEMENT Bayesian theories posit that our brain entertains a generative model to provide hierarchical predictions regarding the causes of sensory information. Here, we use behavioral modeling and fMRI to study the neural underpinnings of such hierarchical predictions. We show that "high-level" predictions about the strength of dynamic cue-target contingencies during crossmodal associative learning correlate with activity in orbitofrontal cortex and the hippocampus, whereas "low-level" conditional target probabilities were reflected in retinotopic visual cortex. Our findings empirically corroborate theorizations on the role of hierarchical predictions in visual perception and contribute substantially to a longstanding debate on the link between sensory predictions and orbitofrontal or hippocampal activity. Our work fundamentally advances the mechanistic understanding of perceptual inference in the human brain.

An Investigation of the Relationships Between Autistic Traits and Crossmodal Correspondences in Typically Developing Adults

Autism spectrum disorder (ASD) includes characteristics such as social and behavioral deficits that are considered common across the general population rather than unique to people with the diagnosis. People with ASD are reported to have sensory irregularities, including crossmodal perception. Crossmodal correspondences are phenomena in which arbitrary crossmodal inputs affect behavioral performance. Crossmodal correspondences are considered to be established through associative learning, but the learning cues are considered to differ across the types of correspondences. In order to investigate whether and how ASD traits affect crossmodal associative learning, this study examined the relationships between the magnitude of crossmodal correspondences and the degree of ASD traits among non-diagnosed adults. We found that, among three types of crossmodal correspondences (brightness-loudness, visual size-pitch, and visual location-pitch pairs), the brightness-loudness pair was related with total ASD traits and a subtrait (social skill). The magnitude of newly learned crossmodal associations (the visual apparent motion direction-pitch pair) also showed a relationship with an ASD subtrait (attention switching). These findings demonstrate that there are unique relationships between crossmodal associations and ASD traits, indicating that each ASD trait is differently involved in sensory associative learning.

Associative Memory Extinction Is Accompanied by Decayed Plasticity at Motor Cortical Neurons and Persistent Plasticity at Sensory Cortical Neurons.

Associative memory is essential for cognition, in which associative memory cells and their plasticity presumably play important roles. The mechanism underlying associative memory extinction vs. maintenance remains unclear, which we have studied in a mouse model of cross-modal associative learning. Paired whisker and olfaction stimulations lead to a full establishment of odorant-induced whisker motion in training day 10, which almost disappears if paired stimulations are not given in a week, and then recovers after paired stimulation for an additional day. In mice that show associative memory, extinction and recovery, we have analyzed the dynamical plasticity of glutamatergic neurons in layers II–III of the barrel cortex and layers IV–V of the motor cortex. Compared with control mice, the rate of evoked spikes as well as the amplitude and frequency of excitatory postsynaptic currents increase, whereas the amplitude and frequency of inhibitory postsynaptic currents (IPSC) decrease at training day 10 in associative memory mice. Without paired training for a week, these plastic changes are persistent in the barrel cortex and decayed in the motor cortex. If paired training is given for an additional day to revoke associative memory, neuronal plasticity recovers in the motor cortex. Our study indicates persistent neuronal plasticity in the barrel cortex for cross-modal memory maintenance as well as the dynamical change of neuronal plasticity in the motor cortex for memory retrieval and extinction. In other words, the sensory cortices are essential for long-term memory while the behavior-related cortices with the inability of memory retrieval are correlated to memory extinction.

Examining the direct and indirect effects of visual–verbal paired associate learning on Chinese word reading

Associative learning, traditionally measured with paired associate learning (PAL) tasks, has been found to predict reading ability in several languages. However, it remains unclear whether it also predicts word reading in Chinese, which is known for its ambiguous print–sound correspondences, and whether its effects are direct or indirect through the effects of other reading-related skills such as phonological awareness and rapid naming. Thus, the purpose of this study was to examine the direct and indirect effects of visual–verbal PAL on word reading in an unselected sample of Chinese children followed from the second to the third kindergarten year. A sample of 141 second-year kindergarten children (71 girls and 70 boys; mean age=58.99months, SD=3.17) were followed for a year and were assessed at both times on measures of visual–verbal PAL, rapid naming, and phonological awareness. In the third kindergarten year, they were also assessed on word reading. The results of path analysis showed that visual–verbal PAL exerted a significant direct effect on word reading that was independent of the effects of phonological awareness and rapid naming. However, it also exerted significant indirect effects through phonological awareness. Taken together, these findings suggest that variations in cross-modal associative learning (as measured by visual–verbal PAL) place constraints on the development of word recognition skills irrespective of the characteristics of the orthography children are learning to read.

Dissociating crossmodal and verbal demands in paired associate learning (PAL): What drives the PAL–reading relationship?

Recent research suggests that visual–verbal paired associate learning (PAL) may tap a crossmodal associative learning mechanism that plays a distinct role in reading development. However, evidence from children with dyslexia indicates that deficits in visual–verbal PAL are strongly linked to the verbal demands of the task. The primary aim of this study was to disassociate the role of modality and verbal demand in driving the PAL–reading relationship. To do so, we compared performance across four PAL mapping conditions: visual–verbal, verbal–verbal, visual–visual and verbal–visual. We reasoned that if crossmodal mapping demand accounts for the PAL–reading relationship, both visual–verbal PAL and verbal–visual PAL should exhibit significant relationships with reading ability. The results were incompatible with the crossmodal hypothesis. Only tasks requiring verbal output (visual–verbal PAL and verbal–verbal PAL) significantly correlated with reading ability. In addition, visual–verbal PAL and verbal–verbal PAL were well represented by a latent “verbal output PAL” factor. Structural equation modeling showed that this factor fully accounted for the PAL–reading relationship; visual–verbal PAL did not add anything to the prediction of reading above and beyond this latent factor. The results are interpreted according to an alternative verbal account of the PAL–reading relationship.

Crossmodal interactions in perception and learning

Humans are generally considered as visual animals. Visual perception, however, can be strongly and radically altered by other modalities. We have found that sound can radically change visual perception, and our neuroimaging studies show that this alteration can occur at short latencies and as early as primary visual cortex. These findings together with a wealth of other recent findings have established that crossmodal interactions are ubiquitous in human perception and can occur at various levels of processing, including very early stages of sensory processing. Therefore, we asked whether crossmodal interactions play a role in perceptual learning. We compared visual training with auditory‐visual training for learning of a low‐level visual task, and discovered that auditory‐visual training accelerates and enhances performance in a visual task even in the absence of sound. Thus, it appears that crossmodal interactions play an important role in perceptual learning, but how are the crossmodal associations acquired themselves? Brain areas mediating multisensory learning, and characteristics of crossmodal associative learning and its relationship with unisensory associative learning will be discussed.

Appetitive and Aversive Olfactory Learning in Humans Studied Using Event-Related Functional Magnetic Resonance Imaging

We combined event-related functional magnetic resonance imaging (fMRI) with olfactory classical conditioning to differentiate the neural responses evoked during appetitive and aversive olfactory learning. Three neutral faces [the conditioned stimuli (CS+)] were repetitively paired with pleasant, neutral, or unpleasant odors [the unconditioned stimuli (UCS)] in a partial reinforcement schedule. A fourth face was never paired to odor [the nonconditioned stimulus (CS-)]. Learning-related neural activity, comparing unpaired (face only) CS+ stimuli with CS-, showed valence-independent activations in rostral and caudal orbitofrontal cortex (OFC). Medial OFC responded to the appetitive (app) CS+, whereas lateral OFC responded to the aversive (av) CS+. Within nucleus accumbens, neural responses showed divergent activation profiles that increased with time in response to the appCS+ but decreased in response to the avCS+. In posterior amygdala, responses were elicited by the appCS+, which habituated over time. In temporal piriform cortex, neural responses were evoked by the avCS+, which progressively increased with time. These results highlight regional and temporal dissociations during olfactory learning and imply that emotionally salient odors can engender cross-modal associative learning. Moreover, the findings suggest that the role of human primary (piriform) and secondary olfactory cortices transcends their function as mere intermediaries of chemosensory information processing.