Bimodal Processing Research Articles

Orthographic influences on spoken word recognition in bilinguals are dependent on the orthographic depth of the target language not the native language

Reading acquisition leads to the restructuring of representational units in the brain, which influences spoken word processing. This makes spoken word recognition a bimodal process. However, the organization of phonological and orthographic units is dependent on the orthographic depth of the writing system and might play a role in the bimodal processing of spoken words. We investigated this question across two EEG experiments with German native speakers using an auditory priming paradigm and manipulating phonological (e.g., Reh – Tee) and orthographic (e.g., See – Tee) overlap between prime and target. Experiment 1 was conducted in German and revealed inhibitory effects for orthographic overlap, but facilitating effects for phonological overlap. Experiment 2 was conducted in English and revealed facilitating effects for orthographic and phonological overlap. We conclude that orthography influences spoken word processing in both languages, but the nature of the influence is dependent on the orthographic depth of the target language.

Temporal Encoding During Unimodal and Bimodal Odor Processing in the Human Brain

In daily life, people encounter a wide range of odors, most of which contain multiple chemical substances. So-called bimodal odors stimulate both the olfactory and trigeminal nerve, and the interaction between these two systems shapes the perception of the odor. However, temporal encoding of these sensory systems during bimodal odor processing has received limited scientific attention. To investigate this, we recorded the electrophysiological response in 17 participants to relatively unimodal olfactory (strawberry), trigeminal (l-isopulegol) and strongly bimodal (strawberry and l-isopulegol) stimuli. ERP amplitudes and intensity ratings were significantly bigger for bimodal stimulation, as compared to unimodal stimulations. No significant difference was observed between N1 and P2 response latencies to olfactory and bimodal stimuli while responses to both stimuli showed longer latencies compared to the response of the trigeminal stimulus. Results provide further evidence of interaction between olfactory and trigeminal systems; additional activation of the olfactory system results in more vigorous electrophysiological responses and the experience of higher intensity. In addition, results indicate that the trigeminal system is faster to react to stimulation. The current study offers a view on the temporal processing of bimodal odorants that are most likely to be encountered in the environment. We link the more vigorous electrophysiological response to a more complex odorant with activation of different neural structures.

Children with dyslexia show a reduced processing benefit from bimodal speech information compared to their typically developing peers

During information processing, individuals benefit from bimodally presented input, as has been demonstrated for speech perception (i.e., printed letters and speech sounds) or the perception of emotional expressions (i.e., facial expression and voice tuning). While typically developing individuals show this bimodal benefit, school children with dyslexia do not. Currently, it is unknown whether the bimodal processing deficit in dyslexia also occurs for visual-auditory speech processing that is independent of reading and spelling acquisition (i.e., no letter-sound knowledge is required). Here, we tested school children with and without spelling problems on their bimodal perception of video-recorded mouth movements pronouncing syllables. We analyzed the event-related potential Mismatch Response (MMR) to visual-auditory speech information and compared this response to the MMR to monomodal speech information (i.e., auditory-only, visual-only). We found a reduced MMR with later onset to visual-auditory speech information in children with spelling problems compared to children without spelling problems. Moreover, when comparing bimodal and monomodal speech perception, we found that children without spelling problems showed significantly larger responses in the visual-auditory experiment compared to the visual-only response, whereas children with spelling problems did not. Our results suggest that children with dyslexia exhibit general difficulties in bimodal speech perception independently of letter-speech sound knowledge, as apparent in altered bimodal speech perception and lacking benefit from bimodal information. This general deficit in children with dyslexia may underlie the previously reported reduced bimodal benefit for letter-speech sound combinations and similar findings in emotion perception.

Cognitive mechanisms underlying the associations between inattention and reading abilities

ABSTRACTThis study aims to test cognitive skills underlying the association between inattention and reading in early primary school. Teachers rated inattention symptoms when children (N = 523–962) were 6–7 years old. Children were assessed at age 7–8 on phonological awareness, rapid automatized naming (RAN), rapid auditory and bimodal processing, vocabulary, and reading (decoding and comprehension). Phonological awareness, RAN of numbers, and vocabulary mediated the association between inattention and both decoding and comprehension. Rapid bimodal processing mediated the association between inattention and decoding, while RAN of colors mediated the association between inattention and comprehension. This study highlights mediators underlying inattention-reading associations.

Bimodal odor processing with a trigeminal component at sub- and suprathreshold levels

Odors are typically bimodal in nature, interacting with the olfactory and trigeminal systems. The trigeminal component may be noticed (e.g. menthol) or perceptually ignored, leading to different neural substrates being recruited during odor encoding. Therefore, the current study was designed to explore the perceptual and central-nervous activations in response to pleasant bimodal odors using functional magnetic resonance imaging (fMRI). In this study, healthy subjects were exposed to odorants alone (unimodal) or with a “cooling” trigeminal component (bimodal) at sub- and suprathreshold concentrations with a portable olfactometer in a 3T fMRI scanner. Within the scanner, subjects reported all odorants as pleasant and intensity increasing with trigeminal concentration. Many of the regions of interest [orbital frontal cortex (OFC), insula, thalamus, cerebellum, postcentral gyrus and cingulate cortex] were activated during bimodal odor conditions when contrasted with unimodal, and interestingly, most of these activations were seen prior to trigeminal perception (e.g. at a sub-threshold level). This includes large bilateral activations within the OFC, insula, cerebellum and parts of the cingulate cortex. Additionally, activation of the thalamus was seen early in the stages of bimodal odor encoding suggesting its role of mediating attention toward the presence of two stimuli. Lastly, intensity encoding during bimodal processing shows overlap of previously demonstrated simple trigeminal encoding areas (medial cingulate cortex) and the more complex olfactory encoding areas (bilateral insula, superior temporal gyrus, OFC, and cerebellum), but not the amygdala.

Bimodal processing of olfactory information in an amphibian nose: odor responses segregate into a medial and a lateral stream

In contrast to the single sensory surface present in teleost fishes, several spatially segregated subsystems with distinct molecular and functional characteristics define the mammalian olfactory system. However, the evolutionary steps of that transition remain unknown. Here we analyzed the olfactory system of an early diverging tetrapod, the amphibian Xenopus laevis, and report for the first time the existence of two odor-processing streams, sharply segregated in the main olfactory bulb and partially segregated in the olfactory epithelium of pre-metamorphic larvae. A lateral odor-processing stream is formed by microvillous receptor neurons and is characterized by amino acid responses and Gαo/Gαi as probable signal transducers, whereas a medial stream formed by ciliated receptor neurons is characterized by responses to alcohols, aldehydes, and ketones, and Gαolf/cAMP as probable signal transducers. To reveal candidates for the olfactory receptors underlying these two streams, the spatial distribution of 12 genes from four olfactory receptor gene families was determined. Several class II and some class I odorant receptors (ORs) mimic the spatial distribution observed for the medial stream, whereas a trace amine-associated receptor closely parallels the spatial pattern of the lateral odor-processing stream. Other olfactory receptors (some class I odorant receptors and vomeronasal type 1 receptors) and odor responses (to bile acids, amines) were not lateralized, the latter not even in the olfactory bulb, suggesting an incomplete segregation. Thus, the olfactory system of X. laevis exhibits an intermediate stage of segregation and as such appears well suited to investigate the molecular driving forces behind olfactory regionalization.

Multi-sensory integration in brainstem and auditory cortex

Tinnitus is the perception of sound in the absence of a physical sound stimulus. It is thought to arise from aberrant neural activity within central auditory pathways that may be influenced by multiple brain centers, including the somatosensory system. Auditory-somatosensory (bimodal) integration occurs in the dorsal cochlear nucleus (DCN), where electrical activation of somatosensory regions alters pyramidal cell spike timing and rates of sound stimuli. Moreover, in conditions of tinnitus, bimodal integration in DCN is enhanced, producing greater spontaneous and sound-driven neural activity, which are neural correlates of tinnitus. In primary auditory cortex (A1), a similar auditory-somatosensory integration has been described in the normal system (Lakatos et al., 2007), where sub-threshold multisensory modulation may be a direct reflection of subcortical multisensory responses (Tyll et al., 2011). The present work utilized simultaneous recordings from both DCN and A1 to directly compare bimodal integration across these separate brain stations of the intact auditory pathway. Four-shank, 32-channel electrodes were placed in DCN and A1 to simultaneously record tone-evoked unit activity in the presence and absence of spinal trigeminal nucleus (Sp5) electrical activation. Bimodal stimulation led to long-lasting facilitation or suppression of single and multi-unit responses to subsequent sound in both DCN and A1. Immediate (bimodal response) and long-lasting (bimodal plasticity) effects of Sp5-tone stimulation were facilitation or suppression of tone-evoked firing rates in DCN and A1 at all Sp5-tone pairing intervals (10, 20, and 40 ms), and greater suppression at 20 ms pairing-intervals for single unit responses. Understanding the complex relationships between DCN and A1 bimodal processing in the normal animal provides the basis for studying its disruption in hearing loss and tinnitus models. This article is part of a Special Issue entitled: Tinnitus Neuroscience.

Auditory influences on non-auditory cortices

Although responses to auditory stimuli have been extensively examined in the well-known regions of auditory cortex, there are numerous reports of acoustic sensitivity in cortical areas that are dominated by other sensory modalities. Whether in ‘polysensory’ cortex or in visual or somatosensory regions, auditory responses in non-auditory cortex have been described largely in terms of auditory processing. This review takes a different perspective that auditory responses in non-auditory cortex, either through multisensory subthreshold or bimodal processing, provide subtle but consistent expansion of the range of activity of the dominant modality within a given area. Thus, the features of these acoustic responses may have more to do with the subtle adjustment of response gain within a given non-auditory region than the encoding of their tonal properties.

Modality and variability as factors in training the elderly

AbstractCognitive load theory (CLT) predicts that bimodal processing of instructional material decreases the level of extraneous cognitive load, whereas increased training variability increases the level of germane cognitive load. It was hypothesized that a combination of these strategies leads to optimal learning, especially in older adults. Forty young and forty elderly learners were trained to solve complex problems. The results showed that bimodal training leads to lower cognitive load than unimodal training. Furthermore, random presentation of examples (high variability) led to higher performance than blocked presentation (low variability) in both age groups. However, there was no combined effect of modality and variability. Moreover, the elderly did not take disproportionate advantage of the bimodal and random conditions. It was concluded that these training methods bear important potential in the light of lifelong learning. Copyright © 2006 John Wiley & Sons, Ltd.

A Fused Hidden Markov Model With Application to Bimodal Speech Processing

This paper presents a novel fused hidden Markov model (fused HMM) for integrating tightly coupled time series, such as audio and visual features of speech. In this model, the time series are first modeled by two conventional HMMs separately. The resulting HMMs are then fused together using a probabilistic fusion model, which is optimal according to the maximum entropy principle and a maximum mutual information criterion. Simulations and bimodal speaker verification experiments show that the proposed model can significantly reduce the recognition errors in noiseless or noisy environments.

Early auditory–visual interactions in human cortex during nonredundant target identification

A common finding of behavioral studies is that objects characterized by redundant multisensory cues are identified more rapidly than the same objects presented in either unimodal condition. In a previous electrophysiological study in humans, we have described a network of crossmodal interactions that could be associated with this facilitation effect [M.H. Giard, F. Peronnet, J. Cogn. Neurosci. 11(5) (1999) 473–490]. Here, we sought to determine whether the recognition of objects characterized by nonredundant bimodal components may still induce crossmodal neural interactions. Subjects had to identify three objects defined either by auditory or visual features alone, or by the combination of nonredundant auditory and visual features. As expected, behavioral measures showed no sign of facilitation in bimodal processing. Yet, event-related potential analysis revealed the existence of early (<200 ms latency) crossmodal activities in sensory-specific and nonspecific cortical areas, that were partly dependent on the sensory dominance of the subjects to perform the task. Comparative analysis of the interaction patterns involved in redundant and nonredundant cue processing provides evidence for the robustness of the principle of crossmodal neural synergy that applies whatever the stimulus content (redundant or nonredundant information), and for the high flexibility of the neural networks of integration that are sensitive both to the nature of the perceptual task and to the sensory skill of the individual in that particular task.

Bidirectional grapheme-phoneme activation in a bimodal detection task.

A divided attention paradigm was used to investigate whether graphemes and phonemes can mutually activate or inhibit each other during bimodal processing. In 3 experiments, Dutch subjects reacted to visual and auditory targets in single-channel or bimodal stimuli. In some bimodal conditions, the visual and auditory targets were nominally identical or redundant (e.g., visual A and auditory /a/); in others they were not (e.g., visual U and auditory /a/). Temporal aspects of cross-modal activation were examined by varying the stimulus onset asynchrony of visual and auditory stimuli. Cross-modal facilitation--but not inhibition--occurred rapidly and automatically between phoneme and grapheme representations. Implications for current models of bimodal processing and word recognition are discussed.

Bidirectional grapheme-phoneme activation in a bimodal detection task.

A divided attention paradigm was used to investigate whether graphemes and phonemes can mutually activate or inhibit each other during bimodal processing. In 3 experiments, Dutch subjects reacted to visual and auditory targets in single-channel or bimodal stimuli. In some bimodal conditions, the visual and auditory targets were nominally identical or redundant (e.g., visual A and auditory /a/); in others they were not (e.g., visual U and auditory /a/). Temporal aspects of cross-modal activation were examined by varying the stimulus onset asynchrony of visual and auditory stimuli. Cross-modal facilitation--but not inhibition--occurred rapidly and automatically between phoneme and grapheme representations. Implications for current models of bimodal processing and word recognition are discussed.

Tussen Lezen En Luisteren

Two divided attention experiments investigated whether graphemes and phonemes can mutually activate each other during bimodal sublexical processing. Dutch subjects reacted to target letters and/or speech sounds in single-channel and bimodal stimuli. In some bimodal conditions, the visual and auditory targets were congruent (e.g., visual A, auditory /a:/), in others they were not (e.g., visual U, auditory /a:/). Temporal aspects of cross-modal activation were examined by varying the stimulus onset asynchrony (SOA) of visual and auditory stimulus components. Processing differences among stimuli (e.g., the letters A and U) were accounted for by correcting the obtained bimodal reaction times by means of the predictions of an independent race-model. Comparing the results of the adapted congruent and incongruent conditions for each SOA, it can be concluded that (a) cross-modal activation takes place in this task situation; (b) it is bidirectional, i.e. it spreads from grapheme to phoneme and vice versa; and (c) it occurs very rapidly.

Minimizing Makespan with Bimodal Processing Time Distributions

We consider the problem of scheduling n tasks on two identical parallel processors. Task i has a processing time of one time unit, but might have to undergo processing for a second time unit with probability pi, i.e., the processing time distributions of the tasks have mass only on one and on two. We are interested in the sequence in which to process these tasks in order to minimize the expected completion time of all tasks. The optimal sequence turns out to be a sequence not well-known in the theory of scheduling.