Auditory Dominance Research Articles

Implicit semantics gates visual awareness

The current study asked whether impoverished peripheral vision led to perception immune from word-based semantic influences. We leveraged a peripheral sound-induced flash illusion. In each trial, two or three Mandarin characters were flashed quickly in the periphery with number-congruent or -incongruent beeps. We first successfully replicated the original illusions, showing auditory dominance. For example, when three characters were presented together with two beeps, the observer reported perceiving only two characters. Similarly, an additional beep induced an illusory visual percept. Crucially, when the three characters formed a meaningful word, the lack of a concurrent beep suppressed the awareness to a greater extent. A separate experiment replicated the effect on participants who were unable to recognize the words. When the reading was disrupted by reversing the presentation order, the effect disappeared. These findings demonstrate the capacity of our visual system to extract peripheral linguistic information without conscious word recognition.

Increased body movement equals better performance? Not always! Musical style determines motion degree perceived as optimal in music performance

Musicians’ body behaviour has a preponderant role in audience perception. We investigated how performers’ motion is perceived depending on the musical style and musical expertise. To further explore the effect of visual input, stimuli were presented in audio-only, audio–visual and visual-only conditions. We used motion and audio recordings of expert saxophone players playing two contrasting excerpts (positively and negatively valenced). For each excerpt, stimuli represented five motion degrees with increasing quantity of motion (QoM) and distinct predominant gestures. In the experiment (online and in-person), 384 participants rated performance recordings for expressiveness, professionalism and overall quality. Results revealed that, for the positively valenced excerpt, ratings increased as a function of QoM, whilst for the negatively valenced, the recording with predominant flap motion was favoured. Musicianship did not have a significant effect in motion perception. Concerning multisensory integration, both musicians and non-musicians presented visual dominance in the positively valenced excerpt, whereas in the negatively valenced, musicians shifted to auditory dominance. Our findings demonstrate that musical style not only determines the way observers perceive musicians’ movement as adequate, but also that it can promote changes in multisensory integration.

In the ear of the viewer: Toward an auditory dominance in early word learning

In everyday life, children hear but also often see their caregiver talking. Children build on this correspondence to resolve auditory uncertainties and decipher words from the speech input. As they hear the name of an object, 18- to 30-month-olds form a representation that permits word recognition in either the auditory (i.e. acoustic form of the word with no accompanying face) or the visual modality (i.e. seeing a silent talking face). Continuing on this work, we ask whether this ability already exists at a younger age. Using a cross-modal word learning task, French-learning 14-month-old infants were taught novel word–object mappings. During learning, they experienced the words auditorily. At test, they experienced the words either in the auditory or the visual modality. Results revealed successful word recognition in the auditory modality only. This suggests that as opposed to older children, 14-month-old infants only interpret novel auditorily learned words auditorily. This finding is discussed in line with the perceptual and lexical achievements that may influence infants’ capacity to navigate from the auditory to the visual modality during word learning.

Why is it difficult for children and adults to follow a person's eye gaze in polynomial social relationships with compound audio-visual stimuli: An eye-tracking study.

Acquiring triadic social interactions could facilitate language and communication skills in early infancy. However, studies have rarely investigated polynomial relationships, defined as relationships among the self, two or more people, and objects. During the development from a child to an adult, the responsiveness to a preferred stimulus modality changes from visual to auditory dominance. Nevertheless, how people observe compound visual stimuli in polynomial social relationships and why it is difficult to ignore auditory cues remain unclear. Moreover, there is a need to identify differences between children's and adults' observing latencies in the time to the first fixation when detecting a stimulus. This study examined whether participants (24 adults and 19 children) demonstrated similar gaze patterns under triadic and polyadic conditions. The participants observed a target visual stimulus looked at by a face stimulus while we presented spoken names, either congruent or incongruent with the target visual stimulus. The results indicated that when the number of people in social relationships increased, children and adults decreased fixations on the target face and the stimulus and showed a shorter mean fixation duration on the face. Moreover, children had longer latencies and more fixation errors for the target stimulus, which might reflect children's difficulties in communicating with others. We expect that understanding children's communication transition from triadic to polynomial social relationships with audio-visual stimulus congruencies would facilitate understanding language development and social communication patterns.

Spatial attention modulates auditory dominance in audiovisual order judgment.

Auditory dominance in audiovisual temporal order judgment is shown here to be modulated by exogenous orienting of attention to a spatial cue independent of the cue modality. The visual stimulus has to lead the auditory one further in advance for cued relative to uncued locations in order for the two to be perceived simultaneously, possibly suggesting an inhibitory function of spatial attention on temporal processing.

Turn that music down! Affective musical bursts cause an auditory dominance in children recognizing bodily emotions

Previous work has shown that different sensory channels are prioritized across the life course, with children preferentially responding to auditory information. The aim of the current study was to investigate whether the mechanism that drives this auditory dominance in children occurs at the level of encoding (overshadowing) or when the information is integrated to form a response (response competition). Given that response competition is dependent on a modality integration attempt, a combination of stimuli that could not be integrated was used so that if children’s auditory dominance persisted, this would provide evidence for the overshadowing over the response competition mechanism. Younger children (≤7 years), older children (8–11 years), and adults (18+ years) were asked to recognize the emotion (happy or fearful) in either nonvocal auditory musical emotional bursts or human visual bodily expressions of emotion in three conditions: unimodal, congruent bimodal, and incongruent bimodal. We found that children performed significantly worse at recognizing emotional bodies when they heard (and were told to ignore) musical emotional bursts. This provides the first evidence for auditory dominance in both younger and older children when presented with modally incongruent emotional stimuli. The continued presence of auditory dominance, despite the lack of modality integration, was taken as supportive evidence for the overshadowing explanation. These findings are discussed in relation to educational considerations, and future sensory dominance investigations and models are proposed.

An auditory processing advantage enables communication in less complex social settings: Signs of an extreme female brain in children and adolescents being assessed for Autism Spectrum Disorders.

The underlying factors of the male predominance in Autism Spectrum Disorders (ASD) are largely unknown, although a female advantage in social communication has been pointed out as a potential factor. Recently, attention has been given to ASD as a sensory processing disorder, focusing on the audio-visual temporal processing paramount for the development of communication. In ASD, a deviant audio-visual processing has been noted, resulting in difficulties interpreting multisensory information. Typically Developed (TD) females have shown an enhanced language processing in unisensory situations compared to multisensory situations. We aim to find out whether such an advantage also can be seen in girls within the ASD population, and if so, is it related to social communication skills? Forty children (IQ > 85), 20 females (mean age = 13.90 years, SD = 2.34) and 20 males (mean age = 12.15 years, SD = 2.83) triaged for an ASD assessment were recruited from a child and youth psychiatric clinic in Sweden. Using The Social Responsiveness Scale (SRS) we looked at associations with child performance on the Integrated Visual and Auditory Continuous Performance Test (IVA-2). An auditory advantage in the female group was associated with less rated problems in social communications in unisensory processing whereas in multisensory processing an auditory dominance was associated with more rated problems in Social Awareness. In the male group, a visual dominance was associated with more rated problems in Social Rigidity. A female unisensory processing advantage in ASD could very well be explaining the male domination in ASD. However, the social difficulties related to multisensory processing indicate that ASD females might be struggling as hard as males in more complex settings. Implications on the assessment procedure are discussed.

Dynamic Weighting of Time-Varying Visual and Auditory Evidence During Multisensory Decision Making

Perceptual decision-making in a dynamic environment requires two integration processes: integration of sensory evidence from multiple modalities to form a coherent representation of the environment, and integration of evidence across time to accurately make a decision. Only recently studies started to unravel how evidence from two modalities is accumulated across time to form a perceptual decision. One important question is whether information from individual senses contributes equally to multisensory decisions. We designed a new psychophysical task that measures how visual and auditory evidence is weighted across time. Participants were asked to discriminate between two visual gratings, and/or two sounds presented to the right and left ear based on respectively contrast and loudness. We varied the evidence, i.e., the contrast of the gratings and amplitude of the sound, over time. Results showed a significant increase in performance accuracy on multisensory trials compared to unisensory trials, indicating that discriminating between two sources is improved when multisensory information is available. Furthermore, we found that early evidence contributed most to sensory decisions. Weighting of unisensory information during audiovisual decision-making dynamically changed over time. A first epoch was characterized by both visual and auditory weighting, during the second epoch vision dominated and the third epoch finalized the weighting profile with auditory dominance. Our results suggest that during our task multisensory improvement is generated by a mechanism that requires cross-modal interactions but also dynamically evokes dominance switching.

Compensation Mechanisms May Not Always Account for Enhanced Multisensory Illusion in Older Adults: Evidence from Sound-Induced Flash Illusion.

Sound-induced flash illusion (SiFI) is typical auditory dominance phenomenon in multisensory illusion. Although a number of studies have explored the SiFI in terms of age-related effects, the reasons for the enhanced SiFI in older adults are still controversial. In the present study, older and younger adults with equal visual discrimination were selected to explore age differences in SiFI effects, and to explore the neural indicators by resting-state functional magnetic resonance imaging (rs-fMRI) signals. A correlation analysis was calculated to examine the relationship between regional homogeneity (ReHo) and the SiFI. The results showed that both younger and older adults experienced significant fission and fusion illusions, and fission illusions of older adults were greater than that of younger adults. In addition, our results showed ReHo values of the left middle frontal gyrus (MFG), the right inferior frontal gyrus (IFG) and right superior frontal gyrus (SFG) were significantly positively correlated with the SiFI in older adults. More importantly, the comparison between older and younger adults showed that ReHo values of the right superior temporal gyrus (STG) decreased in older adults, and this was independent of the SiFI. The results indicated that when there was no difference in unisensory ability, the enhancement of multisensory illusion in older adults may not always be explained by compensation mechanisms.

Exposure to multisensory and visual static or moving stimuli enhances processing of nonoptimal visual rhythms

Research has shown that visual moving and multisensory stimuli can efficiently mediate rhythmic information. It is possible, therefore, that the previously reported auditory dominance in rhythm perception is due to the use of nonoptimal visual stimuli. Yet it remains unknown whether exposure to multisensory or visual-moving rhythms would benefit the processing of rhythms consisting of nonoptimal static visual stimuli. Using a perceptual learning paradigm, we tested whether the visual component of the multisensory training pair can affect processing of metric simple two integer-ratio nonoptimal visual rhythms. Participants were trained with static (AVstat), moving-inanimate (AVinan), or moving-animate (AVan) visual stimuli along with auditory tones and a regular beat. In the pre- and posttraining tasks, participants responded whether two static-visual rhythms differed or not. Results showed improved posttraining performance for all training groups irrespective of the type of visual stimulation. To assess whether this benefit was auditory driven, we introduced visual-only training with a moving or static stimulus and a regular beat (Vinan). Comparisons between Vinan and Vstat showed that, even in the absence of auditory information, training with visual-only moving or static stimuli resulted in an enhanced posttraining performance. Overall, our findings suggest that audiovisual and visual static or moving training can benefit processing of nonoptimal visual rhythms.

Auditory Dominance in Processing Chinese Semantic Abnormalities in Response to Competing Audio-visual Stimuli

Language is a remarkable cognitive ability that can be expressed through visual (written language) or auditory (spoken language) modalities. When visual characters and auditory speech convey conflicting information, individuals may selectively attend to either one of them. However, the dominant modality in such a competing situation and the neural mechanism underlying it are still unclear. Here, we presented participants with Chinese sentences in which the visual characters and auditory speech convey conflicting information, while behavioral and electroencephalographic (EEG) responses were recorded. Results showed a prominent auditory dominance when audio-visual competition occurred. Specifically, higher accuracy (ACC), larger N400 amplitudes and more linkages in the posterior occipital-parietal areas were demonstrated in the auditory mismatch condition compared to that in the visual mismatch condition. Our research illustrates the superiority of the auditory speech over the visual characters, extending our understanding of the neural mechanisms of audio-visual competition in Chinese.

Attentional modulations of audiovisual interactions in apparent motion: Temporal ventriloquism effects on perceived visual speed.

The timing of brief stationary sounds has been shown to alter different aspects of visual motion, such as speed estimation. These effects of auditory timing have been explained by temporal ventriloquism and auditory dominance over visual information in the temporal domain. Although previous studies provide unprecedented evidence for the multisensory nature of speed estimation, how attention is involved in these audiovisual interactions remains unclear. Here, we aimed to understand the effects of spatial attention on these audiovisual interactions in time. We utilized a set of audiovisual stimuli that elicit temporal ventriloquism in visual apparent motion and asked participants to perform a speed comparison task. We manipulated attention either in the visual or auditory domain and systematically changed the number of moving objects in the visual field. When attention was diverted to a stationary object in the visual field via a secondary task, the temporal ventriloquism effects on perceived speed decreased. On the other hand, focusing attention on the auditory stimuli facilitated these effects consistently across different difficulty levels of secondary auditory task. Moreover, the effects of auditory timing on perceived speed did not change with the number of moving objects and existed in all the experimental conditions. Taken together, our findings revealed differential effects of allocating attentional resources in the visual and auditory domains. These behavioral results also demonstrate that reliable temporal ventriloquism effects on visual motion can be induced even in the presence of multiple moving objects in the visual field and under different perceptual load conditions.

Sound-induced flash illusion is modulated by the depth of auditory stimuli: Evidence from younger and older adults

Sound-induced flash illusion (SiFI) refers to an auditory dominance effect in which observers misperceive the number of visual flashes due to the simultaneous presentation of a different number of auditory beeps. Although the SiFI remains robust despite many factors, little is known about how the dominance of audition over vision changes as a function of spatial location in 3D space and how this corresponds with aging. In the present study, we recruited younger (18-26 years old) and older adults (60-76 years old) and presented visual stimuli in near space and auditory stimuli in far space (Vnear_Afar) or visual stimuli in far space and auditory stimuli in near space (Vfar_Anear), with the aim of exploring the effect of aging on spatial dominance in SiFI. The results showed that both participant groups exhibited a SiFI auditory dominance effect; however, older adults were more susceptible to illusions than younger adults. More importantly, for younger adults, when auditory stimuli were presented in far space, the accuracy and d' indicated that SiFI was significantly enhanced in the fission illusion, and the reaction time (RT) results showed that RT became longer in the fusion illusion. However, for older adults, the SiFI was not affected by the spatial location. Therefore, the present results reveal for the first time that the SiFI effect is modulated by the depth of auditory stimuli. Moreover, older adults were not susceptible to the depth dimension, thus showing age-related sensory decline and specifically showing a decreased ability to localize visual and auditory stimuli.

Stimulus modality affects the accuracy of rhythm production in rats

Vocal learning species such as humans and parrots show auditory dominance when they synchronize their actions to an external rhythm. However, whether non-vocal-learners show a specific modality dominance in a rhythmic task has scarcely been examined. We predicted that rats, who are nocturnal and known to rely on acoustic communication, would exhibit higher sensitivity to auditory rhythm compared to visual rhythm. We investigated whether performance of a synchronization task by rats differs based on stimulus modality. We trained five rats to press a lever in time to auditory-visual, isochronous stimuli presented at three different tempos. Rats showed a lower correct response rate when auditory stimuli were presented than when visual or auditory-visual stimuli were presented in the 0.5-s inter-onset interval condition. Neither the asynchrony with the stimulus onset, nor the variability of interval production differed significantly based on the stimulus modality. Therefore, contrary to the prediction, they did not show auditory dominance; rather, rats showed poor performance on the task when a visual stimulus was not present. These results are consistent with the gradual audio-motor evolution hypothesis, and suggest that rats share ability for rhythm production, but this might not necessarily depend on auditory modality.

Temporal Alignment but not Complexity of Audiovisual Stimuli Influences Crossmodal Duration Percepts.

Reliable duration perception is an integral aspect of daily life that impacts everyday perception, motor coordination, and subjective passage of time. The Scalar Expectancy Theory (SET) is a common model that explains how an internal pacemaker, gated by an external stimulus-driven switch, accumulates pulses during sensory events and compares these accumulated pulses to a reference memory duration for subsequent duration estimation. Second-order mechanisms, such as multisensory integration (MSI) and attention, can influence this model and affect duration perception. For instance, diverting attention away from temporal features could delay the switch closure or temporarily open the accumulator, altering pulse accumulation and distorting duration perception. In crossmodal duration perception, auditory signals of unequal duration can induce perceptual compression and expansion of durations of visual stimuli, presumably via auditory influence on the visual clock. The current project aimed to investigate the role of temporal (stimulus alignment) and nontemporal (stimulus complexity) features on crossmodal, specifically auditory over visual, duration perception. While temporal alignment revealed a larger impact on the strength of crossmodal duration percepts compared to stimulus complexity, both features showcase auditory dominance in processing visual duration.

Weighted Integration of Duration Information Across Visual and Auditory Modality Is Influenced by Modality-Specific Attention.

We constantly integrate multiple types of information from different sensory modalities. Generally, such integration is influenced by the modality that we attend to. However, for duration perception, it has been shown that when duration information from visual and auditory modalities is integrated, the perceived duration of the visual stimulus leaned toward the duration of the auditory stimulus, irrespective of which modality was attended. In these studies, auditory dominance was assessed using visual and auditory stimuli with different durations whose timing of onset and offset would affect perception. In the present study, we aimed to investigate the effect of attention on duration integration using visual and auditory stimuli of the same duration. Since the duration of a visual flicker and auditory flutter tends to be perceived as longer than and shorter than its physical duration, respectively, we used the 10 Hz visual flicker and auditory flutter with the same onset and offset timings but different perceived durations. The participants were asked to attend either visual, auditory, or both modalities. Contrary to the attention-independent auditory dominance reported in previous studies, we found that the perceived duration of the simultaneous flicker and flutter presentation depended on which modality the participants attended. To further investigate the process of duration integration of the two modalities, we applied Bayesian hierarchical modeling, which enabled us to define a flexible model in which the multisensory duration is represented by the weighted average of each sensory modality. In addition, to examine whether auditory dominance results from the higher reliability of auditory stimuli, we applied another models to consider the stimulus reliability. These behavioral and modeling results suggest the following: (1) the perceived duration of visual and auditory stimuli is influenced by which modality the participants attended to when we control for the confounding effect of onset–offset timing of stimuli, and (2) the increase of the weight by attention affects the duration integration, even when the effect of stimulus reliability is controlled. Our models can be extended to investigate the neural basis and effects of other sensory modalities in duration integration.

Tones slow down visuomotor responses in a visual-spatial task

The current study examined how simple tones affect speeded visuomotor responses in a visual-spatial sequence learning task. Across the three reported experiments, participants were presented with a visual target that appeared in different locations on a touchscreen monitor and they were instructed to touch the visual targets as quickly as possible. Visual sequences were either paired with sounds that correlated with the location of the target, paired with sounds that did not correlate with the location of the target, or the sequences were presented in silence (baseline). Response times decreased across training and participants were slower to respond to the visual stimuli when the sequences were paired with tones. Moreover, these interference effects were more pronounced early in training and explicit instructions directing attention to the visual modality had little effect on eliminating auditory interference, suggesting that these interference effects may stem from bottom-up factors and do not appear to be under attentional control. These findings have implications on tasks that require the processing of simultaneously presented auditory and visual information and provide support for a proposed mechanism underlying auditory dominance on a task that is typically better suited for the visual modality.

Children cannot ignore what they hear: Incongruent emotional information leads to an auditory dominance in children

Effective emotion recognition is imperative to successfully navigating social situations. Research suggests differing developmental trajectories for the recognition of bodily and vocal emotion, but emotions are usually studied in isolation and rarely considered as multimodal stimuli in the literature. When adults are presented with basic multimodal sensory stimuli, the Colavita effect suggests that they have a visual dominance, whereas more recent research finds that an auditory sensory dominance may be present in children under 8 years of age. However, it is not currently known whether this phenomenon holds for more complex multimodal social stimuli. Here we presented children and adults with multimodal social stimuli consisting of emotional bodies and voices, asking them to recognize the emotion in one modality while ignoring the other. We found that adults can perform this task with no detrimental effects on performance regardless of whether the ignored emotion was congruent or not. However, children find it extremely challenging to recognize bodily emotion while trying to ignore incongruent vocal emotional information. In several instances, they performed below chance level, indicating that the auditory modality actively informs their choice of bodily emotion. Therefore, this is the first evidence, to our knowledge, of an auditory dominance in children when presented with emotionally meaningful stimuli.

Listen to Your Heart: Examining Modality Dominance Using Cross-Modal Oddball Tasks.

The current study used cross-modal oddball tasks to examine cardiac and behavioral responses to changing auditory and visual information. When instructed to press the same button for auditory and visual oddballs, auditory dominance was found with cross-modal presentation slowing down visual response times more than auditory response times (Experiment 1). When instructed to make separate responses to auditory and visual oddballs, visual dominance was found with cross-modal presentation decreasing auditory discrimination, and participants also made more visual-based than auditory-based errors on cross-modal trials (Experiment 2). Experiment 3 increased task demands while requiring a single button press and found evidence of auditory dominance, suggesting that it is unlikely that increased task demands can account for the reversal in Experiment 2. Auditory processing speed was the best predictor of auditory dominance, with auditory dominance being stronger in participants who were slower at processing the sounds, whereas auditory and visual processing speed and baseline heart rate variability did not predict visual dominance. Examination of cardiac responses that were time-locked with stimulus onset showed cross-modal facilitation effects, with auditory and visual discrimination occurring earlier in the course of processing in the cross-modal condition than in the unimodal conditions. The current findings showing that response demand manipulations reversed modality dominance and that time-locked cardiac responses show cross-modal facilitation, not interference, suggest that auditory and visual dominance effects may both be occurring later in the course of processing, not from disrupted encoding.

Stimulus Onset Modulates Auditory and Visual Dominance.

Investigations of multisensory integration have demonstrated that, under certain conditions, one modality is more likely to dominate the other. While the direction of this relationship typically favors the visual modality, the effect can be reversed to show auditory dominance under some conditions. The experiments presented here use an oddball detection paradigm with variable stimulus timings to test the hypothesis that a stimulus that is presented earlier will be processed first and therefore contribute to sensory dominance. Additionally, we compared two measures of sensory dominance (slowdown scores and error rate) to determine whether the type of measure used can affect which modality appears to dominate. When stimuli were presented asynchronously, analysis of slowdown scores and error rates yielded the same result; for both the 1- and 3-button versions of the task, participants were more likely to show auditory dominance when the auditory stimulus preceded the visual stimulus, whereas evidence for visual dominance was observed as the auditory stimulus was delayed. In contrast, for the simultaneous condition, slowdown scores indicated auditory dominance, whereas error rates indicated visual dominance. Overall, these results provide empirical support for the hypothesis that the modality that engages processing first is more likely to show dominance, and suggest that more explicit measures of sensory dominance may favor the visual modality.