Congruent Sounds Research Articles

Product-related sounds speed visual search

Searching for a particular product in a supermarket can be a challenging business. The question therefore arises as to whether cues from the shopper’s other senses can be used to facilitate, guide, or bias visual search toward a particular product or product type. Prior research suggests that characteristic sounds can facilitate visual object localization (Iordanescu et al., 2008, 2010). Extending these findings to an applied setting, we investigated whether product-related sounds would facilitate visual search for products from different categories (e.g., champagne, potato crisps, deodorant) when arranged on a virtual shelf. On each trial, participants were visually presented with the name of a target product and then located the target within a virtual shelf display containing pictures of four different products (randomly selected from a set of nine). The visual display was randomly accompanied by a target-congruent, a target-incongruent, an unrelated, or no sound. Congruent sounds were semantically related to the target (e.g., uncorking a champagne bottle), incongruent sounds were related to the product shown in the corner opposite to the target, and unrelated sounds did not correspond to any of the products shown in the display. Participants found the target product significantly faster when the sound was congruent rather than incongruent with the target. All other pairwise comparisons were non-significant. These results extend the facilitatory crossmodal effect of characteristic sounds on visual search performance described earlier to the more realistic context of a virtual shelf display, showing that characteristic sounds can crossmodally enhance the visual processing of actual products.

Mapping Symbols to Sounds: Electrophysiological Correlates of the Impaired Reading Process in Dyslexia

Dyslexic and control first-grade school children were compared in a Symbol-to-Sound matching test based on a non-linguistic audiovisual training which is known to have a remediating effect on dyslexia. Visual symbol patterns had to be matched with predicted sound patterns. Sounds incongruent with the corresponding visual symbol (thus not matching the prediction) elicited the N2b and P3a event-related potential (ERP) components relative to congruent sounds in control children. Their ERPs resembled the ERP effects previously reported for healthy adults with this paradigm. In dyslexic children, N2b onset latency was delayed and its amplitude significantly reduced over left hemisphere whereas P3a was absent. Moreover, N2b amplitudes significantly correlated with the reading skills. ERPs to sound changes in a control condition were unaffected. In addition, correctly predicted sounds, that is, sounds that are congruent with the visual symbol, elicited an early induced auditory gamma band response (GBR) reflecting synchronization of brain activity in normal-reading children as previously observed in healthy adults. However, dyslexic children showed no GBR. This indicates that visual symbolic and auditory sensory information are not integrated into a unitary audiovisual object representation in them. Finally, incongruent sounds were followed by a later desynchronization of brain activity in the gamma band in both groups. This desynchronization was significantly larger in dyslexic children. Although both groups accomplished the task successfully remarkable group differences in brain responses suggest that normal-reading children and dyslexic children recruit (partly) different brain mechanisms when solving the task. We propose that abnormal ERPs and GBRs in dyslexic readers indicate a deficit resulting in a widespread impairment in processing and integrating auditory and visual information and contributing to the reading impairment in dyslexia.

Audiovisual integration in low vision individuals

Behavioral and neurophysiological studies have shown an enhancement of visual perception in crossmodal audiovisual stimulation conditions, both for sensitivity and reaction times, when the stimulation in the two sensory modalities occurs in condition of space and time congruency. The purpose of the present work is to verify whether congruent visual and acoustic stimulations can improve the detection of visual stimuli in people affected by low vision. Participants were asked to detect the presence of a visual stimulus (yes/no task) either presented in isolation (i.e., unimodal visual stimulation) or simultaneously with auditory stimuli, which could be placed in the same spatial position (i.e., crossmodal congruent conditions) or in different spatial positions (i.e., crossmodal incongruent conditions). The results show for the first time audiovisual integration effects in low vision individuals. In particular, it has been observed a significant visual detection benefit in the crossmodal congruent as compared to the unimodal visual condition. This effect is selective for visual stimulation that occurs in the portion of visual field that is impaired, and disappears in the region of space in which vision is spared. Surprisingly, there is a marginal crossmodal benefit when the sound is presented at 16 degrees far from the visual stimulus. The observed crossmodal effect seems to be determined by the contribution of both senses to a model of optimal combination, in which the most reliable provides the highest contribution. These results, indicating a significant beneficial effect of synchronous and spatially congruent sounds in a visual detection task, seem very promising for the development of a rehabilitation approach of low vision diseases based on the principles of multisensory integration.

The effects of characteristic and spatially congruent sounds on visual search in natural visual scenes

The effects of characteristic and spatially congruent sounds on visual search in natural visual scenes

Audiovisual speech recalibration in children*

In order to examine whether children adjust their phonetic speech categories, children of two age groups, five-year-olds and eight-year-olds, were exposed to a video of a face saying /aba/ or /ada/ accompanied by an auditory ambiguous speech sound halfway between /b/ and /d/. The effect of exposure to these audiovisual stimuli was measured on subsequently delivered auditory-only speech identification trials. Results were compared to a control condition in which the audiovisual exposure stimuli contained non-ambiguous and congruent sounds /aba/ or /ada/. The older children learned to categorize the initially ambiguous speech sound in accord with the previously seen lip-read information (i.e. recalibration), but this was not the case for the younger age group. Moreover, all children displayed a tendency to report the stimulus that they were exposed to during the exposure phase. Methodological improvements for adjusting such a response bias are discussed.

Naturalistic auditory scene analysis in children and adults

In order to make sense of natural auditory environments, the developing child must learn to ‘‘navigate’’ through complex auditory scenes in order to segment out relevant auditory information from irrelevant sounds. This study investigated some of the informational and attentional factors that constrain environmental sound detection in auditory scenes, and how these factors change over development. Thirty‐two children (aged 9–12 years) and 16 adults were asked to detect short target environmental sounds (e.g., a dog barking) within longer environmental background sounds (e.g., a barn) presented dichotically. The target environmental sounds were either congruent (i.e., normally associated with the background) or incongruent. Subjects heard either a single background presented to both the ipsilateral and contralateral ears or else different background sounds presented to the different ears. Results indicate that children’s, but not adults’ target detection is substantially less accurate when listening to two auditory scenes than to a single scene presented in both ears, suggesting a developmental shift in listeners’ abilities to process multiple information streams. Interestingly, both children and adults find incongruent sounds more salient than congruent sounds. Furthermore, this incongruence ‘‘pop‐out’’ effect is greatly reduced in the presence of the dual background (informational masking) condition.

Binding Symbols and Sounds: Evidence from Event-Related Oscillatory Gamma-Band Activity

The present study intended to examine the neural basis of audiovisual integration, hypothetically achieved by synchronized gamma-band oscillations (30-80 Hz) that have been suggested to integrate stimulus features and top-down information. To that end, we studied the impact of visual symbolic information on early auditory sensory processing of upcoming sounds. In particular, we used a symbol-to-sound-matching paradigm in which simple score-like patterns predict corresponding sound patterns. Occasionally, a single sound is incongruent with the corresponding element of the visual pattern. In response to expected sounds congruent with the corresponding visual symbol, a power increase of phase-locked (evoked) activity in the 40-Hz band was observed peaking 42-ms poststimulus onset. Thus, for the first time, we demonstrated that the comparison process between a neural model, the expectation, and the current sensory input is implemented at very early levels of auditory processing. Subsequently, expected congruent sounds elicited a broadband power increase of non-phase-locked (induced) activity peaking 152-ms poststimulus onset, which might reflect the formation of a unitary event representation including both visual and auditory aspects of the stimulation. Gamma-band responses were not present for unexpected incongruent sounds. A model explaining the anticipatory activation of cortical auditory representations and the match of experience against expectation is presented.