Congruent Sounds Research Articles

Type of auditory cues and apparatus influence how healthy young adults integrate sounds for dynamic balance.

It is unclear whether the brain handles auditory cues similarly to visual cues for balance. We investigated the influence of headphones and loudspeaker reproduction of sounds on dynamic balance performance when an individual is facing a cognitive challenge. Twenty participants (16 females, aged 19-36) were asked to avoid a ball according to a specific visual rule. Visuals were projected from the HTC Vive head-mounted display in an acoustically controlled space. We varied the environment by adding congruent sounds (sounds coincide with the visual rule) or incongruent sounds (sounds may or may not coincide with the visual rule) as well as creating a multimodal (visual and congruent sounds) vs. unimodal (visual or congruent sounds only) display of stimuli. Sounds were played over headphones or loudspeakers. We quantified reaction time (RT) and accuracy (choosing the correct direction to move) by capturing the head movement. We found that in the absence of sounds, RT was slower with headphones compared to loudspeakers, but the introduction of either congruent or incongruent sounds resulted in faster movements with headphones such that RT was no longer different between apparatus. Participants used congruent sounds to improve accuracy but disregarded incongruent sounds. This suggests that selective attention may explain how sounds are incorporated into dynamic balance performance in healthy young adults. Participants leveraged sounds played over loudspeakers, but not over headphones, to enhance accuracy in a unimodal dark environment. This may be explained by the natural listening conditions created by loudspeakers where sounds may be perceived as externalized.

Effects of sounds on the visitors' experience in museums

Museums increasingly recognize the significance of acoustics for a high-quality visitor experience. Objective/physical parameters have been extensively studied, but there is a need for research focusing on user perception. This study addressed this gap by investigating visitors' soundscape preferences and the effects of different soundscapes on the museum experience. We hypothesized that exhibition-congruent sounds are more effective than conventional masking sounds in reducing distractions, enhancing engagement with artwork, and creating immersive experiences. The research utilized the Cognitive Immersive Room (CIR) at Rensselaer Polytechnic Institute, featuring a 360° visual display with a multi-channel loudspeaker system for spatial audio rendering to project panoramic photographs and ambisonic audio recordings recorded in 16 exhibitions in five relevant museums—for example, MASSMoca and the New York State Museum. Four scenarios were presented to the participants: originally recorded soundscape, recorded soundscape combined with a conventional sound masker, recorded soundscape combined with a congruent sound masker, and silence. After experiencing the immersive environment, 23 subjects responded to a questionnaire. The results showed congruent sounds increased focus and engagement and contributed to a more comprehensive and immersive experience. In 58% of cases, the participants preferred the congruent-sound scenario over the others.

Audiovisual correspondence facilitates the visual search for biological motion

Hearing synchronous sounds may facilitate the visual search for the concurrently changed visual targets. Evidence for this audiovisual attentional facilitation effect mainly comes from studies using artificial stimuli with relatively simple temporal dynamics, indicating a stimulus-driven mechanism whereby synchronous audiovisual cues create a salient object to capture attention. Here, we investigated the crossmodal attentional facilitation effect on biological motion (BM), a natural, biologically significant stimulus with complex and unique dynamic profiles. We found that listening to temporally congruent sounds, compared with incongruent sounds, enhanced the visual search for BM targets. More intriguingly, such a facilitation effect requires the presence of distinctive local motion cues (especially the accelerations in feet movement) independent of the global BM configuration, suggesting a crossmodal mechanism triggered by specific biological features to enhance the salience of BM signals. These findings provide novel insights into how audiovisual integration boosts attention to biologically relevant motion stimuli and extend the function of a proposed life detection system driven by local kinematics of BM to multisensory life motion perception.

Object memory is multisensory: Task-irrelevant sounds improve recollection

Hearing a task-irrelevant sound during object encoding can improve visual recognition memory when the sound is object-congruent (e.g., a dog and a bark). However, previous studies have only used binary old/new memory tests, which do not distinguish between recognition based on the recollection of details about the studied event or stimulus familiarity. In the present research, we hypothesized that hearing a task-irrelevant but semantically congruent natural sound at encoding would facilitate the formation of richer memory representations, resulting in increased recollection of details of the encoded event. Experiment 1 replicates previous studies showing that participants were more confident about their memory for items that were initially encoded with a congruent sound compared to an incongruent sound. Experiment 2 suggests that congruent object-sound pairings specifically facilitate recollection and not familiarity-based recognition memory, and Experiment 3 demonstrates that this effect was coupled with more accurate memory for audiovisual congruency of the item and sound from encoding rather than another aspect of the episode. These results suggest that even when congruent sounds are task-irrelevant, they promote formation of multisensory memories and subsequent recollection-based retention. Given the ubiquity of encounters with multisensory objects in our everyday lives, considering their impact on episodic memory is integral to building models of memory that apply to naturalistic settings.

Expectations about dynamic visual objects facilitates early sensory processing of congruent sounds

The perception of a moving object can lead to the expectation of its sound, yet little is known about how visual expectations influence auditory processing. We examined how visual perception of an object moving continuously across the visual field influences early auditory processing of a sound that occurred congruently or incongruently with the object's motion. In Experiment 1, electroencephalogram (EEG) activity was recorded from adults who passively viewed a ball that appeared either on the left or right boundary of a display and continuously traversed along the horizontal midline to make contact and elicit a bounce sound off the opposite boundary. Our main analysis focused on the auditory-evoked event-related potential. For audio-visual (AV) trials, a sound accompanied the visual input when the ball contacted the opposite boundary (AV-synchronous), or the sound occurred before contact (AV-asynchronous). We also included audio-only and visual-only trials. AV-synchronous sounds elicited an earlier and attenuated auditory response relative to AV-asynchronous or audio-only events. In Experiment 2, we examined the roles of expectancy and multisensory integration in influencing this response. In addition to the audio-only, AV-synchronous, and AV-asynchronous conditions, participants were shown a ball that became occluded prior to reaching the boundary of the display, but elicited an expected sound at the point of occluded collision. The auditory response during the AV-occluded condition resembled that of the AV-synchronous condition, suggesting that expectations induced by a moving object can influence early auditory processing. Broadly, the results suggest that dynamic visual stimuli can help generate expectations about the timing of sounds, which then facilitates the processing of auditory information that matches these expectations.

Multisensory expectations about dynamic visual objects facilitates early sensory processing of congruent sounds

In everyday life, visual objects are often accompanied by sounds, yet little is known about how information in one sense may influence the processing of information in another sense. We examined how dynamic visual input – an object moving continuously across the visual field – influences early auditory processing of a sound that is either congruent with the object’s motion, and thus likely perceived as being part of the visual object, or incongruent with the object’s motion. We recorded EEG activity from 31 neurotypical adults who passively viewed a red ball that appeared either on the far left or right edge of the display and continuously traversed along the horizontal midline to make contact and bounce off the opposite edge. For multisensory trials, a tone accompanied the visual input the moment the ball made contact with the opposite edge (AV-synchronous), or the sound occurred 450ms before contact (AV-asynchronous). We also included audio-only and visual-only trials. Our main analysis focused on the auditory-evoked event-related potential (ERP) measured at frontal electrode sites and revealed reliable differences in the amplitude and latency of the N1-P2 auditory complex (all p’s < 0.003). Follow-up pairwise comparisons showed a reduced N1-amplitude for the AV-synchronous condition relative to the AV-asynchronous and audio-only conditions, and a delayed latency for the AV-asynchronous condition relative to the AV-synchronous and audio-only conditions. P2-peak analyses revealed greater amplitude for the audio-only relative to the AV-synchronous and AV-asynchronous and delayed latency toward AV-asynchronous relative to AV-synchronous and audio-only conditions. Overall, these results show that audio-visual synchrony elicited a faster and attenuated early auditory response relative to asynchronous or auditory-only events. This suggests that dynamic visual stimuli can help generate expectations about the timing of auditory events, which then facilitates the processing of auditory information that matches these expectations.

Learning to Expect: Predicting Sounds During Movement Is Related to Sensorimotor Association During Listening.

Sensory experiences, such as sound, often result from our motor actions. Over time, repeated sound-producing performance can generate sensorimotor associations. However, it is not clear how sensory and motor information are associated. Here, we explore if sensory prediction is associated with the formation of sensorimotor associations during a learning task. We recorded event-related potentials (ERPs) while participants produced index and little finger-swipes on a bespoke device, generating novel sounds. ERPs were also obtained as participants heard those sounds played back. Peak suppression was compared to assess sensory prediction. Additionally, transcranial magnetic stimulation (TMS) was used during listening to generate finger-motor evoked potentials (MEPs). MEPs were recorded before and after training upon hearing these sounds, and then compared to reveal sensorimotor associations. Finally, we explored the relationship between these components. Results demonstrated that an increased positive-going peak (e.g., P2) and a suppressed negative-going peak (e.g., N2) were recorded during action, revealing some sensory prediction outcomes (P2: p = 0.050, = 0.208; N2: p = 0.001, = 0.474). Increased MEPs were also observed upon hearing congruent sounds compared with incongruent sounds (i.e., associated to a finger), demonstrating precise sensorimotor associations that were not present before learning (Index finger: p < 0.001, = 0.614; Little finger: p < 0.001, = 0.529). Consistent with our broad hypotheses, a negative association between the MEPs in one finger during listening and ERPs during performance of the other was observed (Index finger MEPs and Fz N1 action ERPs; r = −0.655, p = 0.003). Overall, data suggest that predictive mechanisms are associated with the fine-tuning of sensorimotor associations.

Revealing the multisensory modulation of auditory stimulus in degraded visual object recognition by dynamic causal modeling

Recent evidence from neurophysiological and functional imaging research has demonstrated that semantically congruent sounds can modulate the identification of a degraded visual object. However, it remains unclear how different integration regions interact with each other when only a visual object was obscured. The present study aimed to elucidate the neural bases of cross-modal functional interactions in degraded visual object recognition. Naturally degraded images and semantically congruent sounds were used in our experiment. Participants were presented with three different modalities of audio-visual stimuli: auditory only (A), degraded visual only (Vd), and simultaneous auditory and degraded visual (AVd). We used conjunction analysis and the classical 'max criterion' to define three audiovisual integration cortical hubs: the visual association cortex, the superior temporal sulcus and the Heschl's gyrus. Dynamic causal modeling (DCM) was then used to infer effective connectivity between these regions. The DCM results revealed that the modulation of an auditory stimulus resulted in increased connectivity from the Heschl's gyrus to the visual association cortex and from the superior temporal sulcus to the visual association cortex. It was shown that the visual association cortex is modulated not only via feedback and top-down connections from higher-order convergence areas but also via lateral feedforward connectivity from the auditory cortex. The present findings give support to interconnected models of cross-modal information integration.

Semantically Congruent Sounds Facilitate the Decoding of Degraded Images

Semantically congruent sounds can facilitate perception of visual objects in the human brain. However, the manner in which semantically congruent sounds affect cognitive processing for degraded visual stimuli remains unclear. We presented participants with naturalistic degraded images and semantically congruent sounds from different conceptual categories in three modalities: degraded visual only, auditory only, and auditory and degraded visual. Functional magnetic resonance imaging was performed to assess variations in brain-activation spatial patterns. In order to account for the facilitation of auditory modulation at different levels, four conceptual categories of stimuli were divided into coarse and fine groups. Conjunction analysis and multivariate pattern analysis were used to investigate integrative properties. Superadditive interactions were found in the visual association cortex and subadditive interactions were observed in the superior temporal sulcus/superior temporal gyrus (STS/STG). Our results demonstrate that the visual association cortex and STS/STG are involved in the integration of auditory and degraded visual information. In addition, the pattern classification results imply that semantically congruent sounds may facilitate identification of degraded images in both coarse and fine groups. Importantly, when naturalistic visual stimuli were further subdivided, facilitation through auditory modulation exhibited category selectivity.

Neighing, barking, and drumming horses-object related sounds help and hinder picture naming.

The study presented here investigated how environmental sounds influence picture naming. In a series of four experiments participants named pictures (e.g., the picture of a horse) while hearing task-irrelevant sounds (e.g., neighing, barking, or drumming). Experiments 1 and 2 established two findings, facilitation from congruent sounds (e.g., picture: horse, sound: neighing) and interference from semantically related sounds (e.g., sound: barking), both relative to unrelated sounds (e.g., sound: drumming). Experiment 3 replicated the effects in a situation in which participants were not familiarized with the sounds prior to the experiment. Experiment 4 replicated the congruency facilitation effect, but showed that semantic interference was not obtained with distractor sounds which were not associated with target pictures (i.e., were not part of the response set). The general pattern of facilitation from congruent sound distractors and interference from semantically related sound distractors resembles the pattern commonly observed with distractor words. This parallelism suggests that the underlying processes are not specific to either distractor words or distractor sounds but instead reflect general aspects of semantic-lexical selection in language production. The results indicate that language production theories need to include a competitive selection mechanism at either the lexical processing stage, or the prelexical processing stage, or both. (PsycINFO Database Record

Semantic congruency of auditory warnings

The aim of this study was to explore operator experience and performance for semantically congruent and incongruent auditory icons and abstract alarm sounds. It was expected that performance advantages for congruent sounds would be present initially but would reduce over time for both alarm types. Twenty-four participants (12M/12F) were placed into auditory icon or abstract alarm groupings. For each group both congruent and incongruent alarms were used to represent different driving task scenarios. Once sounded, participants were required to respond to each alarm by selecting a corresponding driving scenario. User performance for all sound types improved over time, however even with experience a decrement in speed of response remained for the incongruent iconic sounds and in accuracy of performance for the abstract warning sounds when compared to the congruent auditory icons. Semantic congruency was found to be of more importance for auditory icons than for abstract sounds.Practitioner Summary: Alarms are used in many operating systems as emergency, alerting, or continuous monitoring signals for instance. This study found that the type and representativeness of an auditory warning will influence operator performance over time. Semantically congruent iconic sounds produced performance advantages over both incongruent iconic sounds and abstract warnings.

Improving left spatial neglect through music scale playing.

The study assessed whether the auditory reference provided by a music scale could improve spatial exploration of a standard musical instrument keyboard in right-brain-damaged patients with left spatial neglect. As performing music scales involves the production of predictable successive pitches, the expectation of the subsequent note may facilitate patients to explore a larger extension of space in the left affected side, during the production of music scales from right to left. Eleven right-brain-damaged stroke patients with left spatial neglect, 12 patients without neglect, and 12 age-matched healthy participants played descending scales on a music keyboard. In a counterbalanced design, the participants' exploratory performance was assessed while producing scales in three feedback conditions: With congruent sound, no-sound, or random sound feedback provided by the keyboard. The number of keys played and the timing of key press were recorded. Spatial exploration by patients with left neglect was superior with congruent sound feedback, compared to both Silence and Random sound conditions. Both the congruent and incongruent sound conditions were associated with a greater deceleration in all groups. The frame provided by the music scale improves exploration of the left side of space, contralateral to the right hemisphere, damaged in patients with left neglect. Performing a scale with congruent sounds may trigger at some extent preserved auditory and spatial multisensory representations of successive sounds, thus influencing the time course of space scanning, and ultimately resulting in a more extensive spatial exploration. These findings offer new perspectives also for the rehabilitation of the disorder.

Lack of automatic attentional orienting by gaze cues following a bilateral loss of visual cortex

In social interactions, the location of relevant stimuli is often indicated by the orientation of gaze. It has been proposed that the direction of gaze might produce an automatic cueing of attention, similar to what is observed with exogenous cues. However, several reports have challenged this claim by demonstrating that the behavioral gain that arises with gaze cueing could be explained by shifts of attention, which are intentional and not automatic. We reasoned that if cueing by gaze was truly automatic, it should occur without awareness and should be sustained by subcortical circuits, including the amygdalae, independently of the main geniculo-striate visual pathway. We presented a cross-modal version of the Posner cueing paradigm to a patient (TN) with bilateral lesions of occipital cortex (Burra et al., 2013; Pegna, Khateb, Lazeyras, & Seghier, 2005). TN was asked to localize a sound using a key press. The location of the sound was congruent or incongruent with the direction of gaze of a face-cue. In groups of healthy young and age-matched participants, we observed significantly longer response times for incongruent than congruent sounds, suggesting that gaze direction interfered with processing of localized sounds. By contrast, TN׳s performance was not affected by sound-gaze congruence. The results suggest that the processing of gaze orientation cannot occur in the absence of geniculo-striate processing, suggesting that it is not automatic.

Congruent Sound Can Modulate Odor Pleasantness

This study aimed to determine 1) whether certain background sounds can be matched with specific odors and 2) whether the background sounds can increase pleasantness for their congruent odors. In Experiment 1, congruent sounds increased odor pleasantness, but not odor intensity, significantly more than incongruent sounds. Experiment 2 demonstrated that certain background sounds can be paired with specific odors. For example, cinnamon, clove, and orange odors were rated significantly more congruent with a Christmas carol compared with the sound of brushing teeth and/or the beach sound. The congruent sounds increased odor pleasantness significantly more than incongruent sounds. Similarly, the congruent sound-induced odor pleasantness was observed in Experiment 3. As participants judged the pair of odor and sound to be more congruent, they rated the odor significantly more pleasant. Congruent sound assisted participants in identifying and in being familiar with the odor, thereby leading to an increase in odor pleasantness. However, the congruent sound-induced odor pleasantness was not obtained in all odors. In conclusion, this study provides new empirical evidence that pleasantness ratings for odors can increase in the presence of their congruent sounds.

Color Selectivity of the Spatial Congruency Effect: Evidence from the Focused Attention Paradigm

ABSTRACT The multisensory response enhancement (MRE), occurring when the response to a visual target integrated with a spatially congruent sound is stronger than the response to the visual target alone, is believed to be mediated by the superior colliculus (SC) (Stein & Meredith, 1993). Here, we used a focused attention paradigm to show that the spatial congruency effect occurs with red (SC-effective) but not blue (SC-ineffective) visual stimuli, when presented with spatially congruent sounds. To isolate the chromatic component of SC-ineffective targets and to demonstrate the selectivity of the spatial congruency effect we used the random luminance modulation technique (Experiment 1) and the tritanopic technique (Experiment 2). Our results indicate that the spatial congruency effect does not require the distribution of attention over different sensory modalities and provide correlational evidence that the SC mediates the effect.

Emotional sounds modulate early neural processing of emotional pictures

In our natural environment, emotional information is conveyed by converging visual and auditory information; multimodal integration is of utmost importance. In the laboratory, however, emotion researchers have mostly focused on the examination of unimodal stimuli. Few existing studies on multimodal emotion processing have focused on human communication such as the integration of facial and vocal expressions. Extending the concept of multimodality, the current study examines how the neural processing of emotional pictures is influenced by simultaneously presented sounds. Twenty pleasant, unpleasant, and neutral pictures of complex scenes were presented to 22 healthy participants. On the critical trials these pictures were paired with pleasant, unpleasant, and neutral sounds. Sound presentation started 500 ms before picture onset and each stimulus presentation lasted for 2 s. EEG was recorded from 64 channels and ERP analyses focused on the picture onset. In addition, valence and arousal ratings were obtained. Previous findings for the neural processing of emotional pictures were replicated. Specifically, unpleasant compared to neutral pictures were associated with an increased parietal P200 and a more pronounced centroparietal late positive potential (LPP), independent of the accompanying sound valence. For audiovisual stimulation, increased parietal P100 and P200 were found in response to all pictures which were accompanied by unpleasant or pleasant sounds compared to pictures with neutral sounds. Most importantly, incongruent audiovisual pairs of unpleasant pictures and pleasant sounds enhanced parietal P100 and P200 compared to pairings with congruent sounds. Taken together, the present findings indicate that emotional sounds modulate early stages of visual processing and, therefore, provide an avenue by which multimodal experience may enhance perception.

Changing Pitch of Sounds Alters Perceived Visual Motion Trajectory

Several studies have examined the effects of auditory stimuli on visual perception. In studies of cross-modal correspondences, auditory pitch has been shown to modulate visual motion perception. In particular, low-reliability visual motion stimuli tend to be affected by metaphorically or physically congruent or incongruent sounds. In the present study, we examined the modulatory effects of auditory pitch on visual perception of motion trajectory for visual inputs of varying reliability. Our results indicated that an auditory pitch implying the illusory motion toward the outside of the visual field-modulated perceived motion trajectory. In contrast, auditory pitch implying the illusory motion toward the central visual field did not affect the perception of motion trajectory. This asymmetrical effect of auditory stimuli occurred depending on the reliability of the visual input. Moreover, sounds that corresponded in terms of their pitch-elevation mapping altered the perception of the trajectory of visual motion when apparent motion could be perceived smoothly. Therefore, the present results demonstrate that auditory stimuli modulate visual motion perception especially when smooth motion is perceived in the peripheral visual field.

The Time-Course of the Cross-Modal Semantic Modulation of Visual Picture Processing by Naturalistic Sounds and Spoken Words

The time-course of cross-modal semantic interactions between pictures and either naturalistic sounds or spoken words was compared. Participants performed a speeded picture categorization task while hearing a task-irrelevant auditory stimulus presented at various stimulus onset asynchronies (SOAs) with respect to the visual picture. Both naturalistic sounds and spoken words gave rise to cross-modal semantic congruency effects (i.e., facilitation by semantically congruent sounds and inhibition by semantically incongruent sounds, as compared to a baseline noise condition) when the onset of the sound led that of the picture by 240 ms or more. Both naturalistic sounds and spoken words also gave rise to inhibition irrespective of their semantic congruency when presented within 106 ms of the onset of the picture. The peak of this cross-modal inhibitory effect occurred earlier for spoken words than for naturalistic sounds. These results therefore demonstrate that the semantic priming of visual picture categorization by auditory stimuli only occurs when the onset of the sound precedes that of the visual stimulus. The different time-courses observed for naturalistic sounds and spoken words likely reflect the different processing pathways to access the relevant semantic representations.

Modulation of affective face processing deficits in Schizophrenia by congruent emotional sounds

Schizophrenia is a psychiatric disorder resulting in prominent impairments in social functioning. Thus, clinical research has focused on underlying deficits of emotion processing and their linkage to specific symptoms and neurobiological dysfunctions. Although there is substantial research investigating impairments in unimodal affect recognition, studies in schizophrenia exploring crossmodal emotion processing are rare. Therefore, event-related potentials were measured in 15 patients with schizophrenia and 15 healthy controls while rating the expression of happy, fearful and neutral faces and concurrently being distracted by emotional or neutral sounds. Compared with controls, patients with schizophrenia revealed significantly decreased P1 and increased P2 amplitudes in response to all faces, independent of emotion or concurrent sound. Analyzing these effects with regard to audiovisual (in)congruence revealed that P1 amplitudes in patients were only reduced in response to emotionally incongruent stimulus pairs, whereas similar amplitudes between groups could be observed for congruent conditions. Correlation analyses revealed a significant negative correlation between general symptom severity (Brief Psychiatric Rating Scale-V4) and P1 amplitudes in response to congruent audiovisual stimulus pairs. These results indicate that early visual processing deficits in schizophrenia are apparent during emotion processing but, depending on symptom severity, these deficits can be restored by presenting concurrent emotionally congruent sounds.

Of cats and women: Temporal dynamics in the right temporoparietal cortex reflect auditory categorical processing of vocalizations

Understanding the temporal dynamics underlying cortical processing of auditory categories is complicated by difficulties in equating temporal and spectral features across stimulus classes. In the present magnetoencephalography (MEG) study, female voices and cat sounds were filtered so as to match in most of their acoustic properties, and the respective auditory evoked responses were investigated with a paradigm that allowed us to examine auditory cortical processing of two natural sound categories beyond the physical make-up of the stimuli. Three cat or human voice sounds were first presented to establish a categorical context. Subsequently, a probe sound that was congruent, incongruent, or ambiguous to this context was presented. As an index of a categorical mismatch, MEG responses to incongruent sounds were stronger than the responses to congruent sounds at ~250 ms in the right temporoparietal cortex, regardless of the sound category. Furthermore, probe sounds that could not be unambiguously attributed to any of the two categories ("cat" or "voice") evoked stronger responses after the voice than cat context at 200-250 ms, suggesting a stronger contextual effect for human voices. Our results suggest that categorical templates for human and animal vocalizations are established at ~250 ms in the right temporoparietal cortex, likely reflecting continuous online analysis of spectral stimulus features during auditory categorizing task.