Audiovisual Congruence Research Articles

Audiovisual Congruence and Localization Performance in Virtual Reality: 3D Loudspeaker Model vs. Human Avatar

This paper investigates audiovisual congruence in virtual reality with both horizontal and vertical offsets between audio and visual rendering. Audiovisual congruence and localization errors are assessed using loudspeaker playback and nonindividualized headphone rendering. To account for the influence of different types of visual information on congruence, presentations of a loudspeaker model and 3D human avatar were compared. Therefore, a new dataset of audiovisual speech was recorded. Results show that human avatar rendering increases perceived congruence, and experienced listeners have an increased tendency to respond with “incongruent” when a loudspeaker model is shown but not when the human avatar is presented. Moreover, a correlation is found between localization precision and audiovisual congruence for horizontally offset stimuli and avatar presentation. For vertical offsets, the angular range of congruence is generally large, and localization errors are high, so no correlation can be observed between the two. The paper contributes congruence ranges for audiovisual speech in virtual reality, which also has implications for augmented reality telepresence use.

The role of audiovisual congruence in aesthetic appreciation of contemporary music and visual art

Does congruence between auditory and visual modalities affect aesthetic experience? While cross-modal correspondences between vision and hearing are well-documented, previous studies show conflicting results regarding whether audiovisual correspondence affects subjective aesthetic experience. Here, in collaboration with the Kentler International Drawing Space (NYC, USA), we depart from previous research by using music specifically composed to pair with visual art in the professionally-curated Music as Image and Metaphor exhibition. Our pre-registered online experiment consisted of 4 conditions: Audio, Visual, Audio-Visual-Intended (artist-intended pairing of art/music), and Audio-Visual-Random (random shuffling). Participants (N = 201) were presented with 16 pieces and could click to proceed to the next piece whenever they liked. We used time spent as an implicit index of aesthetic interest. Additionally, after each piece, participants were asked about their subjective experience (e.g., feeling moved). We found that participants spent significantly more time with Audio, followed by Audiovisual, followed by Visual pieces; however, they felt most moved in the Audiovisual (bi-modal) conditions. Ratings of audiovisual correspondence were significantly higher for the Audiovisual-Intended compared to Audiovisual-Random condition; interestingly, though, there were no significant differences between intended and random conditions on any other subjective rating scale, or for time spent. Collectively, these results call into question the relationship between cross-modal correspondence and aesthetic appreciation. Additionally, the results complicate the use of time spent as an implicit measure of aesthetic experience.

The Jena Audiovisual Stimuli of Morphed Emotional Pseudospeech (JAVMEPS): A database for emotional auditory-only, visual-only, and congruent and incongruent audiovisual voice and dynamic face stimuli with varying voice intensities.

We describe JAVMEPS, an audiovisual (AV) database for emotional voice and dynamic face stimuli, with voices varying in emotional intensity. JAVMEPS includes 2256 stimulus files comprising (A) recordings of 12speakers, speaking four bisyllabic pseudowords with six naturalistic induced basic emotions plus neutral, in auditory-only, visual-only, and congruent AV conditions. It furthermore comprises (B) caricatures (140%), original voices (100%), and anti-caricatures (60%) for happy, fearful, angry, sad, disgusted, and surprised voices for eight speakers and two pseudowords. Crucially, JAVMEPS contains (C) precisely time-synchronized congruent and incongruent AV (and corresponding auditory-only) stimuli with two emotions (anger, surprise), (C1) with original intensity (ten speakers, four pseudowords), (C2) and with graded AV congruence (implemented via five voice morph levels, from caricatures to anti-caricatures; eight speakers, two pseudowords). We collected classification data for Stimulus Set A from 22 normal-hearing listeners and four cochlear implant users, for two pseudowords, in auditory-only, visual-only, and AV conditions. Normal-hearing individuals showed good classification performance (McorrAV = .59 to .92), with classification rates in the auditory-only condition ≥ .38 correct (surprise: .67, anger: .51). Despite compromised vocal emotion perception, CI users performed above chance levels of .14 for auditory-only stimuli, with best rates for surprise (.31) and anger (.30). We anticipate JAVMEPS to become a useful open resource for researchers into auditory emotion perception, especially when adaptive testing or calibration of task difficulty is desirable. With its time-synchronized congruent and incongruent stimuli, JAVMEPS can also contribute to filling a gap in research regarding dynamic audiovisual integration of emotion perception via behavioral or neurophysiological recordings.

Eye responses reflect spatial congruency and perceptual bias in audiovisual interaction

Recent studies have reported that increased pupil dilation and reduced microsaccades are associated with perceptual task enhancement in the bisensory over unisensory conditions. However, little is known about the impacts of bimodal stimulus congruency on those eye responses. We adopted the spatial ventriloquist paradigm to investigate whether spatial congruency of audiovisual stimuli and differences in perceptual experience influence eye responses accompanied by behavioral responses. A simple beep sound and a single disk were used as auditory and visual stimuli respectively. We manipulated spatial congruency of audiovisual stimuli by presenting a visual stimulus on the left or right side of a monitor with an auditory stimulus from the left, right, or center of the monitor. 15 participants performed a sound localization task by clicking their perceived location of the auditory stimulus on the monitor. Participants maintained fixation on a small dot at the center of the monitor with their heads fixed on a chin rest, while their eye responses were recorded using an eye tracker (Eyelink-1000, SR Research). To examine differential perceptual experience in the audiovisual congruence conditions, we classified all responses of audiovisual trials into two groups based on whether the perceived location of the auditory stimulus biased toward the visual stimulus. We found greater pupil dilation accompanied by a greater ventriloquist effect (VE) in the spatially incongruent condition compared to the congruent condition. In addition, a decrease in the number of microsaccades was associated with the auditory perceptual bias towards visual stimuli in the spatially incongruent condition. Our results showed that both spatial congruency and perceptual bias modulate the VE. Results from eye responses suggest that pupillometry is a reliable marker for spatial congruency whereas microsaccades reflect the perceptual bias.

Relating Sound and Sight in Simulated Environments.

The auditory signals at the ear can be affected by components arriving both directly from a sound source and indirectly via environmental reverberation. Previous studies have suggested that the perceptual separation of these contributions can be aided by expectations of likely reverberant qualities. Here, we investigated whether vision can provide information about the auditory properties of physical locations that could also be used to develop such expectations. We presented participants with audiovisual stimuli derived from 10 simulated real-world locations via a head-mounted display (HMD; n = 44) or a web-based ( n = 60) delivery method. On each trial, participants viewed a first-person perspective rendering of a location before hearing a spoken utterance that was convolved with an impulse response that was from a location that was either the same as (congruent) or different to (incongruent) the visually-depicted location. We find that audiovisual congruence was associated with an increase in the probability of participants reporting an audiovisual match of about 0.22 (95% credible interval: [ 0.17 , 0.27 ]), and that participants were more likely to confuse audiovisual pairs as matching if their locations had similar reverberation times. Overall, this study suggests that human perceivers have a capacity to form expectations of reverberation from visual information. Such expectations may be useful for the perceptual challenge of separating sound sources and reverberation from within the signal available at the ear.

A Bayesian meta-analysis of infants' ability to perceive audio-visual congruence for speech.

This paper quantifies the extent to which infants can perceive audio-visual congruence for speech information and assesses whether this ability changes with native language exposure over time. A hierarchical Bayesian robust regression model of 92separate effect sizes extracted from 24studies indicates a moderate effect size in a positive direction (0.35, CI [0.21: 0.50]). This result suggests that infants possess a robust ability to detect audio-visual congruence for speech. Moderator analyses, moreover, suggest that infants' audio-visual matching ability for speech emerges at an early point in the process of language acquisition and remains stable for both native and non-native speech throughout early development. A sensitivity analysis of the meta-analytic data, however, indicates that a moderate publication bias for significant results could shift the lower credible interval to include null effects. Based on these findings, we outline recommendations for new lines of enquiry and suggest ways to improve the replicability of results in future investigations.

Effects of urgent speech and congruent/incongruent text on speech intelligibility for older adults in the presence of noise and reverberation

Public address (PA) announcements are widely used, but noise and reverberation can render them unintelligible, and such an environment tends to degrade speech intelligibility more for older adults (OAs) than for younger adults (YAs). Furthermore, in an emergency, textual information available to the public may not coincide with PA announcements, and this mismatch may also contribute to degrading the intelligibility of the announcement. In this study, speech spoken in a normal or urgent style and preceded by congruent or incongruent text was presented to OA participants to investigate the effects of these parameters on word intelligibility in the presence of babble noise and reverberation. This study also investigated age-related differences between OAs and YAs using the same stimuli. The results showed that, while YAs performed better than OAs, the effect of urgent speech did not significantly differ with age. Urgent speech was more intelligible than normal speech, and the congruent text condition was more intelligible than the incongruent and no text conditions. An exploratory analysis showed that urgent speech improved speech intelligibility for OAs when congruent text was available, but not when absent or incongruent. Without congruent information, this listening test may have been too demanding for OAs, who could have had more difficulty processing the fixed timeline of the listening test, the faster urgent speech relative to normal speech, and informational masking because of the slowing of cognitive processing with age. A correlation was found between the average word correct rate for urgent speech and average audiometric thresholds; on the other hand, increased fundamental frequencies in urgent compared with normal speech was not the main predictor of the intelligibility of urgent speech among the OAs. The congruent text benefit in urgent speech was larger for OAs than for YAs in the previous study using the same stimuli. This demonstrates that compared with YAs, OAs may rely more on supportive prior knowledge or receive more benefit from the preceding text in challenging listening environments. These findings imply that age-related cognitive decline may explain the difficulties in speech perception in the presence of noise and reverberation among OAs; however, no definitive conclusions can be drawn. These results suggest that simple combinations of speaking style and textual information affect the intelligibility of emergency PA announcements among OAs, and thus, audiovisual congruence should be considered when announcements are made in public spaces.

Audio-visual interactions enhance soundscape perception in China’s protected areas

Coordination of sound and view management is essential to construct tranquility and enhance the visitor experience in protected areas. However, research on audio-visual interactions has historically primarily focused on urban environments, revealing a significant gap in protected areas. In China, soundscape research is urgently needed in protected areas to preserve the long tradition of soundscape appreciation and the high social and ecological values of soundscapes under overwhelming tourism pressure. Using the Wulingyuan World Heritage Site as the case study and through the lens of ten semantic differentials (SDs), this article employed a lab experiment to explore the influence of audio-visual interactions on soundscape assessment to inform the management of protected areas. Our results show that, regardless of the soundscape composition (i.e., primarily natural vs. mixed vs. anthropogenic), the two types of visual stimuli tested, which either matches or mismatches the sound sampling sites, generally led to higher soundscape assessment than when no visual stimuli were present. This positive effect of audio-visual interaction on soundscape assessment, however, would potentially be offset by the adverse visual impacts of overly artificial non-matching scenes. The nature of the sound, characteristics of the visual stimuli, and the personality dimension of extraversion were significantly associated with soundscape assessment. Different SDs exhibited varied influences from visual vs. audio stimuli. In particular, irritating/happy and ugly/beautiful showed the highest sensitivity to visual stimuli, whereas artificial/natural was mainly determined by the sound characteristics.These results call for multi-scale landscape interventions to address the increasingly prevalent low audio-visual congruence in China’s protected areas. Identifying critical mismatches between the visual and audio environment and developing coordinated management strategies will be essential to preserving the tranquility of protected areas and improving public perception.

Speech-specific audiovisual integration modulates induced theta-band oscillations.

Speech perception is influenced by vision through a process of audiovisual integration. This is demonstrated by the McGurk illusion where visual speech (for example /ga/) dubbed with incongruent auditory speech (such as /ba/) leads to a modified auditory percept (/da/). Recent studies have indicated that perception of the incongruent speech stimuli used in McGurk paradigms involves mechanisms of both general and audiovisual speech specific mismatch processing and that general mismatch processing modulates induced theta-band (4–8 Hz) oscillations. Here, we investigated whether the theta modulation merely reflects mismatch processing or, alternatively, audiovisual integration of speech. We used electroencephalographic recordings from two previously published studies using audiovisual sine-wave speech (SWS), a spectrally degraded speech signal sounding nonsensical to naïve perceivers but perceived as speech by informed subjects. Earlier studies have shown that informed, but not naïve subjects integrate SWS phonetically with visual speech. In an N1/P2 event-related potential paradigm, we found a significant difference in theta-band activity between informed and naïve perceivers of audiovisual speech, suggesting that audiovisual integration modulates induced theta-band oscillations. In a McGurk mismatch negativity paradigm (MMN) where infrequent McGurk stimuli were embedded in a sequence of frequent audio-visually congruent stimuli we found no difference between congruent and McGurk stimuli. The infrequent stimuli in this paradigm are violating both the general prediction of stimulus content, and that of audiovisual congruence. Hence, we found no support for the hypothesis that audiovisual mismatch modulates induced theta-band oscillations. We also did not find any effects of audiovisual integration in the MMN paradigm, possibly due to the experimental design.

The organization and reorganization of audiovisual speech perception in the first year of life

The period between six and 12 months is a sensitive period for language learning during which infants undergo auditory perceptual attunement, and recent results indicate that this sensitive period may exist across sensory modalities. We tested infants at three stages of perceptual attunement (six, nine, and 11 months) to determine (1) whether they were sensitive to the congruence between heard and seen speech stimuli in an unfamiliar language, and (2) whether familiarization with congruent audiovisual speech could boost subsequent non-native auditory discrimination. Infants at six- and nine-, but not 11- months, detected audiovisual congruence of non-native syllables. Familiarization to incongruent, but not congruent, audiovisual speech changed auditory discrimination at test for six-month-olds but not nine- or 11-month-olds. These results advance the proposal that speech perception is audiovisual from early in ontogeny, and that the sensitive period for audiovisual speech perception may last somewhat longer than that for auditory perception alone.

Synchronization and periodicity of auditory and visual accents affect perceived congruence between sound and moving pictures

Continuous rating experiments focused on the congruence between moving pictures and sounds have shown that synchronization between auditory and visual accents is fundamental to audiovisual congruence. We conducted experiments with snare sounds as auditory accents and changes in the color of a full screen as visual accents. We found that the periodicity of the auditory and visual accents contributed to the audiovisual congruence, in addition to the synchronization between auditory and visual accents. We found the highest level of congruence when the accents were periodic and frequently synchronized. Conversely, the congruence level was lower when the auditory and visual accents were less frequently synchronized. The contribution of auditory periodicity to the perceived congruence was greater than that of visual periodicity. Additionally, when we used stimuli consisting of synchronized and asynchronized audiovisual elements, we observed an effect of context on congruence with respect to continuous ratings. When we presented a stream of asynchronized audiovisual accents after a synchronized stream, the level of high congruence was maintained for a short period. In contrast, just after presentation of a steam of asynchronized accents, we found that the congruence level of a stream of synchronized accents

Infants' preference for native audiovisual speech dissociated from congruency preference.

Although infant speech perception in often studied in isolated modalities, infants' experience with speech is largely multimodal (i.e., speech sounds they hear are accompanied by articulating faces). Across two experiments, we tested infants’ sensitivity to the relationship between the auditory and visual components of audiovisual speech in their native (English) and non-native (Spanish) language. In Experiment 1, infants’ looking times were measured during a preferential looking task in which they saw two simultaneous visual speech streams articulating a story, one in English and the other in Spanish, while they heard either the English or the Spanish version of the story. In Experiment 2, looking times from another group of infants were measured as they watched single displays of congruent and incongruent combinations of English and Spanish audio and visual speech streams. Findings demonstrated an age-related increase in looking towards the native relative to non-native visual speech stream when accompanied by the corresponding (native) auditory speech. This increase in native language preference did not appear to be driven by a difference in preference for native vs. non-native audiovisual congruence as we observed no difference in looking times at the audiovisual streams in Experiment 2.

Hearing (rivaling) lips and seeing voices: how audiovisual interactions modulate perceptual stabilization in binocular rivalry.

In binocular rivalry (BR), sensory input remains the same yet subjective experience fluctuates irremediably between two mutually exclusive representations. We investigated the perceptual stabilization effect of an additional sound on the BR dynamics using speech stimuli known to involve robust audiovisual (AV) interactions at several cortical levels. Subjects sensitive to the McGurk effect were presented looping videos of rivaling faces uttering /aba/ and /aga/, respectively, while synchronously hearing the voice /aba/. They reported continuously the dominant percept, either observing passively or trying actively to promote one of the faces. The few studies that investigated the influence of information from an external modality on perceptual competition reported results that seem at first sight inconsistent. Since these differences could stem from how well the modalities matched, we addressed this by comparing two levels of AV congruence: real (/aba/ viseme) vs. illusory (/aga/ viseme producing the /ada/ McGurk fusion). First, adding the voice /aba/ stabilized both real and illusory congruent lips percept. Second, real congruence of the added voice improved volitional control whereas illusory congruence did not, suggesting a graded contribution to the top-down sensitivity control of selective attention. In conclusion, a congruent sound enhanced considerably attentional control over the perceptual outcome selection; however, differences between passive stabilization and active control according to AV congruency suggest these are governed by two distinct mechanisms. Based on existing theoretical models of BR, selective attention and AV interaction in speech perception, we provide a general interpretation of our findings.

No counterpart of visual perceptual echoes in the auditory system.

It has been previously demonstrated by our group that a visual stimulus made of dynamically changing luminance evokes an echo or reverberation at ∼10 Hz, lasting up to a second. In this study we aimed to reveal whether similar echoes also exist in the auditory modality. A dynamically changing auditory stimulus equivalent to the visual stimulus was designed and employed in two separate series of experiments, and the presence of reverberations was analyzed based on reverse correlations between stimulus sequences and EEG epochs. The first experiment directly compared visual and auditory stimuli: while previous findings of ∼10 Hz visual echoes were verified, no similar echo was found in the auditory modality regardless of frequency. In the second experiment, we tested if auditory sequences would influence the visual echoes when they were congruent or incongruent with the visual sequences. However, the results in that case similarly did not reveal any auditory echoes, nor any change in the characteristics of visual echoes as a function of audio-visual congruence. The negative findings from these experiments suggest that brain oscillations do not equivalently affect early sensory processes in the visual and auditory modalities, and that alpha (8–13 Hz) oscillations play a special role in vision.

Modulation of affective face processing deficits in Schizophrenia by congruent emotional sounds

Schizophrenia is a psychiatric disorder resulting in prominent impairments in social functioning. Thus, clinical research has focused on underlying deficits of emotion processing and their linkage to specific symptoms and neurobiological dysfunctions. Although there is substantial research investigating impairments in unimodal affect recognition, studies in schizophrenia exploring crossmodal emotion processing are rare. Therefore, event-related potentials were measured in 15 patients with schizophrenia and 15 healthy controls while rating the expression of happy, fearful and neutral faces and concurrently being distracted by emotional or neutral sounds. Compared with controls, patients with schizophrenia revealed significantly decreased P1 and increased P2 amplitudes in response to all faces, independent of emotion or concurrent sound. Analyzing these effects with regard to audiovisual (in)congruence revealed that P1 amplitudes in patients were only reduced in response to emotionally incongruent stimulus pairs, whereas similar amplitudes between groups could be observed for congruent conditions. Correlation analyses revealed a significant negative correlation between general symptom severity (Brief Psychiatric Rating Scale-V4) and P1 amplitudes in response to congruent audiovisual stimulus pairs. These results indicate that early visual processing deficits in schizophrenia are apparent during emotion processing but, depending on symptom severity, these deficits can be restored by presenting concurrent emotionally congruent sounds.

The effect of lip-reading on primary stream segregation

Lip-reading has been shown to improve the intelligibility of speech in multitalker situations, where auditory stream segregation naturally takes place. This study investigated whether the benefit of lip-reading is a result of a primary audiovisual interaction that enhances the obligatory streaming mechanism. Two behavioral experiments were conducted involving sequences of French vowels that alternated in fundamental frequency. In Experiment 1, subjects attempted to identify the order of items in a sequence. In Experiment 2, subjects attempted to detect a disruption to temporal isochrony across alternate items. Both tasks are disrupted by streaming, thus providing a measure of primary or obligatory streaming. Visual lip gestures articulating alternate vowels were synchronized with the auditory sequence. Overall, the results were consistent with the hypothesis that visual lip gestures enhance segregation by affecting primary auditory streaming. Moreover, increases in the naturalness of visual lip gestures and auditory vowels, and corresponding increases in audiovisual congruence may potentially lead to increases in the effect of visual lip gestures on streaming.

Transitions in neural oscillations reflect prediction errors generated in audiovisual speech

According to the predictive coding theory, top-down predictions are conveyed by backward connections and prediction errors are propagated forward across the cortical hierarchy. Using MEG in humans, we show that violating multisensory predictions causes a fundamental and qualitative change in both the frequency and spatial distribution of cortical activity. When visual speech input correctly predicted auditory speech signals, a slow delta regime (3-4 Hz) developed in higher-order speech areas. In contrast, when auditory signals invalidated predictions inferred from vision, a low-beta (14-15 Hz) / high-gamma (60-80 Hz) coupling regime appeared locally in a multisensory area (area STS). This frequency shift in oscillatory responses scaled with the degree of audio-visual congruence and was accompanied by increased gamma activity in lower sensory regions. These findings are consistent with the notion that bottom-up prediction errors are communicated in predominantly high (gamma) frequency ranges, whereas top-down predictions are mediated by slower (beta) frequencies.

Advancing age alters the influence of eye position on sound localization

Vision and audition provide spatial information about the environment to guide natural behavior. Because the eyes move in the head while the ears remain head-fixed, input conveying eye position in the head is required to maintain audiovisual congruence. Human perception of auditory space was previously shown to shift with changes in eye position, regardless of the target's frequency content and spatial cues underlying horizontal and vertical localization. In this study, we examined whether this interaction is altered by advancing age. Head-restrained young (18-44 yo), middle-aged (45-64 yo), and elderly (65-81 yo) human subjects localized noise bursts under conditions of transient and sustained ocular deflection. All three age groups demonstrated a time-dependent shift of auditory space in the direction of eye position. Moreover, this adaptation showed a clear decline with advancing age, but only for peripheral auditory space (beyond ±10° from midline). Alternatively, adaptation in the periphery may occur, but is more sluggish than in the central field and therefore not fully observed in this experiment. The age-dependent effect cannot be readily explained by senescent peripheral hearing loss, suggesting a change in central processing of auditory space in relation to the control of gaze.