Auditory-only Research Articles

Auditory Event-Related Potentials (ERPs) in Audiovisual Speech Perception

It has recently been reported (e.g., V. van Wassenhove, K. W. Grant, & D. Poeppel, 2005) that audiovisual (AV) presented speech is associated with an N1/P2 auditory event-related potential (ERP) response that is lower in peak amplitude compared with the responses associated with auditory only (AO) speech. This effect was replicated. Further comparisons were made between ERP responses to AV speech in which the visual and auditory components were in or out of synchrony, to test whether the effect is associated with the operation of integration mechanisms, as has been claimed, or occurs because of other factors such as attention. ERPs were recorded from participants presented with recordings of unimodal or AV speech syllables in a detection task. Comparisons were made between AO and AV speech and between synchronous and asynchronous AV speech. Synchronous AV speech produced an N1/P2 with lower peak amplitudes than with AO speech, unaccounted for by linear superposition of visually evoked responses onto auditory-evoked responses. Asynchronous AV speech produced no amplitude reduction. The dependence of N1/P2 amplitude reduction on AV synchrony validates it as an electrophysiological marker of AV integration.

Cross-modal interactions in the experience of musical performances: Physiological correlates

This experiment was conducted to investigate cross-modal interactions in the emotional experience of music listeners. Previous research showed that visual information present in a musical performance is rich in expressive content, and moderates the subjective emotional experience of a participant listening and/or observing musical stimuli [Vines, B. W., Krumhansl, C. L., Wanderley, M. M., & Levitin, D. J. (2006). Cross-modal interactions in the perception of musical performance. Cognition, 101, 80--113.]. The goal of this follow-up experiment was to replicate this cross-modal interaction by investigating the objective, physiological aspect of emotional response to music measuring electrodermal activity. The scaled average of electrodermal amplitude for visual–auditory presentation was found to be significantly higher than the sum of the reactions when the music was presented in visual only (VO) and auditory only (AO) conditions, suggesting the presence of an emergent property created by bimodal interaction. Functional data analysis revealed that electrodermal activity generally followed the same contour across modalities of presentation, except during rests (silent parts of the performance) when the visual information took on particular salience. Finally, electrodermal activity and subjective tension judgments were found to be most highly correlated in the audio-visual (AV) condition than in the unimodal conditions. The present study provides converging evidence for the importance of seeing musical performances, and preliminary evidence for the utility of electrodermal activity as an objective measure in studies of continuous music-elicited emotions.

The effect of viewing speech on auditory speech processing is different in the left and right hemispheres

We used whole-head magnetoencephalograpy (MEG) to record changes in neuromagnetic N100m responses generated in the left and right auditory cortex as a function of the match between visual and auditory speech signals. Stimuli were auditory-only (AO) and auditory–visual (AV) presentations of /pi/, /ti/ and /vi/. Three types of intensity matched auditory stimuli were used: intact speech (Normal), frequency band filtered speech (Band) and speech-shaped white noise (Noise). The behavioural task was to detect the /vi/ syllables which comprised 12% of stimuli. N100m responses were measured to averaged /pi/ and /ti/ stimuli. Behavioural data showed that identification of the stimuli was faster and more accurate for Normal than for Band stimuli, and for Band than for Noise stimuli. Reaction times were faster for AV than AO stimuli. MEG data showed that in the left hemisphere, N100m to both AO and AV stimuli was largest for the Normal, smaller for Band and smallest for Noise stimuli. In the right hemisphere, Normal and Band AO stimuli elicited N100m responses of quite similar amplitudes, but N100m amplitude to Noise was about half of that. There was a reduction in N100m for the AV compared to the AO conditions. The size of this reduction for each stimulus type was same in the left hemisphere but graded in the right (being largest to the Normal, smaller to the Band and smallest to the Noise stimuli). The N100m decrease for the Normal stimuli was significantly larger in the right than in the left hemisphere. We suggest that the effect of processing visual speech seen in the right hemisphere likely reflects suppression of the auditory response based on AV cues for place of articulation.

Auditory and visual biasing for sentence recognition in noise

Previous research [A. E. Carney et al., J. Acoust. Soc. Am. 106, 2270 (1999)] showed that listeners have a range of visual biasing when presented with stimuli that have conflicting auditory and visual cues (e.g., visual /gi/ and /auditory /bi/). Further, individuals with strong visual bias had substantially changed labeling for synthetic continua with visual cues added in noise conditions, while those with auditory biases did not [B. R. Clement and A. E. Carney, J. Acoust. Soc. Am. 106, 2271 (1999)]. The purpose of the current study was to determine if listeners with strong auditory or visual biases had different patterns of auditory sentence recognition in noise with and without added visual cues. Two groups of ten listeners (one with auditory biases and one with visual biases on a screening task) were presented with lists of CID sentences at varying signal-to-noise ratios in auditory-only (AO) and auditory + visual (AV) conditions. Speech reception thresholds for both groups improved in the AV condition; visually biased listeners did show somewhat greater improvement when visual cues were added. In both groups, some listeners were much more adversely affected by the addition of noise than others. Results will be discussed in terms of individual differences among listeners. [For Speech Technical Committee Best Student Paper Award.]

Auditory speech detection in noise enhanced by lipreading

Audiovisual speech stimuli have been shown to produce a variety of perceptual phenomena. Enhanced detectability of acoustic speech in noise, when the talker can also be seen, is one of those phenomena. This study investigated whether this enhancement effect is specific to visual speech stimuli or can rely on more generic non-speech visual stimulus properties. Speech detection thresholds for an auditory /ba/ stimulus were obtained in a white noise masker. The auditory /ba/ was presented adaptively to obtain its 79.4% detection threshold under five conditions. In Experiment 1, the syllable was presented (1) auditory-only (AO) and (2) as audiovisual speech (AVS), using the original video recording. Three types of synthetic visual stimuli were also paired synchronously with the audio token: (3) A dynamic Lissajous (AVL) figure whose vertical extent was correlated with the acoustic speech envelope; (4) a dynamic rectangle (AVR) whose horizontal extent was correlated with the speech envelope; and (5) a static rectangle (AVSR) whose onset and offset were synchronous with the acoustic speech onset and offset. Ten adults with normal hearing and vision participated. The results, in terms of dB signal-to-noise ratio (SNR), were AVS < (AVL ≈ AVR ≈ ASR) < AO. That is, AVS was significantly easiest to detect, there was no difference among the synthesized visual stimuli, and all audiovisual conditions resulted in significantly lower thresholds than AO. To determine the advantage of the AVS stimulus, in Experiment 2, a preliminary mouth gesture was edited from the video speech token. This manipulation defeated the advantage for both the original and the edited AVS stimulus, while the audiovisual detection enhancement persisted. Overall, the results showed enhanced auditory speech detection with visual stimuli but no advantage for a fine-grained correlation between acoustic and optical speech signals.

Auditory-visual context effects on the perception of voicing

Perception of an auditory continuum can be changed by a preceding visual context [K. Green and L. Norrix, in press, JEP: HPP]. This study examines if a visual context for /l/, as in /abla/, influences the voicing distinction for an auditory /aba/-/apa/ continuum. Voice onset time (VOT) and F1 onset frequency were measured during production of /aba/, /apa/ (singleton conditions) and /abla/, /apla/ (cluster conditions). Findings indicate longer VOTs in /apla/ compared to /apa/ and lower F1 onset frequencies for the cluster compared to singleton conditions. An auditory-only (AO) continuum (/aba/–/apa/) was created and presented to listeners for identification. In an auditory-visual (AV) condition each auditory token was paired with a face saying /abla/ and presented for identification. If subjects use knowledge about coarticulation and its effects on VOT during production, then they should make more /b/ responses in the AV compared to AO condition. In contrast, if they recover F1 information from the visual articulation, then more /p/ responses should be obtained. Findings indicate more /p/ responses in the AV compared to AO condition, suggesting that perceivers are recovering information from visual articulations (perhaps tongue height) rather than using stored knowledge about coarticulatory effects on VOT. [Work supported by NSF.]

Confidence ratings in auditory–visual speech perception

In an earlier report, Carney et al. [J. Acoust. Soc. Am. 106, 2270 (1999)] showed that the perception of the McGurk effect, the result of a visual syllable biasing the perception of a simultaneously presented auditory syllable, was affected by the use of multiple talkers. In addition, individual listeners could be strongly auditory in their perception, even when a visually biasing stimulus was present. During the original experiment, listeners were asked to rate each of their perceptions in the auditory-only (AO), visual-only (VO), and auditory–visual (AV) modes for syllabic stimuli. Listeners were very confident of their perceptions for bilabial stimuli in the VO mode but not for alveolar or velar stimuli. Confidence ratings were also high for all AO stimuli and for AV stimuli for matching visual–auditory tokens. However, confidence ratings varied for tokens with mismatched A and V stimuli. Some individual listeners were always confident in their perception regardless of speaker or condition. Other listeners varied in confidence with speaker and condition. Across listeners, certain talkers evoked different ratings of confidence as well. Results support the notion of a graded perception of bimodal stimuli. [Work supported by NIDCD.]

Cross-modal context effects on the perception of /r/ and /l/ in a speech and nonspeech mode

Norrix and Green [J. Acoust. Soc. Am. 99, 2591–2592 (1996)] provided evidence for cross-modal context effects on the perception of /r/ and /l/ in a stop cluster. Tokens from a synthetic /iri–ili/ continuum were dubbed onto a visual /ibi/. When presented in an auditory-visual (AV) condition, the tokens were perceived as ranging from /ibri/ to /ibli/. Results indicated a reliable shift in the AV condition relative to an auditory-only (AO) condition. This shift was in accord with acoustic consequences of articulating /r/ and /l/ in a stop cluster. In the current study, sine-wave analogs of the /iri–ili/ tokens were constructed and presented to two groups of observers in an AO and AV condition. Group One was told they would hear schematic speech sounds and instructed to identify what they heard as /r/ or /l/. Group Two made up their own criteria for classifying the tokens as nonspeech sounds. Results indicated a reliable shift in the /r–l/ boundary between the AO and AV conditions for the speech group only and suggest that the influence of the visual articulatory context depends upon listeners interpreting the auditory tokens as speech. [Work supported by NIDCD, NIH.]

Auditory‐visual context effects on the perception of /r/ and /l/ in a stop cluster.

Context effects in speech perception are thought to reflect knowledge about coarticulatory influences from the surrounding phonetic environment. This study investigated if such effects occur when the context is specified in the visual modality and segmental information in the auditory modality. In the first experiment, two continua were synthesized: one varying from /iri–ili/ and the other from /ibri–ibli/. When presented to listeners for identification as /r/ or /l/, there was a significant shift in the category boundary between the two continua. In a second experiment, the /iri–ili/ tokens were paired with visual tokens of a talker saying /ibi/. When presented in an auditory‐visual (AV) condition, these tokens were perceived as ranging from /ibri/ to /ibli/. Observers also identified the /iri–ili/ tokens in a separate auditory‐only (AO) condition. Results indicated a similar shift in the /r–l/ boundary between the AO and AV conditions. Analysis of /r/ and /l/ productions revealed that the perceptual adjustments were consistent with the acoustic consequences of articulating /r/ and /l/ in a stop cluster. The findings suggest that the perceptual adjustment reflects cross‐modal knowledge of the coarticulatory effects of a bilabial consonant on the acoustic realization of /r/ and /l/. [Work supported by NIDCD, NIH.]