Impact of Audio-Visual Asynchrony on Lip-Reading Effects -Neuromagnetic and Psychophysical Study.

Tetsuaki Kawase,Shiho Takata,Nobukazu Nakasato,Yukio Katori,Yoshitaka Takanashi,Izumi Yahata,Ryuta Kawashima,Akitake Kanno,Shuichi Sakamoto,Christian Friedrich Altmann

doi:10.1371/journal.pone.0168740

Abstract

The effects of asynchrony between audio and visual (A/V) stimuli on the N100m responses of magnetoencephalography in the left hemisphere were compared with those on the psychophysical responses in 11 participants. The latency and amplitude of N100m were significantly shortened and reduced in the left hemisphere by the presentation of visual speech as long as the temporal asynchrony between A/V stimuli was within 100 ms, but were not significantly affected with audio lags of -500 and +500 ms. However, some small effects were still preserved on average with audio lags of 500 ms, suggesting similar asymmetry of the temporal window to that observed in psychophysical measurements, which tended to be more robust (wider) for audio lags; i.e., the pattern of visual-speech effects as a function of A/V lag observed in the N100m in the left hemisphere grossly resembled that in psychophysical measurements on average, although the individual responses were somewhat varied. The present results suggest that the basic configuration of the temporal window of visual effects on auditory-speech perception could be observed from the early auditory processing stage.

Highlights

Visual speech information synchronously presented with speech sound can affect speech perception
The present study examined the effects of asynchrony between audio and visual stimuli on the early auditory evoked fields (AEFs) using whole head MEG [27,28,29,30,31,32,33,34,35,36], focusing on the visual effects on the N100m originating from the left auditory cortex which is the dominant hemisphere in A/V coupling [8,9,10,11,12,13,14, 22, 37]
The fusion rate was remarkably reduced with temporal asynchrony of 500 ms (A/V with -500 ms or +500 ms audio lag), but the temporal window tended to be more robust for audio lags

Summary

Introduction

Visual speech information synchronously presented with speech sound can affect speech perception. Visual speech information, such as the speaker’s face uttering the speech sound, is important in the perceptual process of auditory input in individuals with impaired hearing, as perception of ambiguous speech sound can become clearer with the congruent visual speech information [1, 2]. If the visual speech information is incongruent to the speech sounds, perception of the audio speech may be affected. Presentation of the /be/ sound (audio) with visual /ge/ (speaker’s face uttering the /ge/ sound), resulting in incongruent visual stimulus, often results

Methods

Results

Conclusion