Detection of auditory (cross-spectral) and auditory–visual (cross-modal) synchrony

Abstract

Detection thresholds for temporal synchrony in auditory and auditory–visual sentence materials were obtained on normal-hearing subjects. For auditory conditions, thresholds were determined using an adaptive-tracking procedure to control the degree of temporal asynchrony of a narrow audio band of speech, both positive and negative in separate tracks, relative to three other narrow audio bands of speech. For auditory–visual conditions, thresholds were determined in a similar manner for each of four narrow audio bands of speech as well as a broadband speech condition, relative to a video image of a female speaker. Four different auditory filter conditions, as well as a broadband auditory–visual speech condition, were evaluated in order to determine whether detection thresholds were dependent on the spectral content of the acoustic speech signal. Consistent with previous studies of auditory–visual speech recognition which showed a broad, asymmetrical range of temporal synchrony for which intelligibility was basically unaffected (audio delays roughly between −40 ms and +240 ms), auditory–visual synchrony detection thresholds also showed a broad, asymmetrical pattern of similar magnitude (audio delays roughly between −45 ms and +200 ms). No differences in synchrony thresholds were observed for the different filtered bands of speech, or for broadband speech. In contrast, detection thresholds for audio-alone conditions were much smaller (between −17 ms and +23 ms) and symmetrical. These results suggest a fairly tight coupling between a subject’s ability to detect cross-spectral (auditory) and cross-modal (auditory–visual) asynchrony and the intelligibility of auditory and auditory–visual speech materials.