Audiovisual Asynchrony Detection and Speech Intelligibility in Noise With Moderate to Severe Sensorineural Hearing Impairment

Abstract

The objective of this study is to explore the sensitivity to intermodal asynchrony in audiovisual speech with moderate to severe sensorineural hearing loss. Based on previous studies, two opposing expectations were an increase in sensitivity, as hearing-impaired listeners heavily rely on lipreading in daily life, and a reduction in sensitivity, as hearing-impaired listeners tend to be elderly and advanced age could potentially impair audiovisual integration. Adults with normal (N = 11, ages between 23 and 50 yrs) and impaired hearing (N = 11, ages between 54 and 81 yrs, the pure-tone average between 42 and 67 dB HL) participated in two experiments. In the first experiment, the synchrony judgments were recorded for varying intermodal time differences in audiovisual sentence recordings. In the second experiment, the intelligibility of audiovisual and audio-only speech was measured in speech-shaped noise, and correlations were explored between the synchrony window and intelligibility scores for individual listeners. Similar to previous studies, a sensitivity window on the order of a few hundred milliseconds was observed with all listeners. The average window shapes did not differ between normal-hearing and hearing-impaired groups; however, there was large individual variability. Individual windows were quantified by Gaussian curve fitting. Point of subjective simultaneity, a measure of window peak shift from the actual synchrony point, and full-width at half-maximum, a measure of window duration, were not correlated with participant's age or the degree of hearing loss. Points of subjective simultaneity were also not correlated with speech intelligibility scores. A moderate negative correlation that was significant at most conditions was observed between the full-width at half-maximum values and intelligibility scores. Contrary to either expectation per se, there was no indication of an effect of hearing impairment or age on the sensitivity to intermodal asynchrony in audiovisual speech. It is possible that the negative effects of aging were balanced with the positive effects of increased sensitivity due to reliance on visual cues with hearing impairment. The listeners, normal hearing or hearing impaired, who were more sensitive to asynchrony (with narrower synchrony windows) tended to understand speech in noise better, with both audio-only and audiovisual speech. The practical implication of the results is that delays in audio or video signals of communication systems would affect hearing-impaired listeners in a manner similar to normal-hearing listeners, and due to the importance of visual cues for the hearing-impaired listeners, special attention should be given to limit these delays.