Speechreading enhancement using a sinusoidal substitute for voice fundamental frequency

Abstract

This study compared the effectiveness of three acoustic supplements to speechreading: the low-pass-filtered output of an electroglottograph, a variable-frequency sinusoidal substitute for voice fundamental frequency (FO), and a constant-frequency sinusoidal substitute that served as a representation of voicing. Both sinusoidal signals were synthesized at constant amplitude during periods of voicing. The sinusoidal signals were prepared off-line by a combination of automatic and manual estimation of the FO contours of video-recorded sentences. These signals were then resynchronized with the audio portions of the original recording. In 12 normally-hearing adults, the electroglottograph signal and the variable-frequency sinusoidal FO substitute both increased the number of words recognized in sentences of known topic by between 30 and 35 percentage points. The magnitude of this effect was greater for longer sentences but independent of basic speechreading ability. The constant-frequency substitute provided a 13 percentage point increase, suggesting that approximately one-third of the FO speechreading enhancement effect could be accounted for by voicing detection alone.