Contribution of detailed parts around talker's mouth for audio-visual speech perception

Abstract

In this study, the relationship between speech intelligibility and effects of the parts around talker's mouth was investigated based on the results of audio-visual speech intelligibility tests. As the stimuli, nonsense tri-syllables speech sounds were combined with three kinds of moving image of the talker's face: original face, circumference of the lips (mouth part extracted from the original face), and masked lips (face without the mouth). The extracted area around the mouth was varied as a parameter in the circumference-of-the-lips conditions. Generated stimuli were presented with speech spectrum noise to the participants. The results showed that intelligibility scores of several phonemes (/h/, /m/, /w/, /b/, /p/) were increased by adding the visual information. Moreover, there was no significant difference between the score of the original-face condition and that of the circumference-of-the-lips condition. Interestingly, significant difference was observed between the score of the audio-only condition and that of the masked-lips-condition. These results imply that not only a mouth but also areas around the mouth without mouth itself provide rich information for speech understanding. Effects of the mouth edges on speech intelligibility are being examined as a next step.