Assessing perceived listening difficulty using behavioral gaze patterns for audiovisual speech

Abstract

Listeners alternate their gaze between the talker’s eyes, nose, and mouth during conversation, spending more time gazing directly at the mouth in more difficult listening conditions. Perceived listening difficulty and speech perception performance vary independently, demonstrated with indirect measures of perceived listening difficulty/effort such as pupil dilation and dual task paradigms. This study took a more direct approach, evaluating whether perceived listening difficulty could be assessed by observing a listener’s natural gaze pattern in a challenging speech-perception task. The proportion of time spent gazing at the talker’s mouth was measured for auditory-visual speech perception in conditions that produced equal percentage-correct scores, but where one condition was designed to produce greater perceived difficulty. Experiment 1 compared gaze proportion for a speech-shaped noise versus two same-gender interfering talkers. Experiment 2 examined the effect of sentence context using low and high-probability sentences. The results are discussed in terms of the potential application of the eye-gaze metric to evaluate rehabilitative interventions in cases where subjectively reported benefits do not manifest in improved speech-understanding scores. [The views expressed in this abstract are those of the authors and do not reflect the official policy of the Department of Army/Navy/Air Force, Department of Defense, or U.S. Government.]