Visual speech cues relieve listening effort selectively for listeners with hearing loss

Abstract

Elevated listening effort is one of the most commonly reported and widely impactful problems among individuals with hearing loss. The ability to see a talker’s face may help mitigate listening effort, but research on this topic has generally excluded individuals with hearing loss. In this study, we used pupillometry as a time-sensitive index of listening effort for audio-visual speech while manipulating the availability of visual cues by selectively blurring the talker’s mouth. Blurring removed place-of-articulation information while allowing fair comparison of pupil size across conditions. Effortful listening was elicited using a sentence repetition task in which participants needed to use later context to fill in a missing word from earlier in the sentence. For listeners with typical hearing, visual cues had no measurable effect on listening effort. In contrast, listeners with cochlear implants (CIs) had two distinct benefits from visual cues: first, pupil dilations were smaller, suggesting release from effort. Second, the downstream intelligibility errors caused by the word-repair process were alleviated. These results caution against generalizing findings obtained with typical-hearing participants to a hearing loss population, support deeper analysis of perception error patterns, and demonstrate the importance of multisensory speech cues to relieve listening effort.