Observing lip movements of a speaker facilitates speech understanding, especially in challenging listening situations. Converging evidence from neuroscientific studies shows stronger neural responses to audiovisual stimuli compared to audio-only stimuli. However, the interindividual variability of this contribution of lip movement information and its consequences on behavior are unknown. We analyzed source-localized magnetoencephalographic (MEG) responses from 29 normal-hearing participants (12 female) listening to audiovisual speech, both with and without the speaker wearing a surgical face mask, and in the presence or absence of a distractor speaker. Using temporal response functions (TRFs) to quantify neural speech tracking, we show that neural responses to lip movements are, in general, enhanced when speech is challenging. After controlling for speech acoustics, we show that lip movements contribute to enhanced neural speech tracking, particularly when a distractor speaker is present. However, the extent of this visual contribution to neural speech tracking varied greatly among participants. Probing the behavioral relevance, we demonstrate that individuals who show a higher contribution of lip movements in terms of neural speech tracking, show a stronger drop in comprehension and an increase in perceived difficulty when the mouth is occluded by a surgical face mask. By contrast, no effect was found when the mouth was not occluded. We provide novel insights on how the contribution of lip movements in terms of neural speech tracking varies among individuals and its behavioral relevance, revealing negative consequences when visual speech is absent. Our results also offer potential implications for objective assessments of audiovisual speech perception.Significance Statement In complex auditory environments, simultaneous conversations pose a challenge to speech comprehension. We investigated on a neural level, how lip movements aid in such situations and what the behavioral consequences are, especially when lip information is occluded with a face mask. Using magnetoencephalographic (MEG) responses from participants listening to audiovisual speech, we show that observing lip movements enhances neural speech tracking and participants who rely more on lip movements show behavioral deterioration when the speaker wears a face mask. Remarkably, this is not the case when no face mask was worn by the speaker. Our findings reveal interindividual differences in the contribution of lip movements to neural speech tracking, with potential applications in objective assessments of audiovisual speech perception.
Read full abstract