Shared and modality-specific brain regions that mediate auditory and visual word comprehension.

Anne Keitel,Joachim Gross,Christoph Kayser

doi:10.7554/elife.56972

Anne Keitel, Joachim Gross + Show 1 more

Open Access

https://doi.org/10.7554/elife.56972

Copy DOI

Abstract

Visual speech carried by lip movements is an integral part of communication. Yet, it remains unclear in how far visual and acoustic speech comprehension are mediated by the same brain regions. Using multivariate classification of full-brain MEG data, we first probed where the brain represents acoustically and visually conveyed word identities. We then tested where these sensory-driven representations are predictive of participants' trial-wise comprehension. The comprehension-relevant representations of auditory and visual speech converged only in anterior angular and inferior frontal regions and were spatially dissociated from those representations that best reflected the sensory-driven word identity. These results provide a neural explanation for the behavioural dissociation of acoustic and visual speech comprehension and suggest that cerebral representations encoding word identities may be more modality-specific than often upheld.

Highlights

Acoustic and visual speech signals are both elemental for everyday communication
The comprehension-relevant representations of auditory and visual speech converged only in anterior angular and inferior frontal regions and were spatially dissociated from those representations that best reflected the sensory-driven word identity. These results provide a neural explanation for the behavioural dissociation of acoustic and visual speech comprehension and suggest that cerebral representations encoding word identities may be more modality-specific than often upheld
Participants performed the task with auditory and visual stimuli presented at the same time, but because performance in this condition was near ceiling (Maudiovisual = 96.4%, SD = 3.3%), we present the corresponding data only in the supplementary material (Figure 2—figure supplement 2A)

Summary

Introduction

Acoustic and visual speech signals are both elemental for everyday communication. While acoustic speech consists of temporal and spectral modulations of sound pressure, visual speech consists of movements of the mouth, head, and hands. While a plethora of studies have investigated the cerebral mechanisms underlying speech in general, we still have a limited understanding of the networks mediating visual speech perception, that is lip reading (Bernstein and Liebenthal, 2014; Capek et al, 2008; Crosse et al, 2015). Large inter-individual differences in lip reading skills contrast with the low variability seen in auditory speech tests (Summerfield, 1992) In contrast to this behavioural dissociation, neuroimaging and neuroanatomical studies have suggested the convergence of acoustic and visual speech information in specific brain regions (Calvert, 1997; Campbell, 2008; Ralph et al, 2017; Simanova et al, 2014). Prevalent models postulate a fronto-temporal network mediating acoustic speech representations, comprising

Results

Discussion

Conclusion