Abstract
Face-selective and voice-selective brain regions have been shown to represent face-identity and voice-identity, respectively. Here we investigated whether there are modality-general person-identity representations in the brain that can be driven by either a face or a voice, and that invariantly represent naturalistically varying face videos and voice recordings of the same identity. Models of face and voice integration suggest that such representations could exist in multimodal brain regions, and in unimodal regions via direct coupling between face- and voice-selective regions. Therefore, in this study we used fMRI to measure brain activity patterns elicited by the faces and voices of familiar people in face-selective, voice-selective, and person-selective multimodal brain regions. We used representational similarity analysis to (1) compare representational geometries (i.e. representational dissimilarity matrices) of face- and voice-elicited identities, and to (2) investigate the degree to which pattern discriminants for pairs of identities generalise from one modality to the other. We did not find any evidence of similar representational geometries across modalities in any of our regions of interest. However, our results showed that pattern discriminants that were trained to discriminate pairs of identities from their faces could also discriminate the respective voices (and vice-versa) in the right posterior superior temporal sulcus (rpSTS). Our findings suggest that the rpSTS is a person-selective multimodal region that shows a modality-general person-identity representation and integrates face and voice identity information.
Highlights
Looking at a familiar person’s face or listening to their voice automatically grants us access to a wealth of information regarding the person’s identity, such as their name, our relationship to them, and memories of previous encounters
Analysis A focused on the representational geometry of all identities, i.e. the entire structure of pairwise distances between the activity patterns elicited by these identities in each modality, and compared geometries across modalities (Kriegeskorte et al, 2008a, 2008b; Kriegeskorte and Kievit, 2013)
Our results showed that mean linear discriminant contrast (LDC) values in these representational dissimilarity matrices (RDMs) were significantly greater than zero in the right posterior superior temporal sulcus (rpSTS) (Fig. 6A and Table 1)
Summary
Looking at a familiar person’s face or listening to their voice automatically grants us access to a wealth of information regarding the person’s identity, such as their name, our relationship to them, and memories of previous encounters. Knowledge about how the brain processes faces and voices separately has advanced significantly over the past twenty years: functional magnetic resonance imaging (fMRI) revealed cortical regions that are face-selective (Kanwisher et al, 1997; McCarthy et al, 1997) and regions that are voice-selective (Belin et al, 2000). Face-selective regions in the posterior occipitotemporal lobe, anterior temporal lobe, and posterior superior temporal sulcus (pSTS) can discriminate different face images (Kriegeskorte et al, 2007; Nestor et al, 2011; Goesaert and Op de Beeck, 2013; Verosky et al, 2013; Axelrod and Yovel, 2015; Collins et al, 2016; Visconti Di Oleggio Castello et al, 2017). For voices, Formisano et al (2008) found voice-identity representations in the right STS and Heschl’s gyrus that could both discriminate between speakers and generalise across different vowel sounds spoken by the same voice
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
