Neural correlates of audio‐visual object recognition: Effects of implicit spatial congruency

Abstract

Multisensory integration assists us to identify objects by providing multiple cues with respect to object category and spatial location. We used a semantic audiovisual object matching task to determine the effect of spatial congruency on response behavior and fMRI brain activation. Fifteen subjects responded in a four-alternative response paradigm, which visual quadrant contained the object best matched to the sound presented. Realistic sounds based on head-related transfer functions were presented binaurally with the simulated sound source corresponding to one of the four quadrants. Following a random sequence, the location of the sound corresponded to the quadrant containing the semantically congruent target on half the trials, whereas on other trials the sound arose from an incongruent location. We examined the effects of spatial congruency on response latencies, hit-rates and fMRI responses. Preliminary behavioral results revealed a significant effect of spatial congruency on response latency or performance for stimuli with noise added. In the fMRI experiment, spatial congruency had a significant effect on the BOLD response. A cluster in the right middle and superior temporal gyrus was more activated when the auditory sound sources were spatially congruent with the semantically matching visual stimulus. In an exploratory post-hoc analysis, in which we correlated the BOLD signal with the subjects' ability to locate the sound sources, we found a significant cluster in the left inferior frontal cortex, where the BOLD response increased with sound-source localization performance. Thus spatial congruency appears to enhance the semantic integration of audiovisual object information in these brain regions.