Abstract
Real-world events do not only provide temporally and spatially correlated information across the senses, but also semantic correspondences about object identity. Prior research has shown that object sounds can enhance detection, identification, and search performance of semantically consistent visual targets. However, these effects are always demonstrated in simple and stereotyped displays that lack ecological validity. In order to address identity-based cross-modal relationships in real-world scenarios, we designed a visual search task using complex, dynamic scenes. Participants searched for objects in video clips recorded from real-life scenes. Auditory cues, embedded in the background sounds, could be target-consistent, distracter-consistent, neutral, or just absent. We found that, in these naturalistic scenes, characteristic sounds improve visual search for task-relevant objects but fail to increase the salience of irrelevant distracters. Our findings generalize previous results on object-based cross-modal interactions with simple stimuli and shed light upon how audio–visual semantically congruent relationships play out in real-life contexts.
Highlights
Interactions between sensory modalities are at the core of human perception and behavior
Regarding the possible task-relevance modulation of cross-modal semantic effects, we hypothesized that if audio–visual semantic congruence attracts attention in natural scenes automatically even when the objects are irrelevant to the current behavioral goal, one should expect a slowdown in responses to targets in distracter-consistent trials, with respect to neutral sound trials
In particular we had hypothesized that target-consistent characteristic sounds will help attract attention to the corresponding visual object
Summary
Interactions between sensory modalities are at the core of human perception and behavior. To the study of Nardo et al (2014), which found no effect, Iordanescu et al (2008, 2010) and Mastroberardino et al (2015) used simple static images presented in decontextualized search arrays (Iordanescu et al, 2008, 2010) Both, these differential features (dynamic nature of natural scenes and their complexity) have been pointed out as important components for the generalization of cognitive psychology and neuroimaging findings to real-world contexts (e.g., Hasson et al, 2010). Regarding the possible task-relevance modulation of cross-modal semantic effects, we hypothesized that if audio–visual semantic congruence attracts attention in natural scenes automatically even when the objects are irrelevant to the current behavioral goal, one should expect a slowdown in responses to targets in distracter-consistent trials, with respect to neutral sound trials. We expected that differences due to general alerting of sounds, if any, would affect target-consistent, distractor-consistent, and neutral sound conditions, but not the no-sound baseline
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
