Characteristic Sounds Facilitate Object Search in Real-Life Scenes.

Daria Kvasova,Salvador Soto-Faraco,Laia Garcia-Vernet

doi:10.3389/fpsyg.2019.02511

Daria Kvasova, Salvador Soto-Faraco + Show 1 more

Open Access

https://doi.org/10.3389/fpsyg.2019.02511

Copy DOI

Abstract

Real-world events do not only provide temporally and spatially correlated information across the senses, but also semantic correspondences about object identity. Prior research has shown that object sounds can enhance detection, identification, and search performance of semantically consistent visual targets. However, these effects are always demonstrated in simple and stereotyped displays that lack ecological validity. In order to address identity-based cross-modal relationships in real-world scenarios, we designed a visual search task using complex, dynamic scenes. Participants searched for objects in video clips recorded from real-life scenes. Auditory cues, embedded in the background sounds, could be target-consistent, distracter-consistent, neutral, or just absent. We found that, in these naturalistic scenes, characteristic sounds improve visual search for task-relevant objects but fail to increase the salience of irrelevant distracters. Our findings generalize previous results on object-based cross-modal interactions with simple stimuli and shed light upon how audio–visual semantically congruent relationships play out in real-life contexts.

Highlights

Interactions between sensory modalities are at the core of human perception and behavior
Regarding the possible task-relevance modulation of cross-modal semantic effects, we hypothesized that if audio–visual semantic congruence attracts attention in natural scenes automatically even when the objects are irrelevant to the current behavioral goal, one should expect a slowdown in responses to targets in distracter-consistent trials, with respect to neutral sound trials
In particular we had hypothesized that target-consistent characteristic sounds will help attract attention to the corresponding visual object

Summary

INTRODUCTION

Interactions between sensory modalities are at the core of human perception and behavior. To the study of Nardo et al (2014), which found no effect, Iordanescu et al (2008, 2010) and Mastroberardino et al (2015) used simple static images presented in decontextualized search arrays (Iordanescu et al, 2008, 2010) Both, these differential features (dynamic nature of natural scenes and their complexity) have been pointed out as important components for the generalization of cognitive psychology and neuroimaging findings to real-world contexts (e.g., Hasson et al, 2010). Regarding the possible task-relevance modulation of cross-modal semantic effects, we hypothesized that if audio–visual semantic congruence attracts attention in natural scenes automatically even when the objects are irrelevant to the current behavioral goal, one should expect a slowdown in responses to targets in distracter-consistent trials, with respect to neutral sound trials. We expected that differences due to general alerting of sounds, if any, would affect target-consistent, distractor-consistent, and neutral sound conditions, but not the no-sound baseline

Participants

Procedure

RESULTS AND DISCUSSION

CONCLUSION

ETHICS STATEMENT