When one sees what the other hears: Crossmodal attentional modulation for gazed and non-gazed upon auditory targets

Abstract

Three experiments investigated the nature of visuo-auditory crossmodal cueing in a triadic setting: participants had to detect an auditory signal while observing another agent’s head facing one of the two laterally positioned auditory sources. Experiment 1 showed that when the agent’s eyes were open, sounds originating on the side of the agent’s gaze were detected faster than sounds originating on the side of the agent’s visible ear; when the agent’s eyes were closed this pat-tern of responses was reversed. Two additional experiments showed that the results were sensitive to whether participants could infer a hearing function on the part of the agent. When no ear was depicted on the agent, only a gaze-side advantage was observed (Experiment 2), but when the agent’s ear was covered (Experiment 3), an ear side advantage was observed only when hearing could still be inferred (i.e., wearing the hat) but not when hearing was inferred to be diminished (i.e., wearing a helmet). The findings are discussed in the context of inferential and simulation processes and joint attention mechanisms.