Multi-sensory comparison improves signal discrimination

Abstract

We asked whether a multi-modal signal provides benefits in discrimination performance over and above that afforded by having two samples. Observers judged whether the frequency of a sinusoidal amplitude-modulated 3kHz tone and/or Gaussian blob was greater or less than 10Hz. The signals were embedded in additive band-limited white noise centered on 10Hz. There were 6 conditions in which judgments were based on: single visual (1) or auditory (2) stimuli; simultaneous (3) or sequential bi-modal (4) stimuli (random order) or repeated visual (5) or auditory (6) stimuli. External noise amplitude was adjusted such that performance in conditions 1 and 2 was the same. In conditions 3-6 the signal in both repetitions/modalities was the same while the added noise was independent. Conditions 3 & 4 tested whether bi-modal effects require simultaneous bi-modal presentation. Conditions 5 & 6 tested whether performance improved when information was presented twice in the same modality. We found 1. frequency discrimination improved when stimuli were presented in two modalities; 2. performance improvement did not require simultaneous presentation in both modalities; 3. repeated stimulation in the same modality did not improve discrimination performance. In a separate detection experiment we found no difference between bi-modal and uni-modal repetitions. There was no difference between adding further noise to the first or second paired item demonstrating that both repeated items had equal weight. These results suggest that a-modal signal properties (e.g. amplitude modulation rate) are evaluated by a process of within-modality signal discrimination followed by weighted averaging of the uni-modal estimates.