Audiovisual integration in depth: multisensory binding and gain as a function of distance

Jean-Paul Noel,Mark T Wallace,Kahan Modi,Nathan Van Der Stoep

doi:10.1007/s00221-018-5274-7

Abstract

The integration of information across sensory modalities is dependent on the spatiotemporal characteristics of the stimuli that are paired. Despite large variation in the distance over which events occur in our environment, relatively little is known regarding how stimulus-observer distance affects multisensory integration. Prior work has suggested that exteroceptive stimuli are integrated over larger temporal intervals in near relative to far space, and that larger multisensory facilitations are evident in far relative to near space. Here, we sought to examine the interrelationship between these previously established distance-related features of multisensory processing. Participants performed an audiovisual simultaneity judgment and redundant target task in near and far space, while audiovisual stimuli were presented at a range of temporal delays (i.e., stimulus onset asynchronies). In line with the previous findings, temporal acuity was poorer in near relative to far space. Furthermore, reaction time to asynchronously presented audiovisual targets suggested a temporal window for fast detection—a range of stimuli asynchronies that was also larger in near as compared to far space. However, the range of reaction times over which multisensory response enhancement was observed was limited to a restricted range of relatively small (i.e., 150 ms) asynchronies, and did not differ significantly between near and far space. Furthermore, for synchronous presentations, these distance-related (i.e., near vs. far) modulations in temporal acuity and multisensory gain correlated negatively at an individual subject level. Thus, the findings support the conclusion that multisensory temporal binding and gain are asymmetrically modulated as a function of distance from the observer, and specifies that this relationship is specific for temporally synchronous audiovisual stimulus presentations.

Highlights

Our senses are equipped with an array of transducers capable of converting different forms of environmental energy into neural signals
In two separate studies, it was observed that temporal-binding windows are larger in near space (Noel et al 2016a), while multisensory gain is greater in far space (Van der Stoep et al 2016b)
We hypothesized that distance-related decreases in temporal-binding windows would be associated with increases in multisensory gain at the level of the individual subject

Summary

Introduction

Our senses are equipped with an array of transducers capable of converting different forms of environmental energy into neural signals (e.g., photons impinging on the retina in the case of vision, sound waves moving the cochlear membrane in the case of audition). The principles governing MSI, originally demonstrated at the level of single neurons, have been found to apply at various levels of neural description (e.g., single units, local field potentials, electroencephalography, functional magnetic resonance imaging, and in behavior and perception, e.g., Meredith and Stein 1986a, b; Wallace et al 1992; Stein and Meredith 1993; Cappe et al 2012; though see; Stanford and Stein 2007; Spence 2013) Among these principles, the spatial and temporal principles state that the closer in space and/or time two unisensory stimuli are from one another, the greater the likelihood that these stimuli will be integrated. The importance of this is underscored by the simple observation that manipulating the spatial or temporal features of stimuli in isolation differs from most real-world circumstances, where these features are heavily interrelated and continuously changing

Objectives

Methods

Results

Conclusion