Auditory Sensory Substitution is Intuitive and Automatic with Texture Stimuli.

Noelle R B Stiles,Shinsuke Shimojo

doi:10.1038/srep15628

Noelle R B Stiles, Shinsuke Shimojo

Open Access

https://doi.org/10.1038/srep15628

Copy DOI

Journal: Scientific Reports	Publication Date: Oct 22, 2015
Citations: 38	License type: CC BY 4.0

Affiliation: California Institute of Technology

Abstract

Millions of people are blind worldwide. Sensory substitution (SS) devices (e.g., vOICe) can assist the blind by encoding a video stream into a sound pattern, recruiting visual brain areas for auditory analysis via crossmodal interactions and plasticity. SS devices often require extensive training to attain limited functionality. In contrast to conventional attention-intensive SS training that starts with visual primitives (e.g., geometrical shapes), we argue that sensory substitution can be engaged efficiently by using stimuli (such as textures) associated with intrinsic crossmodal mappings. Crossmodal mappings link images with sounds and tactile patterns. We show that intuitive SS sounds can be matched to the correct images by naive sighted participants just as well as by intensively-trained participants. This result indicates that existing crossmodal interactions and amodal sensory cortical processing may be as important in the interpretation of patterns by SS as crossmodal plasticity (e.g., the strengthening of existing connections or the formation of new ones), especially at the earlier stages of SS usage. An SS training procedure based on crossmodal mappings could both considerably improve participant performance and shorten training times, thereby enabling SS devices to significantly expand blind capabilities.

Highlights

The majority of participants being trained on sensory substitution, as reported in the literature, begin with a simplified lab environment, such as a white isolated object on a black background, and only experience a natural environment with the device after several training sessions
The implication of the crossmodal mapping tests is that two encoding elements are important to image interpretation with vOICe: Brightness correlating with loudness, and the coordinates of the encoding
The similarity of the naive and trained performance in these experiments indicates that pre-existing crossmodal interactions may play an equal role in SS pattern recognition as crossmodal plasticity

Summary

Introduction

We show that intuitive SS sounds can be matched to the correct images by naive sighted participants just as well as by intensivelytrained participants This result indicates that existing crossmodal interactions and amodal sensory cortical processing may be as important in the interpretation of patterns by SS as crossmodal plasticity (e.g., the strengthening of existing connections or the formation of new ones), especially at the earlier stages of SS usage. Participants with no knowledge about the vOICe device may in principle be able to use crossmodal correspondences to naively match images with their correct vOICe sounds Certain stimuli such as textures may have strong intrinsic crossmodal associations and similar structure across modalities (as shown in Fig. 1b), and may be correctly interpreted by naive participants. All crossmodal changes engendered through training in these experiments will be referred to as crossmodal plasticity

Methods

Results

Conclusion