How to teach a blind person to hear colours? Multi-method training for a colour-to-sound sensory substitution device – design and evaluation

Abstract

Sensory substitution is thought to be a promising non-invasive assistive technology for people with complete loss of sight because it provides inaccessible visual information via a preserved modality. However, Sensory Substitution Devices (SSDs) are still rarely used by visually impaired persons, possibly due to a lack of structured and supervised training that could be offered alongside these devices. Here, we developed and evaluated a training program that supports the usage of a recently developed colour-to-sound SSD – the Colorophone. Following our recently proposed theoretical model of SSD development, we propose that this training should help people with complete loss of sight to learn how to efficiently use the device by developing relationships between the components of the user-environment-technology system. We applied systematic case studies combined with a mixed-method approach to evaluate the efficacy of this SSD training program. Five blind users underwent ca. 22 h of training, divided into four main parts: identification of the users’ individual characteristics and adaptations; sensorimotor training with the device; semi-structured explorations with the device; and evaluation of the training. We demonstrated that this training allows users to successfully acquire a set of skills (i.e., master the sensorimotor contingencies required by the device, develop visual-like perceptual skills, as well as learn about colours) and progress along developmental trajectories (e.g., switch from serial to parallel information processing, recognize more complex colours, increase environment and task complexity). Importantly, we identified individual differences in learning strategies (i.e., sensorimotor vs. metacognitive strategy) that had an impact on the users’ training progress and required the training assistants (TAs) to apply different assistive strategies. Additionally, we described the crucial role of a (non-professional) training assistant in the training progress: this person facilitates the development of relationships between elements of the user-environment-technology system by supporting a metacognitive learning strategy, thereby reducing the risk of abandonment of the SSD. Our study shows the importance for SSD development of well-designed, tailored training, and it provides new insights into the process of SSD-related perceptual learning.