Sensory Substitution Devices Research Articles

Word recognition: re-thinking prosthetic vision evaluation

Objective. Evaluations of vision prostheses and sensory substitution devices have frequently relied on repeated training and then testing with the same small set of items. These multiple forced-choice tasks produced above chance performance in blind users, but it is unclear if the observed performance represents restoration of vision that transfers to novel, untrained items. Approach. Here, we tested the generalizability of the forced-choice paradigm on discrimination of low-resolution word images. Extensive visual training was conducted with the same 10 words used in previous BrainPort tongue stimulation studies. The performance on these 10 words and an additional 50 words was measured before and after the training sessions. Main results. The results revealed minimal performance improvement with the untrained words, demonstrating instead pattern discrimination limited mostly to the trained words. Significance. These findings highlight the need to reconsider current evaluation practices, in particular, the use of forced-choice paradigms with a few highly trained items. While appropriate for measuring the performance thresholds in acuity or contrast sensitivity of a functioning visual system, performance on such tasks cannot be taken to indicate restored spatial pattern vision.

Comparison Between Force-Controlled Skin Deformation Feedback and Hand-Grounded Kinesthetic Force Feedback for Sensory Substitution

Teleoperation and virtual reality systems benefit from force sensory substitution when kinesthetic force feedback devices are infeasible due to stability or workspace limitations. We compared the performance of sensory substitution when it is provided through a cutaneous method (skin deformation feedback) and a kinesthetic method (hand-grounded force feedback). For skin deformation feedback, we used a new force-controlled tactile sensory substitution device with the ability to provide tangential and normal force directly to the finger pad. Three-axis force control with 15 Hz bandwidth was achieved using a delta mechanism and three-axis force sensor. For hand-grounded force feedback, forces were grounded against the palm. As a control, world-grounded force feedback was provided using a three-degree-of-freedom kinesthetic force feedback device. Study participants were able to match a reference world-grounded force better with hand-grounded kinesthetic force feedback than with skin deformation feedback. Participants were also able to apply more accurate and precise forces with hand-grounded kinesthetic force feedback than with skin deformation feedback. Conversely, skin deformation feedback resulted in the lowest error during initial force adjustment. These experiments demonstrate relative advantages and disadvantages of skin deformation and hand-grounded kinesthetic force feedback for force sensory substitution.

Perceptually Equivalent Judgments Made Visually and via Haptic Sensory-Substitution Devices

ABSTRACTAccording to the ecological theory of perception–action, perception is primarily of affordances, which are directly perceivable opportunities for behavior. The current study evaluated participants’ ability to use vision and haptic sensory-substitution devices to support perceptual judgments of affordances involving the task of passing through apertures. Sighted participants made perceptual judgments about whether they could walk through apertures of various widths and their level of confidence in each judgment, using unrestricted vision and, when blindfolded, using two haptic sensory-substitution instruments: a cane-like wooden rod and the Enactive Torch, a device that converts distance information into vibrotactile stimuli. The boundary between aperture widths that were judged as pass-through-able versus non-pass-through-able was statistically equivalent across sensory modalities. However, participants were not as confident in their judgments using the rod or Enactive Torch as they were using vision. Additionally, participants’ judgments with the haptic instruments were significantly more accurate than with vision. The results underscore the need to assess sensory-substitution devices in the context of functional behaviors.

Guidelines for quantitative and qualitative studies of sensory substitution experience

Information that is normally accessed through a sensory modality (substituted modality, e.g., vision) is provided by sensory substitution devices (SSDs) through an alternative modality such as hearing or touch (i.e., substituting modality). SSDs usually support disabled users by replacing sensory inputs that have been lost, but they also offer a unique opportunity to study adaptation and flexibility in human perception. Current debates in sensory substitution (SS) literature focus mostly on its neural correlates and behavioural consequences. In particular, studies have demonstrated the neural plasticity of the visual brain regions that are activated by the substituting modality. Participants also adapt to using the devices for a broad spectrum of cognitive tasks that usually require sight. However, little is known about the SS experience. Also, there is no agreement on how the phenomenology of SS should be studied. Here, we offer guidelines for the methodology of studies investigating behavioural adaptation to SS and the effects of this adaptation on the subjective SS experience. We also discuss factors that may influence the results of SS studies: (1) the type of SSD, (2) the effects of training, (3) the role of sensory deprivation, (4) the role of the experimental environment, (5) the role of the tasks participants follow, and (6) the characteristics of the participants. In addition, we propose combining qualitative and quantitative methods and discuss how this should be achieved when studying the neural, behavioural, and experiential consequences of SS.

Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device.

In face-to-face social interactions, blind and visually impaired persons (VIPs) lack access to nonverbal cues like facial expressions, body posture, and gestures, which may lead to impaired interpersonal communication. In this study, a wearable sensory substitution device (SSD) consisting of a head mounted camera and a haptic belt was evaluated to determine whether vibrotactile cues around the waist could be used to convey facial expressions to users and whether such a device is desired by VIPs for use in daily living situations. Ten VIPs (mean age: 38.8, SD: 14.4) and 10 sighted persons (SPs) (mean age: 44.5, SD: 19.6) participated in the study, in which validated sets of pictures, silent videos, and videos with audio of facial expressions were presented to the participant. A control measurement was first performed to determine how accurately participants could identify facial expressions while relying on their functional senses. After a short training, participants were asked to determine facial expressions while wearing the emotion feedback system. VIPs using the device showed significant improvements in their ability to determine which facial expressions were shown. A significant increase in accuracy of 44.4% was found across all types of stimuli when comparing the scores of the control (mean±SEM: 35.0±2.5%) and supported (mean±SEM: 79.4±2.1%) phases. The greatest improvements achieved with the support of the SSD were found for silent stimuli (68.3% for pictures and 50.8% for silent videos). SPs also showed consistent, though not statistically significant, improvements while supported. Overall, our study shows that vibrotactile cues are well suited to convey facial expressions to VIPs in real-time. Participants became skilled with the device after a short training session. Further testing and development of the SSD is required to improve its accuracy and aesthetics for potential daily use.

Sensory substitution: Using a vibrotactile device to orient and walk to targets.

This study investigates how active exploration helps users of sensory substitution devices (SSDs) to detect action-relevant information. A vibrotactile SSD was developed that generates stimulation that is contingent on the users' movements. Target direction was specified by the location of the vibratory stimulation, and target distance by the size and intensity of the pattern of stimulation. A series of experiments was performed with blindfolded participants. In Experiments 1a to 1c, participants used the SSD to align their central body axis with prespecified targets. These experiments differed in the number of actuators that were used and whether online perception-action coupling was present. In Experiment 2, participants approached targets with forward locomotion along a straight line. Experiment 3 combined the previous experiments and studied the concomitant walking and steering toward targets. Oscillatory movements, which facilitated information pickup, were observed in all experiments. The exploratory oscillations were shown to depend on the online perception-action coupling and were related to cases of hyperacuity, for which absolute errors were found to be smaller than the areas of sensitivity of the actuators. It is concluded that, to improve the utility of SSDs, future research with SSDs should pay more attention to the role of active information detection. (PsycINFO Database Record

Tactile recognition of visual stimuli: Specificity versus generalization of perceptual learning

Sensory substitution devices aim at assisting a deficient sensory modality by means of another sensory modality. For instance, to perceive with visual-to-tactile devices, users learn to recognize visual stimuli through their tactile conversion. A crucial characteristic of learning lies in the ability to generalize, that is, the ability to extend the acquired perceptual abilities to both new stimuli and new perceptual conditions. The study reported here investigated the perceptual learning of tactile alphanumerical stimuli. The learning protocol consisted in alternating a repeated list of symbols with lists of new symbols. A first experiment revealed that, when each list consisted of 4 stimuli, recognition performance improved over time only for the repeated list. This result suggests that learning a small set of stimuli involves stimulus-specific learning strategies, preventing generalization. A second experiment revealed that increasing to six the set of learned stimuli results in higher generalization abilities. This result can be explained by greater difficulties in using stimulus-specific strategies in this case, thereby favouring the use of generalization strategies. Feature variability also appeared to be important to achieve generalization. Thus, as in visual perceptual learning, the involvement of stimulus-specific versus general strategies depends on task difficulty and feature variability. A third experiment highlighted that tactile perceptual learning generalizes to changes in orientation. These results are discussed in terms of brain plasticity as they influence the design of learning methods for using sensory substitution devices, with the aim to compensate visual impairments.

Relative vibrotactile spatial acuity of the torso

While tactile acuity for pressure has been extensively investigated, far less is known about acuity for vibrotactile stimulation. Vibrotactile acuity is important however, as such stimulation is used in many applications, including sensory substitution devices. We tested discrimination of vibrotactile stimulation from eccentric rotating mass motors with in-plane vibration. In 3 experiments, we tested gradually decreasing center-to-center (c/c) distances from 30 mm (experiment 1) to 13 mm (experiment 3). Observers judged whether a second vibrating stimulator (‘tactor’) was to the left or right or in the same place as a first one that came on 250 ms before the onset of the second (with a 50-ms inter-stimulus interval). The results show that while accuracy tends to decrease the closer the tactors are, discrimination accuracy is still well above chance for the smallest distance, which places the threshold for vibrotactile stimulation well below 13 mm, which is lower than recent estimates. The results cast new light on vibrotactile sensitivity and can furthermore be of use in the design of devices that convey information through vibrotactile stimulation.

Blindness enhances auditory obstacle circumvention: Assessing echolocation, sensory substitution, and visual-based navigation.

Performance for an obstacle circumvention task was assessed under conditions of visual, auditory only (using echolocation) and tactile (using a sensory substitution device, SSD) guidance. A Vicon motion capture system was used to measure human movement kinematics objectively. Ten normally sighted participants, 8 blind non-echolocators, and 1 blind expert echolocator navigated around a 0.6 x 2 m obstacle that was varied in position across trials, at the midline of the participant or 25 cm to the right or left. Although visual guidance was the most effective, participants successfully circumvented the obstacle in the majority of trials under auditory or SSD guidance. Using audition, blind non-echolocators navigated more effectively than blindfolded sighted individuals with fewer collisions, lower movement times, fewer velocity corrections and greater obstacle detection ranges. The blind expert echolocator displayed performance similar to or better than that for the other groups using audition, but was comparable to that for the other groups using the SSD. The generally better performance of blind than of sighted participants is consistent with the perceptual enhancement hypothesis that individuals with severe visual deficits develop improved auditory abilities to compensate for visual loss, here shown by faster, more fluid, and more accurate navigation around obstacles using sound.

Sensory Substitution and Multimodal Mental Imagery.

Many philosophers use findings about sensory substitution devices in the grand debate about how we should individuate the senses. The big question is this: Is "vision" assisted by (tactile) sensory substitution really vision? Or is it tactile perception? Or some sui generis novel form of perception? My claim is that sensory substitution assisted "vision" is neither vision nor tactile perception, because it is not perception at all. It is mental imagery: visual mental imagery triggered by tactile sensory stimulation. But it is a special form of mental imagery that is triggered by corresponding sensory stimulation in a different sense modality, which I call "multimodal mental imagery."

Software Design for Low-Latency Visuo-Auditory Sensory Substitution on Mobile Devices

Visuo-auditory sensory substitution devices transform a video stream into an audio stream to help visually impaired people in situations where spatial information is required, such as avoiding moving obstacles. In these particular situations, the latency between an event in the real world and its auditory transduction is of paramount importance. In this article, we describe an optimized software architecture for low-latency video-to-audio transduction using current mobile hardware. We explain step-by-step the required computations and we report the corresponding measured latencies. The whole latency is approximately 65 ms with a capture resolution of 160 × 120 at 30 frames-per-second and 1000 sonified pixels per frame.

Improving the Audio Game–Playing Performances of People with Visual Impairments through Multimodal Training

Introduction As the number of people with visual impairments (that is, those who are blind or have low vision) is continuously increasing, rehabilitation and engineering researchers have identified the need to design sensory-substitution devices that would offer assistance and guidance to these people for performing navigational tasks. Auditory and haptic cues have been shown to be an effective approach towards creating a rich spatial representation of the environment, so they are considered for inclusion in the development of assistive tools that would enable people with visual impairments to acquire knowledge of the surrounding space in a way close to the visually based perception of sighted individuals. However, achieving efficiency through a sensory substitution device requires extensive training for visually impaired users to learn how to process the artificial auditory cues and convert them into spatial information. Methods Considering all the potential advantages game-based learning can provide, we propose a new method for training sound localization and virtual navigational skills of visually impaired people in a 3D audio game with hierarchical levels of difficulty. The training procedure is focused on a multimodal (auditory and haptic) learning approach in which the subjects have been asked to listen to 3D sounds while simultaneously perceiving a series of vibrations on a haptic headband that corresponds to the direction of the sound source in space. Results The results we obtained in a sound-localization experiment with 10 visually impaired people showed that the proposed training strategy resulted in significant improvements in auditory performance and navigation skills of the subjects, thus ensuring behavioral gains in the spatial perception of the environment.

Individual Differences in Sensory Substitution.

Sensory substitution devices were developed in the context of perceptual rehabilitation and they aim at compensating one or several functions of a deficient sensory modality by converting stimuli that are normally accessed through this deficient sensory modality into stimuli accessible by another sensory modality. For instance, they can convert visual information into sounds or tactile stimuli. In this article, we review those studies that investigated the individual differences at the behavioural, neural, and phenomenological levels when using a sensory substitution device. We highlight how taking into account individual differences has consequences for the optimization and learning of sensory substitution devices. We also discuss the extent to which these studies allow a better understanding of the experience with sensory substitution devices, and in particular how the resulting experience is not akin to a single sensory modality. Rather, it should be conceived as a multisensory experience, involving both perceptual and cognitive processes, and emerging on each user's pre-existing sensory and cognitive capacities.

Are Supramodality and Cross-Modal Plasticity the Yin and Yang of Brain Development? From Blindness to Rehabilitation.

Research in blind individuals has primarily focused for a long time on the brain plastic reorganization that occurs in early visual areas. Only more recently, scientists have developed innovative strategies to understand to what extent vision is truly a mandatory prerequisite for the brain’s fine morphological architecture to develop and function. As a whole, the studies conducted to date in sighted and congenitally blind individuals have provided ample evidence that several “visual” cortical areas develop independently from visual experience and do process information content regardless of the sensory modality through which a particular stimulus is conveyed: a property named supramodality. At the same time, lack of vision leads to a structural and functional reorganization within “visual” brain areas, a phenomenon known as cross-modal plasticity. Cross-modal recruitment of the occipital cortex in visually deprived individuals represents an adaptative compensatory mechanism that mediates processing of non-visual inputs. Supramodality and cross-modal plasticity appears to be the “yin and yang” of brain development: supramodal is what takes place despite the lack of vision, whereas cross-modal is what happens because of lack of vision. Here we provide a critical overview of the research in this field and discuss the implications that these novel findings have for the development of educative/rehabilitation approaches and sensory substitution devices (SSDs) in sensory-impaired individuals.

Designing sensory-substitution devices: Principles, pitfalls and potential1.

An exciting possibility for compensating for loss of sensory function is to augment deficient senses by conveying missing information through an intact sense. Here we present an overview of techniques that have been developed for sensory substitution (SS) for the blind, through both touch and audition, with special emphasis on the importance of training for the use of such devices, while highlighting potential pitfalls in their design. One example of a pitfall is how conveying extra information about the environment risks sensory overload. Related to this, the limits of attentional capacity make it important to focus on key information and avoid redundancies. Also, differences in processing characteristics and bandwidth between sensory systems severely constrain the information that can be conveyed. Furthermore, perception is a continuous process and does not involve a snapshot of the environment. Design of sensory substitution devices therefore requires assessment of the nature of spatiotemporal continuity for the different senses. Basic psychophysical and neuroscientific research into representations of the environment and the most effective ways of conveying information should lead to better design of sensory substitution systems. Sensory substitution devices should emphasize usability, and should not interfere with other inter- or intramodal perceptual function. Devices should be task-focused since in many cases it may be impractical to convey too many aspects of the environment. Evidence for multisensory integration in the representation of the environment suggests that researchers should not limit themselves to a single modality in their design. Finally, we recommend active training on devices, especially since it allows for externalization, where proximal sensory stimulation is attributed to a distinct exterior object.

Echoic Sensory Substitution Information in a Single Obstacle Circumvention Task.

Accurate motor control is required when walking around obstacles in order to avoid collisions. When vision is unavailable, sensory substitution can be used to improve locomotion through the environment. Tactile sensory substitution devices (SSDs) are electronic travel aids, some of which indicate the distance of an obstacle using the rate of vibration of a transducer on the skin. We investigated how accurately such an SSD guided navigation in an obstacle circumvention task. Using an SSD, 12 blindfolded participants navigated around a single flat 0.6 x 2 m obstacle. A 3-dimensional Vicon motion capture system was used to quantify various kinematic indices of human movement. Navigation performance under full vision was used as a baseline for comparison. The obstacle position was varied from trial to trial relative to the participant, being placed at two distances 25 cm to the left, right or directly ahead. Under SSD guidance, participants navigated without collision in 93% of trials. No collisions occurred under visual guidance. Buffer space (clearance between the obstacle and shoulder) was larger by a factor of 2.1 with SSD guidance than with visual guidance, movement times were longer by a factor of 9.4, and numbers of velocity corrections were larger by a factor of 5 (all p<0.05). Participants passed the obstacle on the side affording the most space in the majority of trials for both SSD and visual guidance conditions. The results are consistent with the idea that SSD information can be used to generate a protective envelope during locomotion in order to avoid collisions when navigating around obstacles, and to pass on the side of the obstacle affording the most space in the majority of trials.

Audio–Vision Substitution for Blind Individuals: Addressing Human Information Processing Capacity Limitations

In this contribution, we consider the factors that influence the information processing capacity of the person using sensory substitution devices, and the influence of how the translated information, here in audio, impacts performance. First, we review aspects of vision substitution by tactile and audio devices, and then we review key theory in human information processing limitations to devise and test use of an audio–vision substitution device, The vOICe, for recognizing visual objects with audio substitution for vision. Participants heard sonifications of two-dimensional (2-D) images and had to match them to alternatives presented either in visual or tactile modalities. To assess whether capacity limits constrain performance, objects were either presented with all information simultaneously (top and bottom as whole objects), or successively (top and bottom of the object one after the other). Performance was superior in the successive trials, indicative of a capacity limit in processing the auditory information. We discuss the implications for training protocols and design to provide a useful accessibility device for blind individuals.

Sensory Substitution: The Spatial Updating of Auditory Scenes "Mimics" the Spatial Updating of Visual Scenes.

Visual-to-auditory sensory substitution is used to convey visual information through audition, and it was initially created to compensate for blindness; it consists of software converting the visual images captured by a video-camera into the equivalent auditory images, or “soundscapes”. Here, it was used by blindfolded sighted participants to learn the spatial position of simple shapes depicted in images arranged on the floor. Very few studies have used sensory substitution to investigate spatial representation, while it has been widely used to investigate object recognition. Additionally, with sensory substitution we could study the performance of participants actively exploring the environment through audition, rather than passively localizing sound sources. Blindfolded participants egocentrically learnt the position of six images by using sensory substitution and then a judgment of relative direction task (JRD) was used to determine how this scene was represented. This task consists of imagining being in a given location, oriented in a given direction, and pointing towards the required image. Before performing the JRD task, participants explored a map that provided allocentric information about the scene. Although spatial exploration was egocentric, surprisingly we found that performance in the JRD task was better for allocentric perspectives. This suggests that the egocentric representation of the scene was updated. This result is in line with previous studies using visual and somatosensory scenes, thus supporting the notion that different sensory modalities produce equivalent spatial representation(s). Moreover, our results have practical implications to improve training methods with sensory substitution devices (SSD).

Bilateral vestibulopathy treatment: update and future directions

Bilateral vestibulopathy or BVH is a disorder of both labyrinths and/or vestibular nerves which have various etiologies. BVH is most often a chronic condition in which patients can suffer from blurred vision (oscillopsia), impaired spatial orientation and postural instability. Moreover, there is no continuing distressing vertigo, spontaneous nystagmus, or postural falls, which are typical signs of a vestibular tone imbalance caused by acute unilateral lesions. Those symptoms lead to an important decrease in physical activity, social functioning and vitality that dramatically impact the patients’ quality of life. The treatment options for various forms of BVH could be one of the following four lines of treatment: (a) Preventive treatment through prevention of ototoxicity, (b) therapeutic treatment through medical treatment of the causative underlying disease, (c) rehabilitative treatment through the vestibular rehabilitation therapy, (d) future directions through sensory substitution devices (balance prostheses technology). The prognosis of BVH is poor and more than 80% of the patients do not improve. The aim of this study was to discuss the update and the future directions in the treatment of the bilateral vestibular hypofunction (BVH).

Cross-Modal Correspondences Enhance Performance on a Colour-to-Sound Sensory Substitution Device.

Visual sensory substitution devices (SSDs) can represent visual characteristics through distinct patterns of sound, allowing a visually impaired user access to visual information. Previous SSDs have avoided colour and when they do encode colour, have assigned sounds to colour in a largely unprincipled way. This study introduces a new tablet-based SSD termed the ‘Creole’ (so called because it combines tactile scanning with image sonification) and a new algorithm for converting colour to sound that is based on established cross-modal correspondences (intuitive mappings between different sensory dimensions). To test the utility of correspondences, we examined the colour–sound associative memory and object recognition abilities of sighted users who had their device either coded in line with or opposite to sound–colour correspondences. Improved colour memory and reduced colour-errors were made by users who had the correspondence-based mappings. Interestingly, the colour–sound mappings that provided the highest improvements during the associative memory task also saw the greatest gains for recognising realistic objects that also featured these colours, indicating a transfer of abilities from memory to recognition. These users were also marginally better at matching sounds to images varying in luminance, even though luminance was coded identically across the different versions of the device. These findings are discussed with relevance for both colour and correspondences for sensory substitution use.