Sensory integration in substitution and augmentation devices

Abstract

Sensory Substitution consists in providing sensory information normally perceived through one sense by means of a different sense. Sensory Augmentation via substitution refers to using this technique to provide sensory information normally inaccessible to human perception. Both techniques have received signifi�cant attention in the last few decades, but there remains doubt about their potential and the extent to which they produce truly sensory effects instead of purely cognitive associations. This thesis seek to address the problems that have made it difficult to clear this doubt, fi�rst by examining the literature in search of both empirical and theoretical works that have previously attempted to address these questions, then carrying out empirical work informed by the conclusions of this examination. This results in two main theoretical contributions, along with three experimental contributions: The fi�rst theoretical contribution is a review of existing literature in the fi�eld, where the history of Sensory Substitution Devices (SSDs) and Sensory substitution via Augmentation Devices (SASDs) development is described, the problems associated with their evaluation are examined, and a set of integration categories (Cognitive, Perceptual and Experiential) and evaluation methods for SSDs and SASDs are reviewed and proposed for these criteria. The second contribution is a more systematic review of the fi�eld employing the Design Dimensions methodology, which categorises SSDs based on a set of common features to allow for direct comparisons through which to gain more general insights about SSD integration as a whole. The fi�ndings of this review show a general lack of results to support integration of SSDs and SASDs beyond the Cognitive category, mainly due to the non-inclusion of measurements that could produce these results in the experimental design, and identify two key areas for future research. The experimental contributions in this thesis begin with the design and testing of a novel mapping for a direction-to-touch vibrotactile belt SASD. This mapping is informed by an initial experiment that shows a consistent bias towards the spine and navel when reporting directions provided via such a belt. The novel mapping is compared to a more traditional mapping in a between/subjects comparison study, with the novel mapping showing, as expected, statistically significant� improvements in accuracy. The second contribution is a replication of Schumann & O'Regan's 2017 work on Contingency-Mimetic (i.e, leveraging natural sensorimotor contingencies to speed up perceptual learning) audio-to-direction sensory augmentation. Their audio-based system is compared to an analogous one using a tactile belt that employs the novel mapping described above. No statistical difference is found between the perceptual learning effects produced by their (allegedly contingency-mimetic) audio system vs our (allegedly non-mimetic) tactile device, in the short or long-term, casting doubt on the contingency-mimetic principle that S&O claim is responsible for their unusual results. The fi�nal contribution is a proposal of a second principle, analogous to contingencymimetism but referring to the meaning of the sensory signal, termed the Semantic-Mimetic principle. The literature is examined for evidence of both this new principle and the prior contingency-mimetic principle, yielding too few suitable comparisons for a conclusion to be drawn. The semantic-mimetic principle is further tested through an experiment that employs a single SASD under three separate target modality conditions (distance, temperature and magnetic fi�elds). No statistically signi�ficant difference is found between conditions for participants' accuracy at determining the level of the stimulus, however participant reports suggest higher levels of perceptual and experiential integration on the devices with lower semantic distance. The overall conclusion of this thesis is that perceptual and experiential integration of SASDs is likely possible in the general case, and that the main barriers to the observation of this kind of integration appear to have been caused largely by experimental design and not some inherent difficulty in creating this integration. Further, we conclude that there seems to be a fundamental difference between the three types of integration, beyond just de�finitional differences, as it is appears possible to observe changes in one but not the other.