Sensory Augmentation Devices Research Articles

Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device.

Current advancements in both technology and science allow us to manipulate our sensory modalities in new and unexpected ways. In the present study, we explore the potential of expanding what we perceive through our natural senses by utilizing a visual-to-auditory sensory substitution device (SSD), the EyeMusic, an algorithm that converts images to sound. The EyeMusic was initially developed to allow blind individuals to create a spatial representation of information arriving from a video feed at a slow sampling rate. In this study, we aimed to use the EyeMusic for the blind areas of sighted individuals. We use it in this initial proof-of-concept study to test the ability of sighted subjects to combine visual information with surrounding auditory sonification representing visual information. Participants in this study were tasked with recognizing and adequately placing the stimuli, using sound to represent the areas outside the standard human visual field. As such, the participants were asked to report shapes' identities as well as their spatial orientation (front/right/back/left), requiring combined visual (90° frontal) and auditory input (the remaining 270°) for the successful performance of the task (content in both vision and audition was presented in a sweeping clockwise motion around the participant). We found that participants were successful at a highly above chance level after a brief 1-h-long session of online training and one on-site training session of an average of 20 min. They could even draw a 2D representation of this image in some cases. Participants could also generalize, recognizing new shapes they were not explicitly trained on. Our findings provide an initial proof of concept indicating that sensory augmentation devices and techniques can potentially be used in combination with natural sensory information in order to expand the natural fields of sensory perception.

Can perception be extended to a “feel of north”? Tests of automaticity with the NaviEar

This study investigated the potential for the development of novel perceptual experiences through sustained training with a sensory augmentation device. We developed (1) a new geomagnetic sensory augmentation device, the NaviEar, and (2) a battery of tests for automaticity in the use of the device. The NaviEar translates head direction toward north into continuous sound according to a “wind coding” principle. To facilitate automatization of use, its design is informed by considerations of the embodiment of spatial orientation and multi-sensory integration, and it uses a sensory coding scheme derived from means for auditory perception of wind direction that is common in sailing because it is easy to understand and use. The test battery assesses different effects of automaticity (interference, rigidity of responses, and dynamic integration) assuming that automaticity is a necessary criterion to show the emergence of perceptual feel, that is, an augmented experience with perceptual phenomenal quality. We measured performance in simple training tasks, administered the tests for automaticity, and assessed subjective reports through a questionnaire. Results suggest that the NaviEar is easy and comfortable to use and has a potential for applications in real-world situations. Despite high usability, however, a 5-day training with the NaviEar did not reach levels of automaticity that are indicative of perceptual feel. We propose that the test battery for automaticity may be used as a benchmark test for iterative research on perceptual experiences in sensory augmentation and sensory substitution.

Augmenting perception: How artificial intelligence transforms sensory substitution

What happens when artificial sensors are coupled with the human senses? Using technology to extend the senses is an old human dream, on which sensory substitution and other augmentation technologies have already delivered. Laser tactile canes, corneal implants and magnetic belts can correct or extend what individuals could otherwise perceive. Here we show why accommodating intelligent sensory augmentation devices not just improves but also changes the way of thinking and classifying former sensory augmentation devices. We review the benefits in terms of signal processing and show why non-linear transformation is more than a mere improvement compared to classical linear transformation.

Self-motion perception in Parkinson's disease.

Parkinson's disease (PD), best characterized by its classic motor symptoms, also manifests non-motor symptoms including perceptual impairments. Normal motor and perceptual brain functions interact continuously in an action-perception loop; hence, perceptual and motor dysfunction in PD are likely also intertwined. A vital skill in order to maintain balance, and to move around in the environment is the ability to perceive one's own motion in space (self-motion perception). Self-motion perception is a complex brain process, that requires the integration of information from visual (optic flow), vestibular (gravito-inertial), and somatosensory senses. Yet, not much is known about self-motion perception or multisensory integration in PD. In this review, we highlight the need to better study these important functions in PD. We review perceptual deficits in underlying functions required for adept self-motion perception (visual, vestibular and somatosensory, as well as multisensory integration) and address how these might affect self-motion perception and motor function in PD. We propose that dysfunction of central brain mechanisms, implicated in impaired visual, vestibular and somatosensory function, likely impact self-motion perception in PD. Recent evidence suggests that visual and multisensory integration mechanisms of self-motion perception are indeed impaired in PD. This can affect motor control, gait and balance. Future research is needed to better investigate this important topic. A better understanding of self-motion perception and multisensory integration in PD may aid diagnosis and subtyping and may open new avenues for novel therapies to treat debilitating motor symptoms, including gait and balance impairment, using sensory augmentation devices or sensory retraining.

Using a Vibrotactile Biofeedback Device to Augment Foot Pressure During Walking in Healthy Older Adults: A Brief Report.

Human movement based on sensory control is significant to motor task performance. Thus, impairments to sensory input significantly limit feedback-type motor control. The present study introduces a vibrotactile biofeedback (BF) system which augments information regarding the user’s foot pressure to enhance gait performance. The effects of the proposed system on the gait patterns of healthy older adults and on the cognitive load during gait were evaluated; these factors are essential to clarify feasibility of the device in real-life settings. The primary task of our study was to evaluate gait along with a cognitively demanding activity in 10 healthy older adults. Regarding kinematic and kinetic data in the BF condition, the subjects had significantly increased ankle dorsiflexion during the heel contact phase in the sagittal plane and marginally increased foot pressure at the toe-off and stride length. However, such kinematic and kinetic changes were not attributed to the increased walking speed. In addition, cognitive performance (i.e., the number of correct answers) was significantly decreased in participants during gait measurements in the BF condition. These data suggest that the system had the potential for modifying the kinematic and kinetic patterns during walking but not the more comprehensive walking performance in older adults. Moreover, the device appears to place a cognitive load on older adults. This short report provides crucial primary data that would help in designing successful sensory augmentation devices and further research on a BF system.

Effects of long-term balance training with vibrotactile sensory augmentation among community-dwelling healthy older adults: a randomized preliminary study

BackgroundSensory augmentation has been shown to improve postural stability during real-time balance applications. Limited long-term controlled studies have examined retention of balance improvements in healthy older adults after training with sensory augmentation has ceased. This pilot study aimed to assess the efficacy of long-term balance training with and without sensory augmentation among community-dwelling healthy older adults.MethodsTwelve participants (four males, eight females; 75.6 ± 4.9 yrs) were randomly assigned to the experimental group (n = 6) or control group (n = 6). Participants trained in their homes for eight weeks, completing three 45-min exercise sessions per week using smart phone balance trainers that provided written, graphic, and video guidance, and monitored trunk sway. During each session, participants performed six repetitions of six exercises selected from five categories (static standing, compliant surface standing, weight shifting, modified center of gravity, and gait). The experimental group received vibrotactile sensory augmentation for four of the six repetitions per exercise via the smart phone balance trainers, while the control group performed exercises without sensory augmentation. The smart phone balance trainers sent exercise performance data to a physical therapist, who recommended exercises on a weekly basis. Balance performance was assessed using a battery of clinical balance tests (Activity Balance Confidence Scale, Sensory Organization Test, Mini Balance Evaluation Systems Test, Five Times Sit to Stand Test, Four Square Step Test, Functional Reach Test, Gait Speed Test, Timed Up and Go, and Timed Up and Go with Cognitive Task) before training, after four weeks of training, and after eight weeks of training.ResultsParticipants in the experimental group were able to use vibrotactile sensory augmentation independently in their homes. After training, the experimental group had significantly greater improvements in Sensory Organization Test and Mini Balance Evaluation Systems Test scores than the control group. Significant improvement was also observed for Five Times Sit to Stand Test duration within the experimental group, but not in the control group. No significant improvements between the two groups were observed in the remaining clinical outcome measures.ConclusionThe findings of this study support the use of sensory augmentation devices by community-dwelling healthy older adults as balance rehabilitation tools, and indicate feasibility of telerehabilitation therapy with reduced input from clinicians.

The role of sensory augmentation for people with vestibular deficits: Real-time balance aid and/or rehabilitation device?

This narrative review highlights findings from the sensory augmentation field for people with vestibular deficits and addresses the outstanding questions that are critical to the translation of this technology into clinical and/or personal use. Prior research has demonstrated that the real-time use of visual, vibrotactile, auditory, and multimodal sensory augmentation technologies can improve balance during static and dynamic stance tasks within a laboratory setting. However, its application in improving gait requires additional investigation, as does its efficacy as a rehabilitation device for people with vestibular deficits. In some locomotor studies involving sensory augmentation, gait velocity decreased and secondary task performance worsened, and subjects negatively altered their segmental control strategies when cues were provided following short training sessions. A further question is whether the retention and/or carry-over effects of training with a sensory augmentation technology exceed the retention and/or carry-over effects of training alone, thereby supporting its use as a rehabilitation device. Preliminary results suggest that there are short-term improvements in balance performance following a small number of training sessions with a sensory augmentation device. Long-term clinical and home-based controlled training studies are needed. It is hypothesized that sensory augmentation provides people with vestibular deficits with additional sensory input to promote central compensation during a specific exercise/activity; however, research is needed to substantiate this theory. Major obstacles standing in the way of its use for these critical applications include determining exercise/activity specific feedback parameters and dosage strategies. This paper summarizes the reported findings that support sensory augmentation as a balance aid and rehabilitation device, but does not critically examine efficacy or the quality of the research methods used in the reviewed studies.

Sensory augmentation: integration of an auditory compass signal into human perception of space

Bio-mimetic approaches to restoring sensory function show great promise in that they rapidly produce perceptual experience, but have the disadvantage of being invasive. In contrast, sensory substitution approaches are non-invasive, but may lead to cognitive rather than perceptual experience. Here we introduce a new non-invasive approach that leads to fast and truly perceptual experience like bio-mimetic techniques. Instead of building on existing circuits at the neural level as done in bio-mimetics, we piggy-back on sensorimotor contingencies at the stimulus level. We convey head orientation to geomagnetic North, a reliable spatial relation not normally sensed by humans, by mimicking sensorimotor contingencies of distal sounds via head-related transfer functions. We demonstrate rapid and long-lasting integration into the perception of self-rotation. Short training with amplified or reduced rotation gain in the magnetic signal can expand or compress the perceived extent of vestibular self-rotation, even with the magnetic signal absent in the test. We argue that it is the reliability of the magnetic signal that allows vestibular spatial recalibration, and the coding scheme mimicking sensorimotor contingencies of distal sounds that permits fast integration. Hence we propose that contingency-mimetic feedback has great potential for creating sensory augmentation devices that achieve fast and genuinely perceptual experiences.

Learning New Sensorimotor Contingencies: Effects of Long-Term Use of Sensory Augmentation on the Brain and Conscious Perception.

Theories of embodied cognition propose that perception is shaped by sensory stimuli and by the actions of the organism. Following sensorimotor contingency theory, the mastery of lawful relations between own behavior and resulting changes in sensory signals, called sensorimotor contingencies, is constitutive of conscious perception. Sensorimotor contingency theory predicts that, after training, knowledge relating to new sensorimotor contingencies develops, leading to changes in the activation of sensorimotor systems, and concomitant changes in perception. In the present study, we spell out this hypothesis in detail and investigate whether it is possible to learn new sensorimotor contingencies by sensory augmentation. Specifically, we designed an fMRI compatible sensory augmentation device, the feelSpace belt, which gives orientation information about the direction of magnetic north via vibrotactile stimulation on the waist of participants. In a longitudinal study, participants trained with this belt for seven weeks in natural environment. Our EEG results indicate that training with the belt leads to changes in sleep architecture early in the training phase, compatible with the consolidation of procedural learning as well as increased sensorimotor processing and motor programming. The fMRI results suggest that training entails activity in sensory as well as higher motor centers and brain areas known to be involved in navigation. These neural changes are accompanied with changes in how space and the belt signal are perceived, as well as with increased trust in navigational ability. Thus, our data on physiological processes and subjective experiences are compatible with the hypothesis that new sensorimotor contingencies can be acquired using sensory augmentation.

Head-Mounted Sensory Augmentation Device: Designing a Tactile Language.

Sensory augmentation operates by synthesizing new information then displaying it through an existing sensory channel and can be used to help people with impaired sensing or to assist in tasks where sensory information is limited or sparse, for example, when navigating in a low visibility environment. This paper presents the design of a 2nd generation head-mounted vibrotactile interface as a sensory augmentation prototype designed to present navigation commands that are intuitive, informative, and minimize information overload. We describe an experiment in a structured environment in which the user navigates along a virtual wall whilst the position and orientation of the user's head is tracked in real time by a motion capture system. Navigation commands in the form of vibrotactile feedback are presented according to the user's distance from the virtual wall and their head orientation. We test the four possible combinations of two command presentation modes (continuous, discrete) and two command types (recurring, single). We evaluated the effectiveness of this 'tactile language' according to the users' walking speed and the smoothness of their trajectory parallel to the virtual wall. Results showed that recurring continuous commands allowed users to navigate with lowest route deviation and highest walking speed. In addition, subjects preferred recurring continuous commands over other commands.

A Portable Sensory Augmentation Device for Balance Rehabilitation Using Fingertip Skin Stretch Feedback.

Neurological disorders are the leading causes of poor balance. Previous studies have shown that biofeedback can compensate for weak or missing sensory information in people with sensory deficits. These biofeedback inputs can be easily recognized and converted into proper information by the central nervous system (CNS), which integrates the appropriate sensorimotor information and stabilizes the human posture. In this study, we proposed a form of cutaneous feedback which stretches the fingertip pad with a rotational contactor, so-called skin stretch. Skin stretch at a fingertip pad can be simply perceived and its small contact area makes it favored for small wearable devices. Taking advantage of skin stretch feedback, we developed a portable sensory augmentation device (SAD) for rehabilitation of balance. SAD was designed to provide postural sway information through additional skin stretch feedback. To demonstrate the feasibility of the SAD, quiet standing on a force plate was evaluated while sensory deficits were simulated. Fifteen healthy young adults were asked to stand quietly under six sensory conditions: three levels of sensory deficits (normal, visual deficit, and visual + vestibular deficits) combined with and without augmented sensation provided by SAD. The results showed that augmented sensation via skin stretch feedback helped subjects correct their posture and balance, especially as the deficit level of sensory feedback increased. These findings demonstrate the potential use of skin stretch feedback in balance rehabilitation.

Sensory Augmentation for the Blind

Common navigational aids used by blind travelers during large-scale navigation divert attention away from important cues of the immediate environment (i.e., approaching vehicles). Sensory augmentation devices, relying on principles similar to those at work in sensory substitution, can potentially bypass the bottleneck of attention through sub-cognitive implementation of a set of rules coupling motor actions with sensory stimulation. We provide a late blind subject with a vibrotactile belt that continually signals the direction of magnetic north. The subject completed a set of behavioral tests before and after an extended training period. The tests were complemented by questionnaires and interviews. This newly supplied information improved performance on different time scales. In a pointing task we demonstrate an instant improvement of performance based on the signal provided by the device. Furthermore, the signal was helpful in relevant daily tasks, often complicated for the blind, such as keeping a direction over longer distances or taking shortcuts in familiar environments. A homing task with an additional attentional load demonstrated a significant improvement after training. The subject found the directional information highly expedient for the adjustment of his inner maps of familiar environments and describes an increase in his feeling of security when exploring unfamiliar environments with the belt. The results give evidence for a firm integration of the newly supplied signals into the behavior of this late blind subject with better navigational performance and more courageous behavior in unfamiliar environments. Most importantly, the complementary information provided by the belt lead to a positive emotional impact with enhanced feeling of security. The present experimental approach demonstrates the positive potential of sensory augmentation devices for the help of handicapped people.

Neural interface technologies: applications of biomedical engineering to neurosurgery

The neurosurgical approach to diseases of the nervous system continues to evolve. As bioengineering technologies have advanced, an ally has been found to expand the role of the neurosurgeon in treating diseases not previously considered eligible for intervention. Not only are they purveyors of life-saving surgical treatments, but neurosurgeons have also embraced the application of new therapies for pain and disability, treating these as targeted disease entities in and of themselves. Consider the dramatic rise in implant technologies such as spinal pedicle screws and spinal cord stimulators to treat back pain, deep brain stimulators for functional disabilities, vagal nerve stimulators for epilepsy and depression, and peripheral nerve stimulators for pain and urinary incontinence. Sensory augmentation devices such as retinal prostheses and cochlear implant devices are in various stages of development by non-neurosurgeons and engineers. Technological developments have provided neuroscience-based clinicians with a venue to consider applications well beyond the purview of their predecessors. Many lessons can be gleaned from these and other “foreign” technologies, thereby accelerating the collaboration between the previously disparate disciplines of engineering and surgery and providing a venue for shared interests in the field of neural interface devices. In the future, leading neurosurgeons will continue to drive their craft forward by identifying the potential neurosurgical uses for emerging technology, directing engineering concepts toward surgical applications, and incorporating the technologies into daily practice.