Sensory Feedback Modalities Research Articles

The interplay of sensory feedback, arousal, and action tremor amplitude in essential tremor.

Essential tremor (ET) amplitude is modulated by visual feedback during target driven movements and in a grip force task. It has not been examined yet whether visual feedback exclusively modulates target force tremor amplitude or if other afferent inputs like auditory sensation has a modulatory effect on tremor amplitude as well. Also, it is unknown whether the enhanced sensory feedback causes an increase of arousal in persons with ET (p-ET). We hypothesized that (1) amplitude of tremor is modulated by variation of auditory feedback in the absence of visual feedback in a force tremor paradigm; (2) increase of tremor amplitude coincides with pupillary size as a measure of arousal. 14 p-ET and 14 matched healthy controls (HC) conducted a computer-based experiment in which they were asked to match a target force on a force sensor using their thumb and index finger. The force-induced movement was fed back to the participant visually, auditory or by a combination of both. Results showed a comparable deviation from the target force (RMSE) during the experiment during all three sensory feedback modalities. The ANOVA revealed an effect of the high vs. low feedback condition on the tremor severity (Power 4-12Hz) for the visual- and also for the auditory feedback condition in p-ET. Pupillometry showed a significantly increased pupil diameter during the auditory involved high feedback conditions compared to the low feedback conditions in p-ET. Our findings suggest that action tremor in ET is firstly modulated not only by visual feedback but also by auditory feedback in a comparable manner. Therefore, tremor modulation seems to be modality independent. Secondly, high feedback was associated with a significant pupil dilation, possibly mirroring an increased arousal/perceived effort.

P 22 Effects of multisensory feedback on tremor severity in essential tremor

Introduction : Essential tremor (ET) amplitude is modulated by visual feedback during target-driven movements: In a grip force task, tremor amplitude increases during large-scale visual feedback compared to low-scale visual feedback. It has not been examined whether visual feedback exclusively modulates action tremor severity or if an increase of other afferent input like auditory information has a modulatory effect on tremor amplitude as well. Also, it is unknown whether the sensory signal itself directly affects the tremor generating network or if the effect is rather indirect: enhanced sensory feedback during targeted driven movements might cause arousal/psychological stress. We hypothesized that (1) amplitude of ET is modulated by variation of auditory feedback in the absence of visual feedback in a force tremor paradigm; (2) increase of tremor amplitude coincides with pupillary dilation as a measure of arousal / psychological stress, independently of the quality of sensory feedback Patients & Methods : 14 ET patients conducted a computer-based experiment in which they were asked to match a target force in a force sensor using their thumb and index finger. The quality of them matching the target force was fed back to the participant visually, auditory or a combination of both. For the visual feedback, 2 bars were displayed on a screen, with the distance between these bars decreasing with approximation of the target force. In auditory feedback, a target tone of 440Hz was asked to match by a second tone which changed pitch when approximating the target force. The amplitude of feedback was manipulated via a scaling factor resulting in a large scale and a small scale feedback condition. To characterize the performance (RMSE to target force) and the tremor severity (Power 4-12 Hz) a 3x2 ANOVA was conducted. To control for arousal / psychological stress-induced via the task, pupillometry was recorded. Results : First results showed a comparable deviation from the target force (RMSE) during the experiment in all three sensory feedback modalities. Additionally, tremor severity did not differ between sensory feedback modalities. The ANOVA revealed an effect of the scaling factor on the tremor severity (Power 4-12Hz) for the visual and also for the auditory feedback condition. Pupillometry consistently showed a significantly increased pupil diameter during the large-scale visual- and auditory feedback conditions compared to the small-scale feedback conditions. Conclusion : Our preliminary findings suggest that action tremor in ET is firstly modulated not only by visual but also auditory feedback in a comparable manner and seems to be modality independent. Secondly, arousal / cognitive stress, as measured here by the pupil size, could mediate the increase of tremor amplitude. Further work is required better understand the role of these two possible mechanisms underlying target-related tremor.

Different contributions of efferent and reafferent feedback to sensorimotor temporal recalibration

Adaptation to delays between actions and sensory feedback is important for efficiently interacting with our environment. Adaptation may rely on predictions of action-feedback pairing (motor-sensory component), or predictions of tactile-proprioceptive sensation from the action and sensory feedback of the action (inter-sensory component). Reliability of temporal information might differ across sensory feedback modalities (e.g. auditory or visual), which in turn influences adaptation. Here, we investigated the role of motor-sensory and inter-sensory components on sensorimotor temporal recalibration for motor-auditory (button press-tone) and motor-visual (button press-Gabor patch) events. In the adaptation phase of the experiment, action-feedback pairs were presented with systematic temporal delays (0 ms or 150 ms). In the subsequent test phase, audio/visual feedback of the action were presented with variable delays. The participants were then asked whether they detected a delay. To disentangle motor-sensory from inter-sensory component, we varied movements (active button press or passive depression of button) at adaptation and test. Our results suggest that motor-auditory recalibration is mainly driven by the motor-sensory component, whereas motor-visual recalibration is mainly driven by the inter-sensory component. Recalibration transferred from vision to audition, but not from audition to vision. These results indicate that motor-sensory and inter-sensory components contribute to recalibration in a modality-dependent manner.

Speech auditory-motor adaptation to formant-shifted feedback lacks an explicit component: Reduced adaptation in adults who stutter reflects limitations in implicit sensorimotor learning.

The neural mechanisms underlying stuttering remain poorly understood. A large body of work has focused on sensorimotor integration difficulties in individuals who stutter, including recently the capacity for sensorimotor learning. Typically, sensorimotor learning is assessed with adaptation paradigms in which one or more sensory feedback modalities are experimentally perturbed in real time. Our own previous work on speech with perturbed auditory feedback revealed substantial auditory-motor learning limitations in both children and adults who stutter (AWS). It remains unknown, however, which subprocesses of sensorimotor learning are impaired. Indeed, new insights from research on upper limb motor control indicate that sensorimotor learning involves at least two distinct components: (a) an explicit component that includes intentional strategy use and presumably is driven by target error and (b) an implicit component that updates an internal model without awareness of the learner and presumably is driven by sensory prediction error. Here, we attempted to dissociate these components for speech auditory-motor learning in AWS versus adults who do not stutter (AWNS). Our formant-shift auditory-motor adaptation results replicated previous findings that such sensorimotor learning is limited in AWS. Novel findings are that neither control nor stuttering participants reported any awareness of changing their productions in response to the auditory perturbation and that neither group showed systematic drift in auditory target judgments made throughout the adaptation task. These results indicate that speech auditory-motor adaptation to formant-shifted feedback relies exclusively on implicit learning processes. Thus, limited adaptation in AWS reflects poor implicit sensorimotor learning. Speech auditory-motor adaptation to formant-shifted feedback lacks an explicit component: Reduced adaptation in adults who stutter reflects limitations in implicit sensorimotor learning.

FaceGuard: A Wearable System To Avoid Face Touching.

Most people touch their faces unconsciously, for instance to scratch an itch or to rest one’s chin in their hands. To reduce the spread of the novel coronavirus (COVID-19), public health officials recommend against touching one’s face, as the virus is transmitted through mucous membranes in the mouth, nose and eyes. Students, office workers, medical personnel and people on trains were found to touch their faces between 9 and 23 times per hour. This paper introduces FaceGuard, a system that utilizes deep learning to predict hand movements that result in touching the face, and provides sensory feedback to stop the user from touching the face. The system utilizes an inertial measurement unit (IMU) to obtain features that characterize hand movement involving face touching. Time-series data can be efficiently classified using 1D-Convolutional Neural Network (CNN) with minimal feature engineering; 1D-CNN filters automatically extract temporal features in IMU data. Thus, a 1D-CNN based prediction model is developed and trained with data from 4,800 trials recorded from 40 participants. Training data are collected for hand movements involving face touching during various everyday activities such as sitting, standing, or walking. Results showed that while the average time needed to touch the face is 1,200 ms, a prediction accuracy of more than 92% is achieved with less than 550 ms of IMU data. As for the sensory response, the paper presents a psychophysical experiment to compare the response time for three sensory feedback modalities, namely visual, auditory, and vibrotactile. Results demonstrate that the response time is significantly smaller for vibrotactile feedback (427.3 ms) compared to visual (561.70 ms) and auditory (520.97 ms). Furthermore, the success rate (to avoid face touching) is also statistically higher for vibrotactile and auditory feedback compared to visual feedback. These results demonstrate the feasibility of predicting a hand movement and providing timely sensory feedback within less than a second in order to avoid face touching.

The effects of dual tasking on gait synchronization during over-ground side-by-side walking

Recent studies have shown that gait synchronization during natural walking is not merely anecdotal, but it is a repeatable phenomenon that is quantifiable and is apparently related to available sensory feedback modalities. However, the mechanisms underlying this phase-locking of gait have only recently begun to be investigated. For example, it is not known what role, if any, attention plays. We employed a dual tasking paradigm in order to investigate the role attention plays in gait synchronization. Sixteen pairs of subjects walked under six conditions that manipulated the available sensory feedback and the degree of difficulty of the dual task, i.e., the attention. Movement was quantified using a trunk-mounted tri-axial accelerometer. A gait synchronization index (GSI) was calculated in order to quantify the degree of synchronization of the gait pattern. A simple dual task resulted in an increased level of synchronization, whereas a more complex dual task lead to a reduction in synchronization. Handholding increased synchronization, compared to the same attention condition without handholding. These results indicate that in order for two walkers to synchronize, some level of attention is apparently required, such that a relatively complex dual task utilizes enough attentional resources to reduce the occurrence of synchronization.

Synergistic Effects on the Elderly People's Motor Control by Wearable Skin-Stretch Device Combined with Haptic Joystick.

Cutaneous sensory feedback can be used to provide additional sensory cues to a person performing a motor task where vision is a dominant feedback signal. A haptic joystick has been widely used to guide a user by providing force feedback. However, the benefit of providing force feedback is still debatable due to performance dependency on factors such as the user's skill-level, task difficulty. Meanwhile, recent studies have shown the feasibility of improving a motor task performance by providing skin-stretch feedback. Therefore, a combination of two aforementioned feedback types is deemed to be promising to promote synergistic effects to consistently improve the person's motor performance. In this study, we aimed at identifying the effect of the combined haptic and skin-stretch feedbacks on the aged person's driving motor performance. For the experiment, 15 healthy elderly subjects (age 72.8 ± 6.6 years) were recruited and were instructed to drive a virtual power-wheelchair through four different courses with obstacles. Four augmented sensory feedback conditions were tested: no feedback, force feedback, skin-stretch feedback, and a combination of both force and skin-stretch feedbacks. While the haptic force was provided to the hand by the joystick, the skin-stretch was provided to the steering forearm by a custom-designed wearable skin-stretch device. We tested two hypotheses: (i) an elderly individual's motor control would benefit from receiving information about a desired trajectory from multiple sensory feedback sources, and (ii) the benefit does not depend on task difficulty. Various metrics related to skills and safety were used to evaluate the control performance. Repeated measure ANOVA was performed for those metrics with two factors: task scenario and the type of the augmented sensory feedback. The results revealed that elderly subjects' control performance significantly improved when the combined feedback of both haptic force and skin-stretch feedback was applied. The proposed approach suggest the feasibility to improve people's task performance by the synergistic effects of multiple augmented sensory feedback modalities.

Multimodal Perception of Histological Images for Persons Who Are Blind or Visually Impaired

Lack of suitable substitute assistive technology is a roadblock for students and scientists who are blind or visually impaired (BVI) from advancing in careers in science, technology, engineering, and mathematics (STEM) fields. It is challenging for persons who are BVI to interpret real-time visual scientific data which is commonly generated during lab experimentation, such as performing light microscopy, spectrometry, and observing chemical reactions. To address this problem, a real-time multimodal image perception system was developed to allow standard laboratory blood smear images to be perceived by BVI individuals by employing a combination of auditory, haptic, and vibrotactile feedback. These sensory feedback modalities were used to convey visual information through alternative perceptual channels, thus creating a palette of multimodal, sensory information. Two sets of image features of interest (primary and peripheral features) were applied to characterize images. A Bayesian network was applied to construct causal relations between these two groups of features. In order to match primary features with sensor modalities, two methods were conceived. Experimental results confirmed that this real-time approach produced higher accuracy in recognizing and analyzing objects within images compared to conventional tactile images.

Emergent effects in multimodal feedback from virtual buttons

The continued advancement in computer interfaces to support 3D tasks requires a better understanding of how users will interact with 3D user interfaces in a virtual workspace. This article presents two studies that investigated the effect of visual, auditory, and haptic sensory feedback modalities presented by a virtual button in a 3D environment on task performance (time on task and task errors) and user rating. Although we expected task performance to improve for conditions that combined two or three feedback modalities over a single modality, we instead found a significant emergent behavior that decreased performance in the trimodal condition. We found a significant increase in the number of presses when a user released the button before closing the virtual switch, suggesting that the combined visual, auditory, and haptic feedback led participants to prematurely believe they actuated a button. This suggests that in the design of virtual buttons, considering the effect of each feedback modality independently is not sufficient to predict performance, and unexpected effects may emerge when feedback modalities are combined.

Artificial Redirection of Sensation From Prosthetic Fingers to the Phantom Hand Map on Transradial Amputees: Vibrotactile Versus Mechanotactile Sensory Feedback

This work assesses the ability of transradial amputees to discriminate multi-site tactile stimuli in sensory discrimination tasks. It compares different sensory feedback modalities using an artificial hand prosthesis in: 1) a modality matched paradigm where pressure recorded on the five fingertips of the hand was fed back as pressure stimulation on five target points on the residual limb; and 2) a modality mismatched paradigm where the pressures were transformed into mechanical vibrations and fed back. Eight transradial amputees took part in the study and were divided in two groups based on the integrity of their phantom map; group A had a complete phantom map on the residual limb whereas group B had an incomplete or nonexisting map. The ability in localizing stimuli was compared with that of 10 healthy subjects using the vibration feedback and 11 healthy subjects using the pressure feedback (in a previous study), on their forearms, in similar experiments. Results demonstrate that pressure stimulation surpassed vibrotactile stimulation in multi-site sensory feedback discrimination. Furthermore, we demonstrate that subjects with a detailed phantom map had the best discrimination performance and even surpassed healthy participants for both feedback paradigms whereas group B had the worst performance overall. Finally, we show that placement of feedback devices on a complete phantom map improves multi-site sensory feedback discrimination, independently of the feedback modality.

Motor learning-induced changes in functional brain connectivity as revealed by means of graph-theoretical network analysis

Complex bimanual motor learning causes specific changes in activation across brain regions. However, there is little information on how motor learning changes the functional connectivity between these regions, and whether this is influenced by different sensory feedback modalities. We applied graph-theoretical network analysis (GTNA) to examine functional networks based on motor-task-related fMRI activations. Two groups learned a complex 90° out-of-phase bimanual coordination pattern, receiving either visual or auditory feedback. 3T fMRI scanning occurred before (day 0) and after (day 5) training. In both groups, improved motor performance coincided with increased functional network connectivity (increased clustering coefficients, higher number of network connections and increased connection strength, and shorter communication distances). Day×feedback interactions were absent but, when examining network metrics across all examined brain regions, the visual group had a marginally better connectivity, higher connection strength, and more direct communication pathways. Removal of feedback had no acute effect on the functional connectivity of the trained networks. Hub analyses showed an importance of specific brain regions not apparent in the standard fMRI analyses. These findings indicate that GTNA can make unique contributions to the examination of functional brain connectivity in motor learning.

Effects of Visual and Auditory Feedback on Sensorimotor Circuits in the Basal Ganglia

Previous work using visual feedback has identified two distinct sensorimotor circuits in the basal ganglia (BG): one that scaled with the duration of force and one that scaled with the rate of change of force. The present study compared functional MRI signal changes in the BG during a grip force task using either visual or auditory feedback to determine whether the BG nuclei process auditory and visual feedback similarly. We confirmed the same two sensorimotor circuits in the BG. Activation in the striatum and external globus pallidus (GPe) scaled linearly with the duration of force under visual and auditory feedback conditions, with similar slopes and intercepts across feedback type. The pattern of signal change for the internal globus pallidus (GPi) and subthalamic nucleus (STN) was nonlinear and parameters of the exponential function were altered by feedback type. Specifically, GPi and STN activation decreased exponentially with the rate of change of force. The rate constant and asymptote of the exponential functions for GPi and STN were greater during auditory than visual feedback. In a comparison of the BOLD signal between BG regions, GPe had the highest percentage of variance accounted for and this effect was preserved for both feedback types. These new findings suggest that neuronal activity of specific BG nuclei is affected by whether the feedback is derived from visual or auditory inputs. Also, the data are consistent with the hypothesis that the GPe has a high level of information convergence from other BG nuclei, which is preserved across different sensory feedback modalities.

Artifact-free sensory nerve signals obtained from cuff electrodes during functional electrical stimulation of nearby muscles

Restoration of the voluntary use of paralyzed limbs using functional neuromuscular stimulation (FNS) is limited by complex muscle properties and unpredictable load behaviors; closed-loop control of FNS would improve performance but requires reliable sensory feedback modalities. Sensory nerve signals recorded by cuff electrodes provide accurate information about forces acting on the skin in anesthetized animals; however, nerve cuff signals are very small (approximately 10 /spl mu/V), and during FNS they become contaminated with large stimulation artifacts and synchronous EMG potentials from nearby muscles. The authors show in this study that it is possible to record neural signals from the cat tibial nerve without interference from distributed stimulation of four calf muscles surrounding the recording electrode by use of high-pass filtering and synchronized bin-integration. Nerve signals sampled in this way retained all the information about footpad contact force that was normally obtained in the absence of muscle stimulation. The authors propose that this approach has wide applicability for rehabilitation of paralyzed people with neural prostheses. >

Interaction between auditory and oral sensory feedback in speech regulation.

To investigate the interaction between the auditory and oral sensory feedback modalities during speech production lingual vibrotactile thresholds were obtained from subjects in the following conditions: (1) before and after speech production with normal auditory feedback, (2) before and after speech production under exposure to auditory masking, and (3) before and after exposure to auditory masking without performing speech tasks. In addition duration measurements were obtained for selected speech sounds to investigate temporal changes in the articulatory patterns of subjects in the various conditions. Lingual sensory decreases and temporal reorganization were observed only in subjects speaking under auditory masking. These data suggest a balanced interaction between auditory and oral sensory feedback modalities which, when disturbed, results in non-phonemic change in speech production.

An interaction between auditory and oral sensory feedback in speech regulation

Investigation into the relative contribution of auditory and oral sensory feedback in speech regulation has received much attention in the last twenty years. Many studies have disrupted one or both of the primary feedback modalities in speech and reported the resulting effects on production. Recently, data has been reported which suggest an interaction between auditory and oral sensory feedback in speech regulation. To investigate further the interaction between the auditory and oral sensory feedback modalities during speech production lingual vibrotactile thresholds were obtained from Ss in the following conditions: (1) before and after speech production with normal auditory feedback, (2) before and after speech production under exposure to auditory masking, and (3) before and after exposure to auditory masking without performing speech tasks. In addition, duration measurements were obtained for selected speech sounds to investigate temporal changes in the articulatory patterns of Ss in the various conditions. Lingual sensory decreases and temporal reorganization were observed only in Ss speaking under auditory masking. These data suggest a balanced interaction between auditory and oral sensory feedback modalities which, when disturbed, results in nonphonemic changes in speech production.