Timing of the intention to act reflects action feedback expectancy.

Many motor tasks are comprised of sequentially linked action phases, as when reaching for, lifting, transporting, and replacing a cup of coffee. During such tasks, discrete visual, auditory and/or haptic feedback are typically associated with mechanical events at the completion of each action phase, as when breaking and subsequently making contact between the cup and the table. An emerging concept is that important sensorimotor control operations, that affect subsequent action phases, are centred on these discrete multisensory events. By predicting sensory feedback at the completion of action phases, and comparing with the actual feedback that arises, task performance can be continuously monitored. If errors are detected, the sensorimotor system can quickly respond with task-protective corrective actions. The aim of this study was to investigate how discrete multisensory feedback at the completion of action phases are used in these control operations. To investigate this question, 42 healthy human participants (both male and female) performed a visually guided sequential reaching task where auxiliary discrete visual, auditory and/or haptic feedback was associated with the completion of action phases. Occasionally however, this feedback was removed in one or two modalities. The results show that although the task was visually guided, its control was critically influenced by discrete auditory and haptic feedback. Multisensory integration effects occurred, that enhanced the corrective actions, when auditory feedback was unexpectedly removed along with haptic or visual feedback. This multisensory enhancement may facilitate the ability to detect errors during sequential actions and amplify task-protective corrective actions.

The use of immersive environments is increasing, especially in rehabilitation and other areas, like sports and gaming. Tasks performed in virtual/augmented reality (VR/AR) often have poorer motor performance, in part due to reduced tactile and proprioceptive inputs. We investigated whether vibrotactile haptic feedback could enhance motor performance within AR. We used the Box and Block Test (BBT) to investigate manual dexterity, where participants moved cubes over a partition using one hand. Participants performed the task in 8 conditions: Real, AR without feedback, and AR with haptic wristband or ring vibrotactile feedback using different vibration patterns (Impulse, Continuous, Hybrid). Movement features were extracted and machine learning was used to classify the experimental conditions. Results revealed that performance was consistently lower in the AR environment compared to the physical BBT. The conditions with the haptic wristband feedback gave similar performance, compared to the AR condition without haptic feedback, in terms of number of cubes moved and successful grasp rate. Conversely, conditions using the ring showed lower performance compared to AR conditions without haptics and the wristband conditions. Motion analysis of the movement trajectory revealed that using vibrotactile devices, particularly the ring, produced slower and shorter movements, as compared to AR alone. A Gradient Boosting machine learning model did not distinguish between all conditions, showing only some accuracy between AR, wristband, and ring conditions. Our study highlights the challenges of integrating haptic sensory enhancements in AR training environments, advocating a nuanced approach to developing haptic feedback systems that complement AR technologies more effectively.

Dual Users. Another way to make use of haptic devices for enhancing hands-on training is illustrated in Zhang et al. In this study, the haptic devices are not used to interact with the objects in a virtual world as in González-Mena and Neri et al., but to reproduce the expert gestures on the hands of the trainees. More precisely, the surgical tools handled by the expert are connected to individual haptic devices, each one recording in real time its connected tool trajectory. These trajectories are sent to the trainees' devices, which, in turn, guide the tools of the trainees. Thus, trainees can follow in their hands the expert tool trajectories, instead of only watching them and reproducing them on their own as usual. This experimental study suggests that haptic feedback superimposed on the trainee's motions can facilitate the performance of novice operators experiencing moments of difficulty, which was something that was already observed in other works (see the discussion of Zhang et al. for references). Even if the small sample size and use of a simple task limit the generalizability of their findings, this study illustrates that haptic training can be realized without any 3D virtual world, which requires accurate modeling for realistic haptic rendering of a complex task. Tactile Feedback. In Ratschat et al., the authors designed a shape exploration experiment to evaluate the effectiveness of multimodal tactile and kinesthetic feedback on shape perception. Sixteen participants were involved to reproduce different two-dimensional shapes with diverse characteristics in free space after exploring the shapes with two haptic feedback conditions: 1) kinesthetic feedback only and 2) kinesthetic plus tactile feedback. The kinesthetic feedback mechanism was implemented through an adapted single-degree-of-freedom SenseGlove Nova mechanism with an integrated electromagnetic brake. And tactile feedback was provided with a cable-driven platform mounted on the fingertip. To measure the participants' ability to perceive and reproduce the rendered shapes, the authors recorded the time participants spent exploring and reproducing the shapes and the error between the rendered and reproduced shapes after exploration and assessed the workload and motivation with questionnaires. Experimental results show that in a virtual shape exploration task without visual feedback, providing tactile and kinesthetic feedback is associated with more accurate and careful shape reproduction compared to exploring shapes with only kinesthetic feedback. Besides, the addition of tactile feedback does not seem to reduce the time spent during exploration, nor does it have an effect on motivation or workload. Thus, combining haptic and kinesthetic feedback could create more realistic virtual environments that may lead to better training results and easier transfer to real-world tasks, having implications across a variety of applications and training scenarios.Vibrotactile. Similarly, in Boutin et al . , the authors combined haptic (vibrotactile) feedback through the use of a haptic glove with a VR simulator for mixed-reality surgical training. They specifically focused on the potential for enhanced sensory feedback within VR. The authors chose to investigate External Ventricular Drain Placement (EVD), a common neurosurgery procedure, as a starting point. Experimental results demonstrated the simulator's accuracy, even though one major limitation was a lack of kinesthetic feedback. Like Ratschat et al., this work shows the potential to create more realistic mixed-reality environments that could extend beyond surgical applications.Kinesthetic. Force feedback plays a vital role in developing surgical skills, yet many virtual reality simulators lack this feature, creating a significant disparity between physical trainers and their digital counterparts, potentially limiting their effectiveness. In Abinaya and Manivannan, the authors take a different approach and use haptic feedback as an assessment metric for surgical training, focussing on laparoscopic surgery using a virtual reality simulator. By incorporating haptic feedback, they replicate the forces between the tool and the tissue, which directly correlate to tissue trauma. A virtual laparoscopic force model is incorporated into the simulator and used to determine the just noticeable differences of the laparoscopic grasping force. The results suggest that a simple linear model is sufficient for gripper force feedback, and a non-linear model does not affect the force perception. Expert laparoscopic surgeons agree that haptic feedback improves learning performance, and the force model improves the accuracy of object interaction during the gripping task.Innovative approaches to enhance surgical training must foster motor and sensory skills while reducing cognitive burden, lowering costs, and being conducive to faster design processes. This collection, the third on the topic of Haptic Training Simulation, underscores the potential of hapticenabled virtual reality tools in shaping the future of surgical education and improving patient outcomes.

The authors employed a virtual environment to investigate how humans use haptic and visual feedback in a simple, rhythmic object-manipulation task. The authors hypothesized that feedback would help participants identify the appropriate resonant frequency and perform online control adjustments. The 1st test was whether sensory feedback is needed at all; the 2nd was whether the motor system combines visual and haptic feedback to improve performance. Task performance was quantified in terms of work performed on the virtual inertia, ability to identify the correct rhythm, and variability of movement. Strict feedforward control was found to be ineffective for this task, even when participants had previous knowledge of the rhythm. Participants (N = 11) performed far better when feedback was available (11 times more work, 2.2 times more precise frequency, 30% less variability; p < .05 for all 3 performance measures). Using sensory feedback, participants were able to rapidly identify 4 different spring-inertia systems without foreknowledge of the corresponding resonant frequencies. They performed over 20% more work with 24% less variability when provided with both visual and haptic feedback than they did with either feedback channel alone (p < .05), providing evidence that they integrated online sensory channels. Whereas feedforward control alone led to poor performance, feedback control led to fast tuning or calibration of control according to the resonant frequency of the object, and to better control of the rhythmic movement itself.

Artificial Intelligence (AI), computer simulations, and virtual reality (VR) are increasingly becoming accessible tools that can be leveraged to implement training protocols and educational resources. Typical assessment tools related to sensory and neural processing associated with task performance in virtual environments often rely on self-reported surveys, unlike electroencephalography (EEG), which is often used to compare the effects of different types of sensory feedback (e.g., auditory, visual, and haptic) in simulation environments in an objective manner. However, it can be challenging to know which aspects of the EEG signal represent the impact of different types of sensory feedback on neural processing. Machine learning approaches offer a promising direction for identifying EEG signal features that differentiate the impact of different types of sensory feedback during simulation training. For the current study, machine learning techniques were applied to differentiate neural circuitry associated with haptic and non-haptic feedback in a simulated drilling task. Nine EEG channels were selected and analyzed, extracting different time-domain, frequency-domain, and nonlinear features, where 360 features were tested (40 features per channel). A feature selection stage identified the most relevant features, including the Hurst exponent of 13-21 Hz, kurtosis of 21-30 Hz, power spectral density of 21-30 Hz, variance of 21-30 Hz, and spectral entropy of 13-21 Hz. Using those five features, trials with haptic feedback were correctly identified from those without haptic feedback with an accuracy exceeding 90%, increasing to 99% when using 10 features. These results show promise for the future application of machine learning approaches to predict the impact of haptic feedback on neural processing during VR protocols involving drilling tasks, which can inform future applications of VR and simulation for occupational skill acquisition.

Lower limb amputation affects an estimated 1.71 million people in the US. The lack of sensory feedback and proprioception often causes loss of balance which heightens the risk of falls and injury. In this presented paper, a haptic feedback system named HapticLink was developed based on the weight distribution of the prosthetic foot to increase the individual's balance and the self-attribution of the prosthesis. The repeatability and linearity of four different force sensors were tested. The FlexiForce A201 sensors were identified as the optimal choice for the parameters and scenarios investigated. HapticLink consists of four A201 sensors, a microcontroller, and four Vibration Motors (VM). The developed system can determine and convey weight distribution on a prosthetic foot to the wearer as haptic feedback. Initial tests with Lower-Limb Prosthetic (LLP) users were conducted with quantitative results (Directional, Frequency, and Manually Applied Directional Perception tests avg. 94.44%, 79.17%, and 100%) and responses from the participants indicating that HapticLink may aid during single or double lower-limb amputee ambulation after establishing haptic feedback intensity comfort. Finally, the successful qualitative tests with a double lower-limb amputee imply the haptic feedback may be sufficient without requiring sensor fusion on the part of the participant from both the VMs and the proprioception of the contralateral leg. Clinical Relevance--- This establishes the utility of a simple, stand-alone 4:4 force sensor and haptic motor feedback system to aid during single or double lower-limb amputee ambulation.

Examining whether touch sensory feedback is necessary for science learning through experimentation: A literature review of two different lines of research across K-16

Amputees living in the developing world can benefit greatly from a dexterous low-cost robotic prosthetic hand that can be controlled via electromyography (EMG). This research addresses part of the challenge of designing and constructing such a low-cost device. In particular, the development of novel and functionally suitable fingertip sensors is presented in this paper. The sensors allowed for the user with a trans-humeral amputation to intuitively control grip strength of the robotic prosthetic hand with the help of an EMG electrode placed on the bicep muscle, as well as, a haptic sensory feedback system. The fingertip sensors illustrated a stable linear relationship with force, an even sensitivity to force over the pulp of finger and the medial and lateral sides of the finger above the distal inter-phalangeal joint across the fingertip. Additionally, it had a low cost of construction ($1.00) and the ability to fit on curved surfaces. Two test subjects evaluated the performance of the sensors in combination with the haptic sensory feedback system. The use of the novel sensors allowed for the test subjects to discriminate the forces experienced by each finger when gripping objects of different shapes, with an accuracy of 80% and 73% accuracy respectively. Hence, the fingertip sensors along with haptic feedback can provide a possible solution for amputees to regain the sense a touch and at a low cost. This is a step towards a cost effective ($(\pm\$ 150)$, yet functional robotic prosthetic hand for amputees.

In robotic bimanual teleoperation, multimodal sensory feedback plays a crucial role, providing operators with a more immersive operating experience, reducing cognitive burden and improving operating efficiency. In this study, we develop an immersive bilateral isomorphic bimanual telerobotic system, which comprises dual arms and dual dexterous hands, with visual and haptic force feedback. To assess the performance of this system, we carried out a series of experiments and investigated the user's teleoperation experience. The results demonstrate that haptic force feedback enhances physical perception capabilities and complex task operating abilities. In addition, it compensates for visual perception deficiencies and reduces the operator's work burden. Consequently, our proposed system achieves more intuitive, realistic and immersive teleoperation, improves operating efficiency and expands the complexity of tasks that robots can perform through teleoperation.

Most prosthetic hand users are limited to visual feedback of movement performance. To characterize the benefit of vibrotactile feedback for a task that lacks haptic feedback, a virtual environment was used to experimentally manipulate visual, task-relevant haptic, and remote vibrotactile feedback on simple object manipulation for unimpaired subjects. The combination of visual and remote vibrotactile feedback was compared to visual feedback alone, and to simultaneous visual and direct haptic feedback to represent ideal performance. Visual and vibrotactile feedback resulted in improvement of most performance variables including difficulty ratings relative to visual feedback alone. However addition of sensory cues to visual feedback increased trial times and the increase was steeper for vibrotactile than for haptic feedback. Specifically, during vibrotactile feedback the velocity did not change, but the duration of execution increased due to improved performance, resulting in increased trial times. This result suggests future exploration of performance improvement and execution speed for augmented sensory feedback.

The aging of the population is an irreversible trend, and the physiological problems caused by aging make the use of mobile devices a challenge. This study started from the placebo theory of human-computer interaction and pointed out that tactile feedback could make users have a positive experience. Based on the attention door model, experiments were conducted to explore whether different types of feedback could reduce the waiting time perception of users. The experiment divided the experimental group and the control group by age, conducted three types of feedback and two tasks. Using the T-test to analyze significance, and then conducted interviews after the experiment to collect qualitative data. The results showed that in short-term tasks, there were significant differences in the types of feedback that include haptic feedback. For the elderly, the addition of haptic feedback might not reduce time perception. However, in terms of preference, the elderly generally liked the type which including haptic and visual feedback. According to the Qualitative result, haptic feedback could make the operation certainly for the elderly.

Humans possess an innate ability to seamlessly coordinate movement across multiple limbs, whether driving a motor vehicle, playing a musical instrument, or performing other daily tasks. Here, supplemental sensory information, such as haptic feedback, can enhance this coordination in applications ranging from controlling teleoperated robots to prosthetic limbs and collaborative robotics. Yet, a critical gap remains in our understanding of how visual and haptic information are integrated within sensorimotor feedback systems, as well as the extent to which these sensory channels may serve as substitutes for one another. To address this gap, we conducted an experiment investigating how sensory feedback can be incorporated in a multi-limb coordination task. To determine the degree to which visual or haptic feedback dominates in multi-limb coordination, 25 participants performed a virtual cursor-to-target task using both upper limbs (via a joystick controller) and one lower limb (via a foot pedal controller). Throughout the task, we systematically manipulated visual and haptic feedback, using a vibrotactile haptic feedback algorithm that delivered task-relevant information to all three limbs. We assessed participants' task performance measures relating to trial success rates, completion times, ability to move their limbs in coordination, and overall movement efficiency. Additionally, participants completed a cognitive workload questionnaire to evaluate their perceived task difficulty level and cognitive demands. Our findings indicate that haptic feedback can effectively substitute for one degree of visual information (cursor movement along one axis). We found no significant difference between conditions where all visual cues were presented in the task and the condition where one aspect of visual feedback was replaced by haptic feedback. These results suggest that haptic feedback can, to an extent, serve as a viable alternative to visual feedback in multi-limb coordination tasks.

Actions involving fine control of the hand, for example, grasping an object, rely heavily on sensory information from the fingertips. Although the integration of feedback during the execution of individual movements is well understood, less is known about the use of sensory feedback in the control of skilled movement sequences. To address this gap, we trained participants to produce sequences of finger movements on a keyboard-like device over a 4-day training period. Participants received haptic, visual, and auditory feedback indicating the occurrence of each finger press. We then either transiently delayed or advanced the feedback for a single press by a small amount of time (30 or 60 ms). We observed that participants rapidly adjusted their ongoing finger press by either accelerating or prolonging the ongoing press, in accordance with the direction of the perturbation. Furthermore, we could show that this rapid behavioral modulation was driven by haptic feedback. Although these feedback-driven adjustments reduced in size with practice, they were still clearly present at the end of training. In contrast to the directionally specific effect we observed on the perturbed press, a feedback perturbation resulted in a delayed onset of the subsequent presses irrespective of perturbation direction or feedback modality. This observation is consistent with a hierarchical organization of even very skilled and fast movement sequences, with different levels reacting distinctly to sensory perturbations.NEW & NOTEWORTHY Sensory feedback is important during the execution of a movement. However, little is known about how sensory feedback is used during the production of movement sequences. Here, we show two distinct feedback processes in the execution of fast finger movement sequences. By transiently delaying or advancing the feedback of a single press within a sequence, we observed a directionally specific effect on the perturbed press and a directionally non-specific effect on the subsequent presses.

Reach-to-grasp movements performed without visual and haptic feedback of the hand are subject to systematic inaccuracies. Grasps directed at an object specified by binocular information usually end at the wrong distance with an incorrect final grip aperture. More specifically, moving the target object away from the observer leads to increasingly larger undershoots and smaller grip apertures. These systematic biases suggest that the visuomotor mapping is based on inaccurate estimates of an object's egocentric distance and 3D structure that compress the visual space. Here we ask whether the appropriate visuomotor mapping can be learned through an extensive exposure to trials where haptic and visual feedback of the hand is provided. By intermixing feedback trials with test trials without feedback, we aimed at maximizing the likelihood that the motor execution of test trials is positively influenced by that of preceding feedback trials. We found that the intermittent presence of feedback trials both (1) largely reduced the positioning error of the hand with respect to the object and (2) affected the shaping of the hand before the final grasp, leading to an overall more accurate performance. While this demonstrates an effective transfer of information from feedback trials to test trials, the remaining biases indicate that a compression of visual space is still taking place. The correct visuomotor mapping, therefore, could not be learned. We speculate that an accurate reconstruction of the scene at movement onset may not actually be needed. Instead, the online monitoring of the hand position relative to the object and the final contact with the object are sufficient for a successful execution of a grasp.

Bilateral telemanipulation, which applies haptic feedback to the operator, is not yet available in most commercial robot-assisted surgical systems. We have shown in previous work that the lack of haptic (force or tactile) feedback is detrimental in applications requiring fine suture manipulation. In this paper, we study the effect of substituting direct haptic feedback with visual and auditory cues. Using the da Vinci robot from Intuitive Surgical, we observed the difference between applied forces during a knot tying procedure for four different sensory feedback substitution scenarios: no feedback, auditory feedback, visual feedback, and a combination of auditory and visual feedback. Our results indicate that visual feedback, which provides continuous force information, would improve robot-assisted performance during complex surgical tasks such as knot tying with fine sutures. Discrete auditory feedback gives additional useful support to the surgeon.