Visuo-haptic Feedback Research Articles

Responsive-ExtendedHand: Adaptive Visuo-Haptic Feedback Recognizing Object Property With RGB-D Camera for Projected Extended Hand

Responsive-ExtendedHand: Adaptive Visuo-Haptic Feedback Recognizing Object Property With RGB-D Camera for Projected Extended Hand

Hearing loss prevention at loud music events via real-time visuo-haptic feedback

Hearing loss is becoming a global problem, partly as a consequence of exposure to loud music. People may be unaware about the harmful sound levels and consequent damages caused by loud music at venues such as discotheques or festivals. Earplugs are effective in reducing the risk of noise-induced hearing loss but have been shown to be an insufficient prevention strategy. Thus, when it is not possible to lower the volume of the sound source, a viable solution to the problem is to relocate to quieter locations from time to time. In this context, this study introduces a bracelet device with the goal of warning users when the music sound level is too loud in their specific location, via haptic, visual or visuo-haptic feedback. The bracelet embeds a microphone, a microcontroller, an LED strip and four vibration motors. We performed a user study where thirteen participants were asked to react to the three kinds of feedback during a simulated disco club event where the volume of music pieces varied to reach a loud intensity. Results showed that participants never missed the above threshold notification via all types of feedback, but visual feedback led to the slowest reaction times and was deemed the least effective. In line with the findings reported in the hearing loss prevention literature, the perceived usefulness of the proposed device was highly dependent on participants’ subjective approach to the topic of hearing risks at loud music events as well as their willingness to take action regarding its prevention. Ultimately, our study shows how technology, no matter how effective, may not be able to cope with these kinds of cultural issues concerning hearing loss prevention. Educational strategies may represent a more effective solution to the real problem of changing people’s attitudes and motivations to want to protect their hearing.

Augmenting interoceptive awareness with off-the-shelf sensors using visuo-haptic emotional stimulus

Wearable sensing technologies allow us to monitor and track a wealth of information about bodily states. Tracking applications rely on abstract numerical or graphical visualizations to make this information accessible to us. However, these visualizations can be hard to interpret, and even be harmful to already vulnerable groups. Hence, we propose to give feedback in the form of an enhanced heart rate interoception and an embodied artificial agent. This method relies on the person’s inherent understanding of their own body. It is a subtle and more natural way to gauge the meaning of off-the-shelf sensors’ feedback. A wearable pet prototype that presents emotion through visuo-haptic feedback is evaluated in a match and a mismatch group. Participants in both groups answered self-report and perceived affect of the interoceptive feedback without significant differences. However, the groups’ perceived closeness to the pet differed significantly.

Wearable Haptic Device for Stiffness Rendering of Virtual Objects in Augmented Reality

We propose a novel wearable haptic device that can provide kinesthetic haptic feedback for stiffness rendering of virtual objects in augmented reality (AR). Rendering stiffness of objects using haptic feedback is crucial for realistic finger-based object manipulation, yet challenging particularly in AR due to the co-presence of a real hand, haptic device, and rendered AR objects in the scenes. By adopting passive actuation with a tendon-based transmission mechanism, the proposed haptic device can generate kinesthetic feedback strong enough for immersive manipulation and prevention of inter-penetration in a small-form-factor, while maximizing the wearability and minimizing the occlusion in AR usage. A selective locking module is adopted in the device to allow for the rendering of the elasticity of objects. We perform an experimental study of two-finger grasping to verify the efficacy of the proposed haptic device for finger-based manipulation in AR. We also quantitatively compare/articulate the effects of different types of feedbacks across haptic and visual sense (i.e., kinesthetic haptic feedback, vibrotactile haptic feedback, and visuo-haptic feedback) for stiffness rendering of virtual objects in AR for the first time.

Neural Signatures of Interface Errors in Remote Agent Manipulation

The manipulation of remote agents such as robotic arms in remote surgery or in BCI-wheelchair control are prone to errors. Some of these are related to user intent misclassification or other interface system errors, which lead to an incorrect movement. Here we focused on errors originating from unpredicted interface movements violating user intent and producing sensory conflicts. In addition, we examined effects of incongruent/congruent sensory stimuli induced by interface errors, focusing on haptic and visual cues in the system. The overarching goal was to identify the prototypical patterns of electroencephalogram (EEG) error signals associated with two types of interface errors rising when the visual and proprioceptive feedback are congruent or incongruent. For purposes of comparison validity, both types of errors were recorded in the same 3D virtual game environment.The comparison of congruent and incongruent interface errors revealed significant and marginally significant differences in EEG potentials with respect to profile, latencies, scalp distribution and sources. Different EEG time–frequency combinations had high power content. Incongruence between visual and proprioceptive feedback in interface errors not only elicited distinct EEG signal characteristics, but also produced a marginally significant Stroop effect. Incongruency in visuo-haptic feedback modalities cause a delayed user response. This effect is of major importance for the design of controlling interfaces and can provide designers with crucial information when aiming to control human response time.

Promoting eco-driving behavior through multisensory stimulation: a preliminary study on the use of visual and haptic feedback in a virtual reality driving simulator

This paper describes the design and preliminary test of a virtual reality driving simulator capable of conveying haptic and visual messages to promote eco-sustainable driving behavior. The driving simulator was implemented through the Unity game engine; a large street environment, including high-speed and urban sections, was created to examine different driving behaviors. The hardware setup included a gaming driving seat, equipped with a steering wheel and pedals; the virtual scenarios were displayed through an Oculus Rift headset to guarantee an immersive experience. Haptic stimulation (i.e., vibrations) was delivered to the driver through the accelerator pedal, while visual stimuli (i.e., icons and colors) were shown on a virtual head-up display. The sensory feedbacks were presented both alone and in combination, providing information about excessive acceleration and speed. Four different virtual scenarios, each one including a distracting element (i.e., navigator, rain, call, and traffic), were also created. Ten participants tested the simulator. Fuel consumption was evaluated by calculating a mean power index (MPI) in reference to the sensory feedback presentation; physiological reactions and responses to a usability survey were also collected. The results revealed that the haptic and visuo-haptic feedback were responsible for an MPI reduction, respectively, for 14% and 11% compared with a condition of no feedback presentation; while visual feedback alone resulted in an MPI increase of 11%. The efficacy of haptic feedback was also accompanied by a more relaxing physiological state of the users, compared with the visual stimulation. The system’s usability was adequate, although haptic stimuli were rated slightly more intrusive than the visual ones. Overall, these preliminary results highlight how promising the use of the haptic channel can be in communicating and guiding the driver toward a more eco-sustainable behavior.

Perceptual post-effects of left neck muscle vibration with visuo-haptic feedback in healthy individuals: A potential approach for treating spatial neglect

Among the techniques used to reduce spatial neglect’s symptoms, left neck muscle vibration (NMV) is alluring because it does not require the patient’s attentional co-operation. The aim of this study was to determine the type of NMV-associated feedback that induced the most intense and longest-lasting egocentric post-effects. Eighty-seven healthy individuals were randomly assigned to four intervention groups: “neck muscle vibration, blindfolded” (NMV), “neck muscle vibration with vision” (NMV + V), “neck muscle vibration and visual finger-pointing” (NMV + P), and “visual finger-pointing” (P). An eyes-closed finger-pointing subjective straight-ahead (SSA) test was carried out before the intervention, immediately afterwards, and 30 min afterwards. The results showed that only the NMV + P intervention induced a lasting leftward bias of SSA. In addition, the deviation reported in this intervention group differed significantly from those observed in the other interventions. The combination of visuo-haptic feedback and neck-somatosensory stimulation may enable a full, lasting intermodal recalibration, which could be potentiated by the attention level engaged during voluntary pointing. These outcomes highlighted that the NMV technique could easily integrate into routine occupational therapy sessions for treating various aspects of neglect disorders.

Toward Assistive Technologies for Focus Adjustment in Teleoperated Robotic Non-Contact Laser Surgery

In the last decade, focused laser radiation has become an alternative instrumentation for soft tissue surgery due to precise ablation with simultaneous coagulation. However, adjustment of the focal position with respect to the tissue surface is crucial for optimal energy input and minimised trauma. Advantageous laser delivery without tissue contact poses challenges and demands for assistance. This contribution introduces three novel concepts for user assistance during focal adaptation in noncontact laser surgery with robotic teleoperation. Visual, haptic, and visuo-haptic feedback modalities are designed based on real-time scene reconstruction and laser-to-camera registration. The performance of proposed methods was assessed in a phantom study with positioning tasks under different assistance conditions using a teleoperated extensible continuum robot. Focal position error and task execution time were selected as performance metrics. In total, 15 subjects conducted trials under four different conditions ( $N=240$ ). A mean error reduction from 1.2 mm for nonassisted to 0.25 mm for visuo-haptic feedback was revealed and error outliers were eliminated efficiently. Mean execution times of below 13 s were achieved for applied conditions. Presented work decreased the error to one-fifth in comparison to absence of assistance in teleoperation. The results strongly motivate prospective integration to surgical devices for noncontact laser delivery.

A survey of telerobotic surface finishing

Abstract This is a survey of research published on the subjects of telerobotics, haptic feedback, and mixed reality applied to surface finishing. The survey especially focuses on how visuo-haptic feedback can be used to improve a grinding process using a remote manipulator or robot. The benefits of teleoperation and reasons for using haptic feedback are presented. The use of genetic algorithms for optimizing haptic sensing is briefly discussed. Ways of augmenting the operator’s vision are described. Visual feedback can be used to find defects and analyze the quality of the surface resulting from the surface finishing process. Visual cues can also be used to aid a human operator in manipulating a robot precisely and avoiding collisions.

Point-based visuo-haptic simulation of multi-organ for virtual surgery

ABSTRACT Background and Objectives: Realistically and efficiently simulating dynamic behavior of human organs under interactions is crucial for the immersive user experience of the surgical simulator. Conventional methods are time-consuming to simulate this phenomenon due to topological modifications. Materials and Methods: This paper proposes a robust and efficient point-based framework for surgical simulation, allowing realistically simulating mechanical response of human organs under interactions with visual and tactile feedback. Considering the inevitable topological modifications occurred in surgical simulation, we adopt sparse point cloud to model the mechanics of deformable bodies while employ surface mesh to represent morphological details of human organ, which can not only disconnect mechanical complexity from geometrical details, but also enable precise boundary conditions to be solved with surface mesh. Results: We validate our method on a variety of challenging surgical scenarios, and the results demonstrate that our method can realistically and efficiently provide the visuo-haptic feedback for surgical simulation. Conclusions: Our method can well tackle the inefficiency limitation of mesh-based methods related to topological modifications issue, and has great potential to be adopted in practical surgical simulators.

An Empirical Evaluation of Visuo-Haptic Feedback on Physical Reaching Behaviors During 3D Interaction in Real and Immersive Virtual Environments

In an initial study, we characterized the properties of human reach motion in the presence or absence of visuo-haptic feedback in real and Immersive Virtual Environments (IVEs) or virtual reality within a participant’s maximum arm reach. Our goal is to understand how physical reaching actions to the perceived location of targets in the presence or absence of visuo-haptic feedback are different between real and virtual viewing conditions. Typically, participants reach to the perceived location of objects in the three-dimensional (3D) environment to perform selection and manipulation actions during 3D interaction in applications such as virtual assembly or rehabilitation. In these tasks, participants typically have distorted perceptual information in the IVE as compared to the real world, in part due to technological limitations such as minimal visual field of view, resolution, latency, and jitter. In an empirical evaluation, we asked the following questions: (i) how do the perceptual differences between virtual and real world affect our ability to accurately reach to the locations of 3D objects, and (ii) how do the motor responses of participants differ between the presence or absence of visual and haptic feedback? We examined factors such as velocity and distance of physical reaching behavior between the real world and IVE, both in the presence or absence of visuo-haptic information. The results suggest that physical reach responses vary systematically between real and virtual environments, especially in situations involving the presence or absence of visuo-haptic feedback. The implications of our study provide a methodological framework for the analysis of reaching motions for selection and manipulation with novel 3D interaction metaphors and to successfully characterize visuo-haptic versus non-visuo-haptic physical reaches in virtual and real-world situations.

Visuohaptic augmented feedback for enhancing motor skills acquisition

Serious games are accepted as an effective approach to deliver augmented feedback in motor (re-)learning processes. The multi-modal nature of the conventional computer games (e.g. audiovisual representation) plus the ability to interact via haptic-enabled inputs provides a more immersive experience. Thus, particular disciplines such as medical education in which frequent hands on rehearsals play a key role in learning core motor skills (e.g. physical palpations) may benefit from this technique. Challenges such as the impracticality of verbalising palpation experience by tutors and ethical considerations may prevent the medical students from correctly learning core palpation skills. This work presents a new data glove, built from off-the-shelf components which captures pressure sensitivity designed to provide feedback for palpation tasks. In this work the data glove is used to control a serious game adapted from the infinite runner genre to improve motor skill acquisition. A comparative evaluation on usability and effectiveness of the method using multimodal visualisations, as part of a larger study to enhance pressure sensitivity, is presented. Thirty participants divided into a game-playing group (n=15) and a control group (n=15) were invited to perform a simple palpation task. The game-playing group significantly outperformed the control group in which abstract visualisation of force was provided to the users in a blind-folded transfer test. The game-based training approach was positively described by the game-playing group as enjoyable and engaging.

A Multi-User Surface Visuo-Haptic Display Using Electrostatic Friction Modulation and Capacitive-Type Position Sensing.

This paper proposes a visuo-haptic feedback system that can provide haptic feedback to multiple users on an LCD monitor using electrostatic adhesion and built-in capacitive sensors. The prototype system consists of a 40-inch LCD monitor, an ITO electrode sheet arranged on the monitor, and multiple contact pads for electrostatic adhesion. By applying low-frequency haptic voltage and high-frequency sensing voltage to each pad, the system realizes passive haptic feedback, as well as position sensing using the same components. The passive haptic feedback force exceeds 1 N at 300 V rms. The performance of the sensing system is investigated in terms of interference, which shows the haptic voltage and existence of multiple pads can affect the sensing and can shift the output by a few millimeters. A pilot application, virtual hockey game, successfully demonstrates the sensing and haptic rendering capability of the proposed system. The effect of the interference on the demonstration is discussed.

Co-located haptic interaction for virtual USG exploration.

Ultrasonography is a widespread diagnostic technique that can take advantage of virtual reality for the purpose of training and rehearsal. The placement, orientation and body interaction of the probe is fundamental for the execution of the exploration. This paper introduces a virtual reality setup that employs visuo-haptic feedback for virtual ultrasonography. The haptic feedback is provided by desktop delta-like haptic interface with a 3D printed ultrasonography probe, and features haptic point-cloud rendering with implicit surface rendering. The visual feedback is provided by a Head Mounted Display that displays the virtual body, the probe and the operator's hand while not in contact with the probe. The system provides a co-located experience by means of precise calibration of the reference system allowing to synchronize the display of the hand and the probe with the location of the corresponding physical entities.

Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning.

Concurrent augmented feedback has been shown to be less effective for learning simple motor tasks than for complex tasks. However, as mostly artificial tasks have been investigated, transfer of results to tasks in sports and rehabilitation remains unknown. Therefore, in this study, the effect of different concurrent feedback was evaluated in trunk-arm rowing. It was then investigated whether multimodal audiovisual and visuohaptic feedback are more effective for learning than visual feedback only. Naïve subjects (N = 24) trained in three groups on a highly realistic virtual reality-based rowing simulator. In the visual feedback group, the subject's oar was superimposed to the target oar, which continuously became more transparent when the deviation between the oars decreased. Moreover, a trace of the subject's trajectory emerged if deviations exceeded a threshold. The audiovisual feedback group trained with oar movement sonification in addition to visual feedback to facilitate learning of the velocity profile. In the visuohaptic group, the oar movement was inhibited by path deviation-dependent braking forces to enhance learning of spatial aspects. All groups significantly decreased the spatial error (tendency in visual group) and velocity error from baseline to the retention tests. Audiovisual feedback fostered learning of the velocity profile significantly more than visuohaptic feedback. The study revealed that well-designed concurrent feedback fosters complex task learning, especially if the advantages of different modalities are exploited. Further studies should analyze the impact of within-feedback design parameters and the transferability of the results to other tasks in sports and rehabilitation.

The Impact of Combining Kinesthetic and Facial Expression Displays on Emotion Recognition by Users

Several studies have investigated the relevance of haptics to convey various types of emotions physically. This article investigates the improvement of the recognition rate of emotions using visuo-haptic feedback compared to facial and haptic expressions alone. Four experiments were conducted in which the recognition rates of emotions using facial, haptic and visuo-haptic expressions were tested. The first experiment evaluates the recognition rate of emotions using facial expressions. The second experiment collects a large corpus of 3D haptic expressions of certain emotions and subsequently identifies the relevant haptic expression for each emotion. The third experiment evaluates the selected haptic expressions through statistical and perceptive tests to retain the ones that result in the most accurate identification of the corresponding emotion. Finally, the fourth experiment studies the effect of visuo–haptic coupling on the recognition of the investigated emotions. Generally, emotions with high amplitudes of pleasure are better recognized in the visual modality. However, emotions with high activation are better recognized in the haptic modality. These results also highlighted the finding that participants are not equally aided by each modality when recognizing emotions efficiently. Beyond the recognition rate, multimodal expressions improved the sensation of presence and expressivity.

An interactive surgical planning tool for acetabular fractures: initial results

BackgroundAcetabular fractures still are among the most challenging fractures to treat because of complex anatomy, involved surgical access to fracture sites and the relatively low incidence of these lesions. Proper evaluation and surgical planning is necessary to achieve anatomic reduction of the articular surface and stable fixation of the pelvic ring. The goal of this study was to test the feasibility of preoperative surgical planning in acetabular fractures using a new prototype planning tool based on an interactive virtual reality-style environment.Methods7 patients (5 male and 2 female; median age 53 y (25 to 92 y)) with an acetabular fracture were prospectively included. Exclusion criterions were simple wall fractures, cases with anticipated surgical dislocation of the femoral head for joint debridement and accurate fracture reduction. According to the Letournel classification 4 cases had two column fractures, 2 cases had anterior column fractures and 1 case had a T-shaped fracture including a posterior wall fracture.The workflow included following steps: (1) Formation of a patient-specific bone model from preoperative computed tomography scans, (2) interactive virtual fracture reduction with visuo-haptic feedback, (3) virtual fracture fixation using common osteosynthesis implants and (4) measurement of implant position relative to landmarks. The surgeon manually contoured osteosynthesis plates preoperatively according to the virtually defined deformation. Screenshots including all measurements for the OR were available.The tool was validated comparing the preoperative planning and postoperative results by 3D-superimposition.ResultsPreoperative planning was feasible in all cases. In 6 of 7 cases superimposition of preoperative planning and postoperative follow-up CT showed a good to excellent correlation. In one case part of the procedure had to be changed due to impossibility of fracture reduction from an ilioinguinal approach. In 3 cases with osteopenic bone patient-specific prebent fixation plates were helpful in guiding fracture reduction. Additionally, anatomical landmark based measurements were helpful for intraoperative navigation.ConclusionThe presented prototype planning tool for pelvic surgery was successfully integrated in a clinical workflow to improve patient-specific preoperative planning, giving visual and haptic information about the injury and allowing a patient-specific adaptation of osteosynthesis implants to the virtually reduced pelvis.