Haptic Assistance Research Articles

Accuracy of on-site teleoperated milling with haptic assistance

PurposeIn bone surgery specialties, like orthopedics, neurosurgery, and oral and maxillofacial surgery patient safety and treatment success depends on the accurate implementation of computer-based surgical plans. Unintentional plan deviations can result in long-term functional damage to the patient. With on-site teleoperation, the surgeon operates a slave robot with a physically-decoupled master device, while being directly present at the operation site. This allows the surgeon to perform surgical tasks with robotic accuracy, while always remaining in the control loop.MethodsIn this study the master- and slave-side accuracy of an on-site teleoperated miniature cooperative robot (minaroHD) is evaluated. Master-side accuracy is investigated in a user study regarding scale factor, target feed rate, movement direction and haptic guidance stiffness. Scale factors are chosen to correspond to primarily finger, hand, and arm movements. Slave-side accuracy is investigated in autonomous milling trials regarding stepover, feed rate, movement direction, and material density.ResultsMaster-side user input errors increase with increasing target feed rate and scale factor, and decrease with increasing haptic guidance stiffness. Resulting slave-side errors decrease with increasing scale factor and are < 0.07 mm for optimal guidance parameters. Slave-side robot position errors correlate with the feed rate but show little correlation with stepover distance. For optimal milling parameters, the 95th percentile of tracked slave-side position error is 0.086 mm with a maximal error of 0.16 mm.ConclusionFor optimal guidance and milling parameters, the combined error of 0.23 mm is in the range of the dura mater thickness (< 0.27 mm) or mandibular canal wall (~ 0.85 mm). This corresponds to safety margins in high-demand surgical procedures like craniotomies, laminectomies, or decortication of the jaw. However, for further clinical translation, the performance and usability of on-site teleoperated milling must be further evaluated for real-life clinical application examples with consideration of all error sources in a computer-assisted surgery workflow.

Tele-Guidance of a Soft Magnetic Microrobot Transported by a Fluid in a Vascular Network

Electromagnetic actuation represents a novel wireless control approach utilized for the manipulation of magnetic microrobots, particularly in the context of diverse minimally invasive therapeutic applications. This study presented contributions relating to the integration of a human operator into the control system of an electromagnetic actuation framework through haptic assistance. The intervention of a human operator serves multiple purposes, encompassing the safe piloting of the microrobot during the procedure and the utilization of the doctor’s expertise. Consequently, this human-in-the-loop approach not only ensures heightened safety but also enhances public acceptability, particularly in the realm of drug delivery within the human body. To facilitate these objectives, a haptic device was proposed to propel and orient the microrobots within blood vessels, thereby enabling their targeted delivery. Additionally, a novel magnetic guidance strategy was introduced, relying on the utilization of two magnetic forces to simplify and optimize the guidance process. The electromagnetic actuation system, developed in our research laboratory, offers a comprehensive workspace that has been obtained through analytical and quantitative modeling of the magnetic field generated by the system. With an accessible workspace encompassing a cubic volume of 70 mm in length, the system facilitates easy access from all four lateral sides. Such an architectural design allows for efficient manipulation of microparticles within a significantly larger 3D workspace, surpassing the limitations imposed by traditional systems primarily confined to a small central area, as observed in existing literature. Experimental evaluations encompassing both 2D and 3D scenarios were conducted to validate the efficacy of the magnetic navigation platform.

User study of sensorless adaptive haptic assistance for bilateral teleoperation

This paper presents a sensor-less haptic adaptive assistance scheme for bilateral shared controlled in telerobotics. The paper focuses on the implementation of an adaptive assistance scheme previously developed using force sensors at the master’s side, extending it to the case where this measurement is no longer available. The experimental setup was composed of a virtual slave robot and a real master robot (a commercially-of-the-shelf, low-cost haptic device), namely the Novint Falcon haptic device. Due to the lack of sensors to measure the contact force between the human operator and the haptic control device i.e., only the position of the master robot is measured, a data-driven Unknown Input Observer with disturbance estimation augmentation is proposed, allowing the estimation of the (external) human force under plant uncertainty and external disturbances. The proposed approach is tested via a path tracking (known) with (unknown) obstacle avoidance task, and statistical analysis regarding the scheme’s effectiveness is presented.

Sensorimotor impairments during spaceflight: Trigger mechanisms and haptic assistance.

In a few years, manned space missions are planned in which the sensorimotor performance of humans will be of outstanding importance. However, research has repeatedly shown that human sensorimotor function can be impaired under conditions of microgravity. One way to compensate for these impairments is haptic feedback provided by the human-machine interface. In the current series of studies, sensorimotor performance was measured in basic aiming and tracking tasks. These tasks had to be performed using a force feedback joystick with different haptic settings (three spring stiffnesses, two dampings, two virtual masses, and no haptics). In two terrestrial studies, we investigated (1) the effects of cognitive load on performance in a dual-task paradigm (N = 10) and (2) which learning effects can be expected in these tasks in a longitudinal study design (N = 20). In the subsequent space study (N = 3 astronauts), the influence of microgravity and haptic settings of the joystick were investigated. For this purpose, three mission sessions after 2, 4, and 6 weeks on board the International Space Station (ISS), as well as terrestrial pre- and post-flight sessions, were conducted. The results of the studies indicated that (1) additional cognitive load led to longer reaction times during aiming and increased tracking error while aiming precision was not affected. (2) Significant learning effects were evident for most measures in the study on time effects. (3) Contrary to the expected learning trend, microgravity impaired the aiming precision performance of all astronauts in the initial phase of adaptation (2 weeks in space). No other significant effects were found. Intriguingly, these performance decrements could be compensated for with low to medium spring stiffness and virtual mass. The general result pattern provides further evidence that distorted proprioception during early adaptation to microgravity conditions is one main mechanism underlying sensorimotor impairment.

Review on Haptic Assistive Driving Systems Based on Drivers’ Steering-Wheel Operating Behaviour

With the availability of modern assistive techniques, ambient intelligence, and the Internet of Things (IoT), various innovative assistive environments have been developed, such as driving assistance systems (DAS), where the human driver can be provided with physical and emotional assistance. In this human–machine collaboration system, haptic interaction interfaces are commonly employed because they provide drivers with a more manageable way to interact with other components. From the view of control system theory, this is a typical closed-loop feedback control system with a human in the loop. To make such a system work effectively, both the driving behaviour factors, and the electrical–mechanical components should be considered. However, the main challenge is how to deal with the high degree of uncertainties in human behaviour. This paper aims to provide an insightful overview of the relevant work. The impact of various types of haptic assistive driving systems (haptic guidance and warning systems) on driving behaviour performance is discussed and evaluated. In addition, various driving behaviour modelling systems are extensively investigated. Furthermore, the state-of-the-art driving behaviour controllers are analysed and discussed in driver–vehicle–road systems, with potential improvements and drawbacks addressed. Finally, a prospective approach is recommended to design a robust model-free controller that accounts for uncertainties and individual differences in driving styles in a haptic assistive driving system. The outcome indicated that the haptic feedback system applied to the drivers enhanced their driving performance, lowered their response time, and reduced their mental workload compared to a system with auditory or visual signals or without any haptic system, despite some annoyances and system conflicts. The driving behaviour modelling techniques and the driving behaviour control with a haptic feedback system have shown good matching and improved the steering wheel’s base operation performance. However, mathematical principles, a statistical approach, and the lack of consideration of the individual differences in the driver–vehicle–road system make the modelling and the controller less robust and inefficient for different driving styles.

A Digital Twin Approach for Contextual Assistance for Surgeons During Surgical Robotics Training.

Minimally invasive robotic surgery copes with some disadvantages for the surgeon of minimally invasive surgery while preserving the advantages for the patient. Most commercially available robotic systems are telemanipulated with haptic input devices. The exploitation of the haptics channel, e.g., by means of Virtual Fixtures, would allow for an individualized enhancement of surgical performance with contextual assistance. However, it remains an open field of research as it is non-trivial to estimate the task context itself during a surgery. In contrast, surgical training allows to abstract away from a real operation and thus makes it possible to model the task accurately. The presented approach exploits this fact to parameterize Virtual Fixtures during surgical training, proposing a Shared Control Parametrization Engine that retrieves procedural context information from a Digital Twin. This approach accelerates a proficient use of the robotic system for novice surgeons by augmenting the surgeon’s performance through haptic assistance. With this our aim is to reduce the required skill level and cognitive load of a surgeon performing minimally invasive robotic surgery. A pilot study is performed on the DLR MiroSurge system to evaluate the presented approach. The participants are tasked with two benchmark scenarios of surgical training. The execution of the benchmark scenarios requires basic skills as pick, place and path following. The evaluation of the pilot study shows the promising trend that novel users profit from the haptic augmentation during training of certain tasks.

Usability of Haptic Volumetric Assistance for Surgical Navigation Tasks

Abstract Manual control of surgical instruments represents a sensorimotor control task with at least 3-6 degrees of freedom (DoF). The impact of haptic guidance on volumetric navigation tasks, such as milling of planned volumes for prosthesis fits or preserving sensitive tissues, is investigated. Interaction centered studies are performed to evaluate the usability of the assistance modes for navigation within a volume, along the surface of a volume and around forbidden regions. Results show that haptic assistance can reduce the number of constraint violations, if the virtual stiffness is high enough. However, haptic assistance also can increase error rates when counterforces are close to the absolute perception threshold, as a false sense of security can arise. For navigation along complex surfaces bilateral haptic constraints should be preferred, while unilateral constraints are sufficient for simple geometries. This study complements previous publications as a basis for a flexible rule-based selection or adaptation of modular haptic assistance systems.

Human–Machine Collaboration for Automated Driving Using an Intelligent Two‐Phase Haptic Interface

Automated Driving In article number 2000229, Chen Lv and co-workers propose a novel human-machine collaboration system based on an intelligent haptic interface for addressing the takeover control problem for automated vehicles. The intelligent haptic torque is applied to the steering wheel and switches its functionality between predictive guidance and haptic assistance according to the varying state and control ability of human drivers. This helps drivers gradually resume manual control during takeover.

Driving Skill Modeling Using Neural Networks for Performance-Based Haptic Assistance

This article addresses a data-driven framework, modeling expert driving skills for performance-based haptic assistance using neural networks (NNs). We have built a haptic driving training simulator to collect expert driving data and to provide proper haptic feedback. We establish an expert driving skill model by training NNs with the collected data. Then, the skill model is applied to the performance-based haptic assistance to provide optimized references of the steering/pedaling movements. We evaluate the skill model and its application to the performance-based haptic assistance in two user experiments. The results of the first experiment demonstrate that our skill model has appropriately captured experts' steering/pedaling skills. The results of the second experiment show that our performance-based haptic assistance can help novice drivers perform steering as expert drivers, but cannot assist their pedaling performance.

An MR-Safe Endovascular Robotic Platform: Design, Control, and Ex-Vivo Evaluation.

Cardiovascular diseases are the most common cause of global death. Endovascular interventions, in combination with advanced imaging technologies, are promising approaches for minimally invasive diagnosis and therapy. More recently, teleoperated robotic platforms target improved manipulation accuracy, stabilisation of instruments in the vasculature, and reduction of patient recovery times. However, benefits of recent platforms are undermined by a lack of haptics and residual patient exposure to ionising radiation. The purpose of this research was to design, implement, and evaluate a novel endovascular robotic platform, which accommodates emerging non-ionising magnetic resonance imaging (MRI). We proposed a pneumatically actuated MR-safe teleoperation platform to manipulate endovascular instrumentation remotely and to provide operators with haptic feedback for endovascular tasks. The platform task performance was evaluated in an ex vivo cannulation study with clinical experts ( N = 7) under fluoroscopic guidance and haptic assistance on abdominal and thoracic phantoms. The study demonstrated that the robotic dexterity involving pneumatic actuation concepts enabled successful remote cannulation of different vascular anatomies with success rates of 90%-100%. Compared to manual cannulation, slightly lower interaction forces between instrumentation and phantoms were measured for specific tasks. The maximum robotic interaction forces did not exceed 3N. This research demonstrates a promising versatile robotic technology for remote manipulation of endovascular instrumentation in MR environments. The results pave the way for clinical translation with device deployment to endovascular interventions using non-ionising real-time 3D MR guidance.

Constrained haptic-guided shared control for collaborative human–robot percutaneous nephrolithotomy training

Percutaneous nephrolithotomy is a procedure used to treat patients with large or irregularly shaped kidney stones. Surgical instruments are inserted through a small incision to access the kidney and remove the calculi. Surgeons who have less experience with the procedure manifest significantly higher rates of complications due to the procedure’s steep learning curve. This issue is further exacerbated by a lack of training opportunities in clinical settings. This paper introduces a teleoperative framework that can provide training to surgeons as well as assistance during procedures, based on two main components. Firstly, a type of constrained inverse kinematics that decouples the tooltip position from its orientation using a remote centre of motion, and incorporates the joint limits analytically. This reduces the workload of the procedure by having the surgeon control only the tooltip position rather than the position and the orientation while preventing the inverse kinematics from returning joint angles outside of the robot’s abilities. This kinematic framework also allows a three-degrees-of-freedom haptic device to control a six-degrees-of-freedom manipulator. Secondly, haptic feedback is provided to help guide and teach the surgeon during the procedure. Haptic feedback allows the surgeon to remain in full control during the procedure while still receiving haptic cues and assistance.Experimental results indicate that the haptic cues improved user’s accuracy, and they had shorter and smoother paths. This leads to a shorter procedure time overall. The results also indicate that the haptic assistance helped teach users the ideal trajectory of the procedure and that users who were taught with haptic feedback performed better than those who never experienced any haptic feedback.

Human–Machine Collaboration for Automated Driving Using an Intelligent Two‐Phase Haptic Interface

Prior to realizing fully autonomous driving, human intervention is periodically required to guarantee vehicle safety. This poses a new challenge in human–machine interaction, particularly during the control authority transition from automated functionality to a human driver. Herein, this challenge is addressed by proposing an intelligent haptic interface based on a newly developed two‐phase human–machine interaction model. The intelligent haptic torque is applied to the steering wheel and switches its functionality between predictive guidance and haptic assistance according to the varying state and control ability of human drivers. This helps drivers gradually resume manual control during takeover. The developed approach is validated by conducting vehicle experiments with 26 participants. The results suggest that the proposed method effectively enhances the driving state recovery and control performance of human drivers during takeover compared with an existing approach. Thus, this new method further improves the safety and smoothness of human–machine interaction in automated vehicles.

Haptic Lane-Keeping Assistance for Truck Driving: A Test Track Study.

ObjectiveThis study aims to compare the effectiveness and subjective acceptance of three designs for haptic lane-keeping assistance in truck driving.BackgroundHaptic lane-keeping assistance provides steering torques toward a reference trajectory, either continuously or only when exceeding a bandwidth. These approaches have been previously investigated in driving simulators, but it is unclear how these generalize toward real-life truck driving.MethodThree haptic lane-keeping algorithms to assist truck drivers were evaluated on a 6.3-km-long oval-shaped test track: (1) a single-bandwidth (SB) algorithm, which activated assistance torques when the predicted lateral deviation from lane center exceeded 0.4 m; (2) a double-bandwidth (DB) algorithm, which activated as SB, but deactivated after returning within 0.15 m lateral deviation; and (3) an algorithm providing assistance torques continuously (Cont) toward the lane center. Fifteen participants drove four trials each, one trial without and one for each haptic assistance design. Furthermore, participants drove with and without a concurrent visually distracting task.ResultsCompared to unsupported driving, all three assistance systems provided similar safety benefits in terms of decreased absolute lateral position and number of lane departures. Participants reported higher satisfaction and usability for Cont compared to SB.ConclusionThe continuous assistance was better accepted than bandwidth assistance, a finding consistent with prior driving simulator research. Research is still needed to investigate the long-term effects of haptic assistance on reliance and after-effects.ApplicationThe present results are useful for designers of haptic lane-keeping assistance, as driver acceptance and performance are determinants of reliance and safety, respectively.

Exploring disturbance as a force for good in motor learning.

Disturbance forces facilitate motor learning, but theoretical explanations for this counterintuitive phenomenon are lacking. Smooth arm movements require predictions (inference) about the force-field associated with a workspace. The Free Energy Principle (FEP) suggests that such ‘active inference’ is driven by ‘surprise’. We used these insights to create a formal model that explains why disturbance might help learning. In two experiments, participants undertook a continuous tracking task where they learned how to move their arm in different directions through a novel 3D force field. We compared baseline performance before and after exposure to the novel field to quantify learning. In Experiment 1, the exposure phases (but not the baseline measures) were delivered under three different conditions: (i) robot haptic assistance; (ii) no guidance; (iii) robot haptic disturbance. The disturbance group showed the best learning as our model predicted. Experiment 2 further tested our FEP inspired model. Assistive and/or disturbance forces were applied as a function of performance (low surprise), and compared to a random error manipulation (high surprise). The random group showed the most improvement as predicted by the model. Thus, motor learning can be conceptualised as a process of entropy reduction. Short term motor strategies (e.g. global impedance) can mitigate unexpected perturbations, but continuous movements require active inference about external force-fields in order to create accurate internal models of the external world (motor learning). Our findings reconcile research on the relationship between noise, variability, and motor learning, and show that information is the currency of motor learning.

Haptic Assistance That Restricts the Use of Redundant Solutions is Detrimental to Motor Learning.

Understanding the use of haptic assistance to facilitate motor learning is a critical issue, especially in the context of tasks requiring control of motor variability. However, the question of how haptic assistance should be designed in tasks with redundancy, where multiple solutions are available, is currently unknown. Here we examined the effect of haptic assistance that either allowed or restricted the use of redundant solutions on the learning of a bimanual steering task. 60 college-aged participants practiced steering a single cursor placed in between their hands along a smooth W-shaped track of a certain width as quickly as possible. Haptic assistance was either applied at (i) the 'task' level using a force channel that only constrained the cursor to the track, allowing for the use of different hand trajectories, or (ii) the 'individual effector' level using a force channel that constrained each hand to a specific trajectory. In addition, we also examined the effect of simply 'fading' assistance in a linear fashion- i.e., decreasing force gains with practice to reduce dependence on haptic assistance. Results showed all groups improved with practice - however, groups with haptic assistance at the individual effector level performed worse than those at the task level. Besides, we did not find sufficient evidence for the benefits of linearly fading assistance in our task. Overall, the results suggest that haptic assistance is not effective for motor learning when it restricts the use of redundant solutions.

Effect of Haptic Assistance Strategy on Mental Engagement in Fine Motor Tasks

This study investigates the effect of haptic control strategies on a subject’s mental engagement during a fine motor handwriting rehabilitation task. The considered control strategies include an error-reduction (ER) and an error-augmentation (EA), which are tested on both dominant and nondominant hand. A noninvasive brain–computer interface is used to monitor the electroencephalogram (EEG) activities of the subjects and evaluate the subject’s mental engagement using the power of multiple frequency bands (theta, alpha, and beta). Statistical analysis of the effect of the control strategy on mental engagement revealed that the choice of the haptic control strategy has a significant effect ([Formula: see text]) on mental engagement depending on the type of hand (dominant or nondominant). Among the evaluated strategies, EA is shown to be more mentally engaging when compared with the ER under the nondominant hand.

Augmentation of haptic feedback for teleoperated robotic surgery

PurposeA frequently mentioned lack of teleoperated surgical robots is the lack of haptic feedback. Haptics are not only able to mirror force information from the situs, but also to provide spatial guidance according to a surgical plan. However, superposition of the two haptic information can lead to overlapping and masking of the feedback and guidance forces. This study investigates different approaches toward a combination of both information and investigates effects on system usability.MethodsPreliminary studies are conducted to define parameters for two main experiments. The two main experiments constitute simulated surgical interventions where haptic guidance as well as haptic feedback provide information for the surgeon. The first main experiment considers drilling for pedicle screw placements, while the second main experiment refers to three-dimensional milling tasks such as during partial knee replacements or craniectomies. For both experiments, different guidance modes in combination with haptic feedback are evaluated regarding effectiveness (e.g., distance to target depth), efficiency and user satisfaction (e.g., detectability of discrepancies in case of technical guidance error).ResultsRegarding pedicle screw placements a combination of a peripheral visual signal and a vibration constitutes a good compromise regarding distance to target depth and detectability of discrepancies. For milling tasks, trajectory guidance is able to improve efficiency and user satisfaction (e.g., perceived workload), while boundary constraints improve effectiveness. If, assistance cannot be offered in all degrees of freedom (e.g., craniectomies), a visual substitution of the haptic force feedback shows the best results, though participants prefer using haptic force feedback.ConclusionOur results suggest that in case haptic feedback and haptic assistance are combined appropriately, benefits of both haptic modalities can be exploited. Thereby, capabilities of the human–machine system are improved compared to usage of exclusively one of the haptic information.

Virtual Reality Environments and Haptic Strategies to Enhance Implicit Learning and Motivation in Robot-Assisted Training.

Motivation plays a crucial role in motor learning and neurorehabilitation. Participants' motivation could decline to a point where they may stop training when facing a very difficult task. Conversely, participants may perform well and consider the training boring if the task is too easy. In this paper, we present a combination of a virtual reality environment with different robotic training strategies that modify task functional difficulty to enhance participants' motivation. We employed a pneumatically driven robotic stepper as a haptic interface. We first evaluated the use of disturbance observers as acceleration controllers to provide high robustness to varying system parameters, unmodeled dynamics and unknown disturbances associated with pneumatic control. The locomotor task to be learned in the virtual reality environment consisted of steering a recumbent bike to follow a desired path by changing the movement frequency of the dominant leg. The motor task was specially designed to engage implicit learning -i.e., learning without conscious recognition of what is learned. A haptic assistance strategy was developed in order to reduce the task functional difficulty during practice. In a feasibility study with eight healthy participants, we found that the haptic assistance provided by the robotic device successfully contributed to improve task performance during training, especially for less skilled participants. Furthermore, we found a negative correlation between participants' motivation and performance error when training with haptic assistance, suggesting that haptic assistance has a great potential to enhance motivation during motor training.

MEMS Based Haptic Assistive System for Physical Impairments

MEMS Based Haptic Assistive System for Physical Impairments

An embodiment paradigm in evaluation of human-in-the-loop control

This study introduces a novel approach for evaluating the quality of human-in-the-loop control using the psychological construct of embodiment in a haptic human-machine interaction task. Despite the fact that various forms of assistive control have been introduced, these methods mainly design semi-autonomous control to improve task-specific interaction performance. From a user perspective, however, the introduction of semi-autonomous control reduces controllability of the system, which could lead to negative user experience. Psychological research suggests sensory-motor factors dynamically modulate cognition of an external entity belonging to one’s own body, i.e., embodiment. In our paradigm, we apply methods for evaluating embodiment in a virtual reality (VR) environment where the human users perform a reaching task with semi-autonomous haptic assistance to measure the degree to which the embodiment is effected by the quality of semi-autonomy Our results with 8 participants show good persistence of subjective embodiment and subcomponents of presence within VR environment when a predictable assistive control is introduced while unpredictable assistance hindered the subjective embodiment. Results indicate embodiment can be exploited as a quantitative evaluation method of semi-autonomous controllers from a user-centric perspective.