Virtual Fixtures Research Articles

Toward Robot-Assisted Photoacoustic Imaging: Implementation Using the da Vinci Research Kit and Virtual Fixtures

Photoacoustic imaging of the prostate is challenging due to the limited access and limited acoustic windows to the prostate gland. We aim to develop intraoperative prostate photoacoustic imaging using the da Vinci robotic system and a pick-up ultrasound transducer that can be easily picked up and manipulated by the robot. We propose a new approach in which the da Vinci robot is programmed to acquire trajectories in a shared control configuration with “virtual fixtures”; the pick-up transducer is controlled so that it stays parallel to a single axis defined as the tomography axis, and its translation is fixed to a single plane normal to this axis. The surgeon controls the transducer motion on the tissue along this virtual fixture while the laser is fired and photoacoustic data are collected periodically. The RMS errors of the photoacoustic tomography images are 0.06 a.u. This study confirms that intraoperative da Vinci robot-assisted photoacoustic imaging with a pick-up transducer is feasible.

An operating smooth man–machine collaboration method for cataract capsulorhexis using virtual fixture

Abstract Techniques in ophthalmic surgery require higher precision while cataract capsulorhexis calls for enough operating space. There is no specific methods focusing on improving the safety and flexibility of Capsulorhexis operation in recent study. Traditionally, Guidance Virtual Fixture based on parameter curve failed to be flexible, while Forbidden-Region Virtual Fixture using PD control failed to be steady enough. This paper proposes a teleoperation control scheme using integrated virtual fixture to assist in performing cataract capsulorhexis, improving operation performance in precision and flexibility. Specifically, based on the ring pipe model presented simulating the working path and permitted operating space, the control scheme is combined attractive error-reducing constraints with variety stiffness. It works by providing composite feedback force to the master-side via haptic device and letting the user aware if he on the right path and assist him in correcting operation mistakes. Experimental evaluation of the functionality of the control scheme was tested under four different configurations to verify the effectiveness of each algorithm and the combination. Significantly, the average error of operation reduces from 0.56 mm to 0.33 mm by 41% under the control scheme and reduces from 0.56 mm to 0.11 mm by 80% under the control scheme combined with motion scaling, demonstrating improvement in control by providing accurate assistance to the operator.

Stiffness Control of Deformable Robots Using Finite Element Modeling

Due to the complexity of modeling deformable materials and infinite degrees of freedom, the rich background of rigid robot control has not been transferred to soft robots. Thus, most model-based control techniques developed for soft robots and soft haptic interfaces are specific to the particular device. In this letter, we develop a general method for stiffness control of soft robots suitable for arbitrary robot geometry and many types of actuation. Extending previous work that uses finite element modeling for position control, we determine the relationship between end-effector and actuator compliance, including the inherent device compliance, and use this to determine the appropriate controlled actuator stiffness for a desired stiffness of the end-effector. Such stiffness control, as the first component of impedance control, can be used to compensate for the natural stiffness of the deformable device and to control the robot's interaction with the environment or a user. We validate the stiffness projection on a deformable robot and include this stiffness projection in a haptic control loop to render a virtual fixture.

Enhanced teleoperation performance using hybrid control and virtual fixture

ABSTRACTTo develop secure, natural and effective teleoperation, the perception of the slave plays a key role for the interaction of a human operator with the environment. By sensing slave information, the human operator can choose the correct operation in a process during the human–robot interaction. This paper develops an integrated scheme based on a hybrid control and virtual fixture approach for the telerobot. The human operator can sense the slave interaction condition and adjust the master device via the surface electromyographic signal. This hybrid control method integrates the proportional-derivative control and the variable stiffness control, and involves the muscle activation at the same time. It is proposed to quantitatively analyse the human operator's control demand to enhance the control performance of the teleoperation system. In addition, due to unskilful operation and muscle physiological tremor of the human operator, a virtual fixture method is developed to ensure accuracy of operation and to reduce the operation pressure on the human operator. Experimental results demonstrated the effectiveness of the proposed method for the teleoperated robot.

Dynamic Virtual Fixtures Based on Path Following Control

This paper presents a path following control (PFC) approach to generate guidance virtual fixtures with dynamic constraints (time-varying constraints) for physical human-robot interaction tasks. Using input-output feedback linearization and a parallel transport frame, the system dynamics are linearized and decomposed into tangential and orthogonal direction with respect to a time-varying path, which is parametrized as a regular threefold continuously differentiable curve. The presented PFC approach is combined with compliance control, which enables the systematic generation of virtual fixtures. An experiment on a six-axis industrial robot reveals the potential of this approach to generate virtual fixtures with dynamic constraints.

A Virtual Fixtures Control Method of Surgical Robot Based on Human Arm Kinematics Model

Physical human-robot interaction(pHRI) is the most popular way to ensure the safety in robot surgery. Current research of pHRI in operation room focuses on the inside of the surgical area, which adapts virtual fixtures method for ensuring the safety of drag. But the safety problems caused by human error in the stage of dragging the robot from the outside surgical area to the inside surgical area are ignored. Therefore, a method of applying virtual fixtures to the outside of the surgical area is proposed to solve the safety issues during dragging stage outside surgical area. This method takes the kinematics model of human arm as the reference motion trajectory to construct the guided virtual fixtures, which is to restrict the robot movement in a defined area during the pHRI drag. This drag is based on admittance control method to improve safety and ensure flexibility. Experiment results show that the constructed guided virtual fixtures with the trajectory of the human arm model as the central axis, with radius 30mm, and restrict area 5mm can effectively limit the robot motion to a certain range. Simultaneously, the output speed of the robot in tangent direction of the central axis can well follow the change of the force applied by the doctor, and the output speed in the normal direction of the central axis can converge to zero stably at the pipeline boundary. Consequently, the purpose of improving the safety and flexibility of the surgical robot before surgical operation is realized.

Evaluation of Different Modes of Haptic Guidance for Robotic Surgery

Abstract The evolution of surgery has resulted in a plethora of systems varying in their application field, size, as well as degree of autonomy. Especially systems which combine principles of so-called synergistic robotic systems and master-slave-telemanipulators are interesting concerning cooperative aspects of surgeon and robotic system. While the former provide haptic guidance information (“virtual fixtures”), the latter provide haptic feedback from position or force sensor information of the slave device. During the design of these combined systems particular attention has to be paid during the allocation of information to the haptic information channel as superimposing of haptic guidance information and haptic sensor feedback can lead to concealment of essential feedback information. This paper reports on an experimental usability evaluation of different haptic guidance modes varying in their degree of autonomy as well as degree of freedom (DOF) with respect to three surgical scenarios, namely reaching a predefined position and orientation, tracking a predefined 3D trajectory, and applying a defined force (such as during e.g. a 3D-bone milling task). The goal was to evaluate whether some DOF of haptic guidance information can be left free for force sensor information feedback. General findings indicate that haptic guidance does not have to be augmented on three DOF to improve usability. Therefore, a combination of haptic sensor information feedback and haptic guidance information divided between individual DOF seems to be a potential solution.

Electromagnetic Position Sensing and Force Feedback for a Magnetic Stylus With an Interactive Display

This letter describes the design, implementation, validation, and demonstration of an electromagnetic system that can be incorporated into a graphical display to provide computer-controlled planar feedback forces on the tip of a stylus or fingertip-mounted magnet held near the display surface, according to the magnet position and virtual fixtures implemented in software. An array of magnetometer sensors is used to detect the position of the magnet, while a pair of box-shaped coils behind the display produces feedback forces on the stylus parallel to the plane of the display. Electromagnetic analysis for the system design is presented and system implementation is described. Validation results are given for force generation within a 100 mm × 100 mm area and force interaction with a virtual obstacle is demonstrated.

A Path/Surface Following Control Approach to Generate Virtual Fixtures

The workspace of a robot can be restricted by virtual fixtures to assist an operator in physical human–robot interaction tasks. This paper introduces a combination of surface following control (SFC) with compliance control and presents a path/SFC approach to systematically generate virtual fixtures. This approach allows implementation of numerous types of constraints like guidance and forbidden region virtual fixtures, hard and soft constraints, as well as static and dynamic virtual fixtures, and their combinations. Additionally, closed-loop stability proofs of the proposed control concepts are given. The flexibility of the presented approach is demonstrated by a series of measurement results from an industrial robot.

Passive Virtual Fixtures Adaptation in Minimally Invasive Robotic Surgery

During robot-aided surgical interventions, the surgeon can be benefitted from the application of virtual fixtures (VFs). Though very effective, this technique is very often not practicable in unstructured surgical environments. In order to comply with the environmental deformation, both the VF geometry and the constraint enforcement parameters need to be online defined/adapted. This letter proposes a strategy for an effective use of VF assistance in minimally invasive robotic surgical tasks. An online VF generation technique based on the interaction force measurements is presented. Pose and geometry adaptations of the VF are considered. Passivity of the overall system is guaranteed by using energy tanks passivity-based control. The proposed method is validated through experiments on the da Vinci Research Kit.

Scene Modeling and Augmented Virtuality Interface for Telerobotic Satellite Servicing

Teleoperation in extreme environments can be hindered by limitations in telemetry and in operator perception of the remote environment. Often, the primary mode of perception is visual feedback from remote cameras, which do not always provide suitable views and are subject to telemetry delays. To address these challenges, we propose to build a model of the remote environment and provide an augmented virtuality visualization system that augments the model with projections of real camera images. The approach is demonstrated in a satellite servicing scenario, with a multisecond round-trip telemetry delay between the operator on Earth and the satellite on orbit. The scene modeling enables both virtual fixtures to assist the human operator and augmented virtuality visualization that allows the operator to teleoperate a virtual robot from a convenient virtual viewpoint, with the delayed camera images projected onto the three-dimensional model. Experiments on a ground-based telerobotic platform, with software-created telemetry delays, indicate that the proposed method leads to better teleoperation performance with 30% better blade alignment and 50% reduction in task execution time compared to the baseline case where visualization is restricted to the available camera views.

Augmenting Fine Motor Skill Training with Haptic Error Amplification

This work compared two methods to augment fine motor skill training using haptic control, including techniques that guide (e.g., virtual fixtures) and challenge the trainee (e.g., error amplification). A prototype system capable of displaying virtual fixtures or error amplification during a writing task was evaluated to determine the extent to which performance in each condition transfers to an unassisted test condition. An experiment comparing training effects was executed in which participants were trained to draw a series of letters from a foreign alphabet with the non-dominant hand. The results showed that task accuracy when training with virtual fixtures did not transfer to an unassisted test condition. Training with error amplification, in contrast, increased accuracy in unaided test trials. These findings suggest that amplifying errors may be used to accelerate fine motor training for unimpaired individuals.

Haptic Guidance Based on All-Optical Ultrasound Distance Sensing for Safer Minimally Invasive Fetal Surgery.

By intervening during the early stage of gestation, fetal surgeons aim to correct or minimize the effects of congenital disorders. As compared to postnatal treatment of these disorders, such early interventions can often actually save the life of the fetus and also improve the quality of life of the newborn. However, fetal surgery is considered one of the most challenging disciplines within Minimally Invasive Surgery (MIS), owing to factors such as the fragility of the anatomic features, poor visibility, limited manoeuvrability, and extreme requirements in terms of instrument handling with precise positioning. This work is centered on a fetal laser surgery procedure treating placental disorders. It proposes the use of haptic guidance to enhance the overall safety of this procedure and to simplify instrument handling. A method is described that provides effective guidance by installing a forbidden region virtual fixture over the placenta, thereby safeguarding adequate clearance between the instrument tip and the placenta. With a novel application of all-optical ultrasound distance sensing in which transmission and reception are performed with fibre optics, this method can be used with a sole reliance on intraoperatively acquired data. The added value of the guidance approach, in terms of safety and performance, is demonstrated in a series of experiments with a robotic platform.

Relieving operators’ workload: Towards affective robotics in industrial scenarios

This paper proposes a novel approach based on affective robotics that can be applied to industrial applications. Considering a human-robot interaction task, we propose to analyze the mental workload of the operator, and subsequently adapt the behavior of the robotic system, introducing assistive technologies. These technologies would prevent the performances deterioration caused by the human stress, helping him/her only when needed and decreasing the user’s mental workload. This represents a general methodology, which can be applied to several industrial applications, leading to increase the overall performances of human-robot interaction exploiting principles of human-centered design. As a case study, we consider a teleoperation task, where virtual fixtures are utilized as an assistive technology. The stress of the operator is monitored in terms of heart rate variability, measured by means of a wearable sensor tied at the operator’s wrist. Experimental validation of the proposed architecture is performed on a group of 15 users that teleoperate an industrial robot for performing a pick and place task.

Design and implementation of haptic virtual fixtures for preoperative surgical planning

Haptic virtual fixtures (VFs) are force feedback mechanisms that enhance the human performance of tele-operative procedures where visual guidance suffers from many limitations. Many papers have explored the integration of VFs with tele-robotic procedures. However, only a few studies have included preoperative planning to their assembly. We created a novel VF design using procedures that require navigation along a path. Our design is based on the assembly of VF elements that fit along the path. To improve the design, we performed experiments to define the optimal properties in terms of highest accuracy and shortest task completion times for path-following procedures. The feasibility of the proposed method was tested on a preoperative simulated surgical planning task. Results demonstrate that the integration of the proposed VFs, which combine haptic force-field guidance and forbidden-region constraints with visual cues, increases accuracy and reduces the time taken to perform tasks.

Formal probabilistic analysis of a surgical robot control algorithm with different virtual fixtures

Surgical robots are increasingly being used in operation theaters involving normal or laparoscopic surgeries. The working of these surgical robots is highly dependent on their control algorithms, which require very rigorous analysis to ensure their correct functionality due to the safety-critical nature of surgeries. Traditionally, safety of control algorithms is ensured by simulations, but they provide incomplete and approximate analysis results due to their inherent sampling-based nature. We propose to use probabilistic model checking, which is a formal verification method, for quantitative analysis, to verify the control algorithms of surgical robots in this paper. As an illustrative example, the paper provides a formal analysis of a virtual fixture control algorithm, implemented in a neuro-surgical robot, using the PRISM model checker. In particular, we provide a formal discrete-time Markov chain-based model of the given control algorithm and its environment. This formal model is then analyzed for multiple virtual fixtures, like cubic, hexagonal and irregular shapes. This verification allowed us to discover new insights about the considered algorithm that allow us to design safer control algorithms.

3-D Pose Estimation of Articulated Instruments in Robotic Minimally Invasive Surgery.

Estimating the 3-D pose of instruments is an important part of robotic minimally invasive surgery for automation of basic procedures as well as providing safety features, such as virtual fixtures. Image-based methods of 3-D pose estimation provide a non-invasive low cost solution compared with methods that incorporate external tracking systems. In this paper, we extend our recent work in estimating rigid 3-D pose with silhouette and optical flow-based features to incorporate the articulated degrees-of-freedom (DOFs) of robotic instruments within a gradient-based optimization framework. Validation of the technique is provided with a calibrated ex-vivo study from the da Vinci Research Kit (DVRK) robotic system, where we perform quantitative analysis on the errors each DOF of our tracker. Additionally, we perform several detailed comparisons with recently published techniques that combine visual methods with kinematic data acquired from the joint encoders. Our experiments demonstrate that our method is competitively accurate while relying solely on image data.

TeMoto: Intuitive Multi-Range Telerobotic System with Natural Gestural and Verbal Instruction Interface

Teleoperated mobile robots, equipped with object manipulation capabilities, provide safe means for executing dangerous tasks in hazardous environments without putting humans at risk. However, mainly due to a communication delay, complex operator interfaces and insufficient Situational Awareness (SA), the task productivity of telerobots remains inferior to human workers. This paper addresses the shortcomings of telerobots by proposing a combined approach of (i) a scalable and intuitive operator interface with gestural and verbal input, (ii) improved Situational Awareness (SA) through sensor fusion according to documented best practices, (iii) integrated virtual fixtures for task simplification and minimizing the operator’s cognitive burden and (iv) integrated semiautonomous behaviors that further reduce cognitive burden and negate the impact of communication delays, execution latency and/or failures. The proposed teleoperation system, TeMoto, is implemented using ROS (Robot Operating System) to ensure hardware agnosticism, extensibility and community access. The operator’s command interface consists of a Leap Motion Controller for hand tracking, Griffin PowerMate USB as turn knob for scaling and a microphone for speech input. TeMoto is evaluated on multiple robots including two mobile manipulator platforms. In addition to standard, task-specific evaluation techniques (completion time, user studies, number of steps, etc.)—which are platform and task dependent and thus difficult to scale—this paper presents additional metrics for evaluating the user interface including task-independent criteria for measuring generalized (i) task completion efficiency and (ii) operator context switching.

Safety Tracking Motion Control Based on Forbidden Virtual Fixtures in Robot Assisted Nasal Surgery

Traditional nasal surgery requires the doctor’s left hand to carry the endoscope and right hand to complete the surgical operation using unstable images, potentially leading to danger due to mental fatigue, which may greatly prolong the operation time. Previously, robots in robot assisted endoscope sinus surgery (RAESS) only used simple linear motion control, lacked a dynamic tracking mechanism, and had no effective safety protection constraints, such as virtual fixtures or real-time force sensors. This paper describes an innovative and secure robotic control strategy for a dynamic automatic tracking endoscope with doctor input based on the mixed constraint of a dual-layer hyperboloid virtual fixture generated by a series of key points around the nasal cavity and a model of the threshold contact force. A tracking control strategy is set up after calibrating the navigation system and motion filtering via a human-machine interaction. Several experiments show that the hyperboloid virtual fixtures can generate an efficient constraint of different robotic motions and that the robot can automatically execute a track mission with high precision. The proposed safety protection method can be easily applied to patients with different nose shapes and greatly improve the quality of surgery.

Haptic-Based Manipulation Scheme of Magnetic Nanoparticles in a Multi-Branch Blood Vessel for Targeted Drug Delivery.

Magnetic drug targeting is a promising technique that can deliver drugs to the diseased region, while keeping the drug away from healthy parts of body. Introducing a human in the control loop of a targeted drug delivery system and using inherent bilateralism of a haptic device at the same time can considerably improve the performance of targeted drug delivery systems. In this paper, we suggest a novel intelligent haptic guidance scheme for steering a number of magnetic nanoparticles (MNPs) using forbidden region virtual fixtures and a haptic rendering scheme with multi particles. Forbidden region virtual fixtures are a general class of guidance modes implemented in software, which help a human-machine collaborative system accomplish a specific task by constraining a movement into limited regions. To examine the effectiveness of our proposed scheme, we implemented a magnetic guided drug delivery system in a virtual environment using a physics-based model of targeted drug delivery including a multi-branch blood vessel and realistic blood dynamics. We performed user studies with different guidance modes: unguided, semi virtual fixture and full virtual fixture modes. We found out that the efficiency of targeting was significantly improved using the forbidden region virtual fixture and the proposed haptic rendering of MNPs. We can expect that using intelligent haptic feedback in real targeted drug delivery systems can improve the targeting efficiency of MNPs in multi-branch vessels.