Virtual Fixtures Research Articles

Intelligent Control for Human-Robot Cooperation in Orthopedics Surgery.

Cooperation between surgeon and robot is one of the key technologies that limit the robot to be widely used in orthopedic clinics. In this study, the evolution of human-robot cooperation methods and the control strategies for typical human-robot cooperation in robot-assisted orthopedics surgery were reviewed at first. Then an intelligent admittance control method, which combines the fuzzy model reference learning control with the virtual constraint control, is proposed to solve the requirements of intuitive human-robot interaction during orthopedics surgery. That is, a variable damping parameter model of the admittance control based on fuzzy model learning control algorithm is introduced to make the robot move freely by using the reference model of surgeon's motion equation with the minimum jerk trajectory. And the virtual constraint control method based on the principle of virtual fixture is adopted to make the robot move within the pre-defined area so as to perform more safe surgery. The basic principle and its realization of this intelligent control method are described in details. At last, a test platform is built based on our designed 6 DOF articulated robot. Experiments of safety and precision on acrylic model with this method show that the robot has the ability of better intuitive interaction and the high precision. And the pilot experiment of bone tumor resection on sawbone model shows the effectiveness of this method.

An evaluation for tube assembly performance based on virtual fixtures

There are a lot of complex tubing in aviation, aerospace, automobile, and other industrial machinery and equipment. The reliable assembly of these tubing directly affects the quality of the product...

Performance metrics for guidance active constraints in surgical robotics.

Active constraint (AC)/virtual fixture (VF) is among the most popular approaches towards the shared execution of subtasks by the surgeon and robotic systems. As more possibilities appear for the implementation of ACs in surgical scenarios, the need to introduce methods that guarantee a safe and intuitive user-interaction increases. The presence of the human in the loop adds a layer of interactivity and adaptability that renders the assessment of such methods non-trivial. In most works, guidance ACs have been evaluated mainly in terms of enhancement of accuracy and completion time with little regard for other aspects such as human factors, even though the continuous engagement of these methods can considerably degrade the user experience. This paper proposes a set of performance metrics and considerations that can help evaluate guidance ACs with reference to accuracy enhancement, force characteristics and subjective aspects. The use of these metrics is demonstrated through two sets of experiments on 12 surgeons and 6 inexperienced users.

Development of an Assistive Robotic System with Virtual Assistance to Enhance Play for Children with Disabilities: A Preliminary Study

Children with disabilities typically have fewer opportunities for manipulation and play, due to their physical limitations, resulting in delayed cognitive and perceptual development. A switched-controlled device can remotely do tasks for a child or a human helper can mediate the child’s interaction with the environment during play. However, these approaches disconnect children from the environment and limit their opportunities for interactive play with objects. This paper presents a novel application of a robotic system with virtual assistance, implemented by virtual fixtures, to enhance interactive object play for children in a set of coloring tasks. The assistance conditions included zero assistance (No-walls), medium level assistance (Soft-walls) and high level assistance (Rigid-walls), which corresponded to the magnitude of the virtual fixture forces. The system was tested with fifteen able-bodied adults and results validated the effectiveness of the system in improving the user’s performance. The Soft- and Rigid-walls conditions significantly outperformed the No-walls condition and led to relatively the same performance improvements in terms of: (a) a statistically significant reduction in the ratio of the colored area outside to the colored area inside the region of interest (with large effect sizes, Cohen’s d > .8), (b) and a substantial reduction in the travelled distance outside the borders (with large effect sizes). The developed platform will next be tested with typically developing children and then children with disabilities. Future development will include adding artificial intelligence to adaptively tune the level of assistance according to the user’s level of performance (i.e. providing more assistance only when the user is committing more errors).

Preliminary testing by adults of a haptics-assisted robot platform designed for children with physical impairments to access play

ABSTRACTDevelopment of children’s cognitive and perceptual skills depends heavily on object exploration and experience in their physical world. For children who have severe physical impairments, one of the biggest concerns is the loss of opportunities for meaningful play with objects, including physical contact and manipulation. Assistive robots can enable children to perform object manipulation through the control of simple interfaces. Touch sensations conveyed through haptic interfaces in the form of force reflection or force assistance can help a child to sense the environment and to control a robot. A robotic system with forbidden region virtual fixtures (VFs) was tested in an object sorting task. Three sorting tasks—by color, by shape, and by both color and shape—were performed by 10 adults without disability and one adult with cerebral palsy. Tasks performed with VFs were accomplished faster than tasks performed without VFs, and deviations of the motion area were smaller with VFs than without VFs. For the participant with physical impairments, two out of three tasks were slower with the VFs. This implies that forbidden region VFs are not always able to improve user task performance. Alignment with an individual’s unique motion characteristics can improve VF assistance.

A Hierarchical Optimization Approach to Robot Teleoperation and Virtual Fixtures Rendering

This paper presents a novel controller for bilateral teleoperation based on hierarchical constrained optimization techniques, that exploits motion constraints for the definition of virtual fixtures and haptic rendering. Our method allows the description of both rigid and compliant virtual fixtures, without the need of relative weight tuning as in previous approaches. Moreover, kinesthetic feedback due to each active constraint can be directly evaluated from the dual solution of the optimization algorithm, removing constraint penetration typical of penalty-based approaches and critical for telemanipulation where high accuracy is required. A transparency and stability analysis in case of communication delays is provided by relying on the classical two-port network formalism and linear control theory. An experimental evaluation of the approach for an object tracking task is carried out on a teleoperation system consisting of a 7-axis position-controlled slave robot with an eye-in-hand camera, and a 3 d.o.f. force-controlled master device.

Force-Controlled Exploration for Updating Virtual Fixture Geometry in Model-Mediated Telemanipulation

This paper proposes an approach for using force-controlled exploration data to update and register an a priori virtual fixture geometry to a corresponding deformed and displaced physical environment. An approach for safe exploration implementing hybrid motion/force control is presented on the slave robot side. During exploration, the shape and the local surface normals of the environment are estimated and saved in an exploration data set. The geometric data collected during this exploration scan are used to deform and register the a priori environment model to the exploration data set. The environment registration is achieved using a deformable registration based on the coherent point drift method. The task-description of the high-level assistive telemanipulation law, called a virtual fixture (VF), is then deformed and registered in the new environment. The new model is updated and used within a model-mediated telemanipulation framework. The approach is experimentally validated using a da-Vinci research kit (dVRK) master interface, a dVRK patient side manipulator, and a Cartesian stage robot. Experiments demonstrate that the updated VF and the updated model allow the users to improve their path following performance and to shorten their completion time when the updated path following VF is applied. The approach presented has direct bearing on a multitude of surgical applications including force-controlled ablation.

A prototype telerobotic platform for live transmission line maintenance: Review of design and development

This paper reports technical design of a novel experimental test facility, using haptic-enabled teleoperation of robotic manipulators, for live transmission line maintenance. The goal is to study and develop appropriate techniques in repair overhead power transmission lines by allowing linemen to wirelessly guide a remote manipulator, installed on a crane bucket, to execute dexterous maintenance tasks, such as twisting a tie wire around a cable. Challenges and solutions for developing such a system are outlined. The test facility consists of a PHANToM Desktop haptic device (master site), an industrial hydraulic manipulator (slave site) mounted atop a Stewart platform, and a wireless communication channel connecting the master and slave sites. The teleoperated system is tested under different force feedback schemes, while the base is excited and the communication channel is delayed and/or lossy to emulate realistic network behaviors. The force feedback schemes are: virtual fixture, augmentation force and augmented virtual fixture. Performance of each scheme is evaluated under three measures: task completion time, number of failed trials and displacement of the slave manipulator end-effector. The developed test rig has been shown to be successful in performing haptic-enabled teleoperation for live-line maintenance in a laboratory setting. The authors aim at establishing a benchmark test facility for objective evaluation of ideas and concepts in the teleoperation of live-line maintenance tasks.

Applying Virtual Fixtures to the Distal End of a Minimally Invasive Surgery Instrument

The comanipulation paradigm, in which a user and a robot simultaneously hold a tool, allows for gesture guidance. In particular, virtual fixtures, which are geometrical constraints imposed to the tool by the robot, have received great interest in the domain of surgical applications. So far, this concept has been implemented in the context of open surgery. This paper explores the application of virtual fixtures for minimally invasive surgery, in which the tool is inserted in the patient through a fulcrum. Here, a key issue is to return to the surgeon forces that are virtually applied at the instrument distal tip, while the robot is physically attached to the instrument proximal handle. To this aim, two approaches are investigated. A first approach consists of applying a full wrench at the proximal end of the instrument that is equal to the wrench constituted by a pure force applied to the instrument's distal tip. A second approach consists of applying a pure force to the instrument proximal end, thanks to a lever model about the fulcrum. The two approaches are compared through experiments, during which naive subjects blindly perform virtual object palpation and robot-guided movements. During experiments, indicators involving motion and force analysis are computed. The user capacity to distinguish between several virtual objects is evaluated as well. Although drastically different, the two approaches provide assistance with a similar level of efficiency.

Haptic Assisted Teleoperation Based on Virtual Fixture and Dynamic Modelling

Haptic Assisted Teleoperation Based on Virtual Fixture and Dynamic Modelling

Human-in-the-loop active electrosense

Active electrosense is a non-visual, short range sensing system used by weakly electric fish, enabling such fish to locate and identify objects in total darkness. Here we report initial findings from the use of active electrosense for object localization during underwater teleoperation with a virtual reality (VR) head-mounted display (HMD). The advantage of electrolocating with a VR system is that it naturally allows for aspects of the task that are difficult for a person to perform to be allocated to the computer. However, interpreting weak and incomplete patterns in the incoming data is something that people are typically far better at than computers. To achieve human–computer synergy, we integrated an active electrosense underwater robot with the Oculus Rift HMD. The virtual environment contains a visualization of the electric images of the objects surrounding the robot as well as various virtual fixtures that guide users to regions of higher information value. Initial user testing shows that these fixtures significantly reduce the time taken to localize an object, but may not increase the accuracy of the position estimate. Our results highlight the advantages of translating the unintuitive physics of electrolocation to an intuitive visual representation for accomplishing tasks in environments where imaging systems fail, such as in dark or turbid water.

Calibration and Integration of B-Mode Optical Coherence Tomography for Assistive Control in Robotic Microsurgery

During retinal microsurgery, surgeons cannot adequately visualize subsurface anatomical structures. In our previous work, a customized B-mode optical coherence tomography (OCT) probe was integrated into an ophthalmic robotic system to provide depth perception. This paper presents new approaches for implementing and achieving real-time feedback and assistive robotic control based on B-mode OCT imaging. The robotic system was comprised of a parallel robot, a micro-injection tool, and a telemanipulation master interface. A method for calibrating the B-mode OCT image scaling and distortion was presented using thin plate splines. Determining the OCT scanning plane relative to the robot base frame is presented through experiments and analyzed for sensitivity. A dual-rate controller using low frequency OCT feedback and high frequency position servoing was presented and tested for accuracy and latency. Three-dimensional assistive telemanipulation virtual fixtures based on microscope and OCT feedback are presented. The experimental evaluation demonstrated following target anatomy and semi-automated micro-injection. These results present the key steps towards achieving an integrated system for OCT feedback control using a miniature intraocular B-mode probe.

System Integration and In Vivo Testing of a Robot for Ultrasound Guidance and Monitoring During Radiotherapy.

We are developing a cooperatively controlled robot system for image-guided radiation therapy (IGRT) in which a clinician and robot share control of a 3-D ultrasound (US) probe. IGRT involves two main steps: 1) planning/simulation and 2) treatment delivery. The goals of the system are to provide guidance for patient setup and real-time target monitoring during fractionated radiotherapy of soft tissue targets, especially in the upper abdomen. To compensate for soft tissue deformations created by the probe, we present a novel workflow where the robot holds the US probe on the patient during acquisition of the planning computerized tomography image, thereby ensuring that planning is performed on the deformed tissue. The robot system introduces constraints (virtual fixtures) to help to produce consistent soft tissue deformation between simulation and treatment days, based on the robot position, contact force, and reference US image recorded during simulation. This paper presents the system integration and the proposed clinical workflow, validated by an in vivo canine study. The results show that the virtual fixtures enable the clinician to deviate from the recorded position to better reproduce the reference US image, which correlates with more consistent soft tissue deformation and the possibility for more accurate patient setup and radiation delivery.

Application of Virtual Reality in Side Clamp Fixture Assembly

Background/Objectives: The main objective of this paper is dealt with the application of virtual reality in side clamp fixture assembly. Methods/Statistical Analysis: The proposed methodology utilized the geometry and assembly details of side clamp assembly and makes the fixture assembly as close as possible to the real one. It includes two stages. The first stage demonstrates the procedure used for creating virtual prototype of an assembly and the second stage includes the creation of virtual fixture assembly. Findings: VRML coding is used to simulate virtual assembly process. Based on interference analysis errors can be rectified. Fixture assembly planning in virtual environment gives better understanding of the design engineer and the virtual assembly planning gives clear detail for virtual manufacturing. It enables intuitive assembly sequences which are then evaluated by Vizard inspector module. Application/Improvement: In this paper fixture assembly sequences are simulated using VIZARD software. Future research will investigate the fixture assembly plan in a complex virtual manufacturing environment.

Cooperative Control with Ultrasound Guidance for Radiation Therapy

Radiation therapy typically begins with the acquisition of a CT scan of the patient for planning, followed by multiple days where radiation is delivered according to the plan. This requires that the patient be reproducibly positioned (set up) on the radiation therapy device (linear accelerator) such that the radiation beams pass through the target. Modern linear accelerators provide cone-beam computed tomography (CBCT) imaging, but this does not provide sufficient contrast to discriminate many abdominal soft-tissue targets and therefore patient setup is often done by aligning bony anatomy or implanted fiducials. Ultrasound (US) can be used to both assist with patient setup and to provide real-time monitoring of soft-tissue targets. One challenge, however, is that the ultrasound probe contact pressure can deform the target area and cause discrepancies with the treatment plan. Another challenge is that radiation therapists typically do not have ultrasound experience and therefore cannot easily find the target in the US image. We propose cooperative control strategies to address both challenges. First, we use cooperative control with virtual fixtures (VFs) to enable acquisition of a planning CT that includes the soft-tissue deformation. Then, for the patient setup during the treatment sessions, we propose to use real-time US image feedback to dynamically update the VFs; this co-manipulation strategy provides haptic cues that guide the therapist to correctly place the US probe. A phantom study is performed to demonstrate that the co-manipulation strategy enables inexperienced operators to quickly and accurately place the probe on a phantom to reproduce a desired reference image. This is a necessary step for patient setup and, by reproducing the reference image, creates soft-tissue deformations that are consistent with the treatment plan, thereby enabling real-time monitoring during treatment delivery.

An automatic switching approach to teleoperation of mobile-manipulator systems using virtual fixtures

SUMMARYThis work presents a novel command strategy developed to improve operator performance and minimize difficulties in teleoperation tasks for mobile-manipulator systems with a holonomic base. Aimed specifically at novice operators, virtual fixtures are introduced as a means to minimize collisions and assist in navigation. Using the 6-degree-of-freedom (DOF) Omnibot mobile-manipulator system (MMS), a command strategy is implemented such that the operator need only control a 3-DOF haptic joystick to achieve full control of the Omnibot MMS. The command strategy is used to coordinate control between the arm and the base of the system, prevent collisions with known obstacles, and alert the operator of proximity to those obstacles with haptic forces. Through experimental testing it is shown that operator performance improved with the use of virtual fixtures.

Improving Precision in Navigating Laparoscopic Surgery Instruments toward a Planar Target Using Haptic and Visual Feedback

The poor ergonomics of laparoscopic surgery are a widely recognized source of difficulty for surgeons, leading to sub-optimal surgeon performance and sometimes injury to the patient. The main causes for this are lost and distorted perception of interaction forces and instrument position. The latter, due to losses in visual and kinaesthetic depth perception and modified hand-eye coordination, can prevent precise navigation of instruments towards surgical targets or away from sensitive anatomic structures. This situation prompts us to explore methods for efficiently assisting the surgeon during instrument navigation. Here, we present experiments aimed at providing insights into the effectiveness of haptic (tactile and kinaesthetic), visual and combined feedback in assisting the navigation of a laparoscopic instrument tip towards a surgical target. Subjects placed in front of a laparoscopic trainer were tasked with following various instrument tip trajectories within a target plane while minimizing both deviations and task execution times. Feedback on the level of deviation was provided alternately through visual on-screen cues, tactile cues provided by vibration motors and / or kinaesthetic cues provided by a haptic interface co-manipulating the surgical instrument. Evaluations of these forms of feedback over two series of experiments implicating a total of 35 subjects (34 non-surgeon novices, 1 surgeon intern with experience in laparoscopy) show positive impacts of providing such feedback on precision in instrument navigation. Visual, tactile and combined cues lead to increased precision in navigation (up to 25\% increase in time spent on target, and 32\% reduction in deviation amplitudes), but usually at the cost of reduced task execution speed. However, the use of kinaesthetic feedback through soft virtual fixtures provided in a co-manipulated robot-assisted surgery set-up both significantly improved precision (32\% increase in time spent on target, and 70\% reduction in deviation amplitudes) and task execution speed (30\% reduction in task completion times). The combination of visual and tactile feedback was shown to be helpful in correcting larger deviations from the target. These preliminary results are promising for implementation of low-cost tactile or combined visual and tactile feedback in applications to conventional laparoscopic instrument navigation, as well as to robot-assisted laparoscopic surgery.

Design of Virtual Fixtures for Robotic Cholecystectomy.

With the ongoing developments in robotic surgery, the associated adverse events need to be carefully evaluated. Virtual fixtures (VFs), a safety design feature against unintended motion during robotic surgery, have been proposed, but the methodology for designing VFs remains experimental. In this study, we propose a novel methodology for designing VFs for robotic cholecystectomy. Laparoscopic cholecystectomy (LC) was performed in 24 patients with cholecystitis. Active working space (AWS), the distance between instrument heads (DBIH), motion speed of bilateral hands, and instrument heads were calculated and analyzed. DBIH was 14.78 ± 6.94 cm. Diameter of right and left AWS was 15.81 ± 3.69 cm and 15.33 ± 1.52 cm, respectively. DBIH was found to significantly correlate with the surgeon's experience. Bilateral AWS was found to be significantly associated with body circumference at Murphy's point level. However, no association was observed between bilateral AWS and surgeon's experience. A novel methodology to build VFs for designing VFs for robotic cholecystectomy is established. Surgeon's experience appears to play an important role in determining the DBIH during robotic laparoscopic cholecystectomy, but does not affect bilateral AWS.

Haptic Assembly Using Skeletal Densities and Fourier Transforms1

Haptic-assisted virtual assembly and prototyping has seen significant attention over the past two decades. However, in spite of the appealing prospects, its adoption has been slower than expected. We identify the main roadblocks as the inherent geometric complexities faced when assembling objects of arbitrary shape, and the computation time limitation imposed by the notorious 1 kHz haptic refresh rate. We addressed the first problem in a recent work by introducing a generic energy model for geometric guidance and constraints between features of arbitrary shape. In the present work, we address the second challenge by leveraging Fourier transforms to compute the constraint forces and torques. Our new concept of “geometric energy” field is computed automatically from a cross-correlation of “skeletal densities” in the frequency domain, and serves as a generalization of the manually specified virtual fixtures or heuristically identified mating constraints proposed in the literature. The formulation of the energy field as a convolution enables efficient computation using fast Fourier transforms (FFTs) on the graphics processing unit (GPU). We show that our method is effective for low-clearance assembly of objects of arbitrary geometric and syntactic complexity.

脳深部狭所のロボット手術におけるVirtual Fixtureを用いたフィードバックによる接触回避

Robotic assistance enables surgeons to perform dexterous and precise manipulations; however, robot assisted neurosurgery in deep and narrow spaces in the brain has a risk of unexpected collisions between the shafts of robotic instruments and their surroundings out of the microscopic view. Thus, we propose to provide the surgeon with feedback using a truncated cone shaped virtual fixture generated by marking the internal surface of the workspace. Three types of feedback methods, namely force feedback, visual feedback, and motion scaling feedback, were developed and implemented to the microsurgical robotic system in order to present the risk of collision to the surgeon. Performance of each feedback and their combinations was evaluated, and the experimental results showed that the combinations including visual feedback were effective in collision avoidance.