Virtual Fixtures Research Articles

Method for generating real-time interactive virtual fixture for shared teleoperation in unknown environments

A virtual fixture (VF) is a constraint built into software that assists a human operator in moving a remote tool along a preferred path via an augmented guidance force to improve teleoperation performance. However, teleoperation generally applies to unknown or dynamic environments, which are challenging for VF use. Most researchers have assumed that VFs are pre-defined or generated automatically; however, these processes are complicated and unreliable in unknown environments where teleoperation is in high demand. Recently, a few researchers have addressed this issue by introducing a user-interactive method of generating VFs in unknown environments. However, these methods are limited to generating a single type of primitive for a single robot tool. Moreover, the accuracy of the VF generated by these methods depends on the accuracy of the human input. Thus, applications of these methods are limited. To overcome those limitations, this work introduces a novel interactive VF generation method that includes a new method of representing VFs as a composition of components. A feature-based user interface allows the human operator to intuitively specify the VF components. The new VF representation accommodates a variety of robot tools and actions. Using the feature-based interface, the process of VF generation is more intuitive and accurate. In this study, the proposed method is evaluated with human subjects in three teleoperation experiments: peg-in-hole, pipe-sawing, and pipe-welding. The experimental results show that the VFs generated by the proposed approach result in a higher manipulation quality while demonstrating the lowest total workload in all experiments. The peg-in-hole task teleoperation was the safest in terms of failure proportion and exerted force of the robot tool. In the pipe-sawing task, the positioning of the robot tool was the most accurate. In the pipe-welding task, the quality of weld was the best in terms of measured tool-trajectory smoothness and visual weld observation.

Design of a "Cobot Tactile Display" for Accessing Virtual Diagrams by Blind and Visually Impaired Users.

Access to graphical information plays a very significant role in today’s world. Access to this information can be particularly limiting for individuals who are blind or visually impaired (BVIs). In this work, we present the design of a low-cost, mobile tactile display that also provides robotic assistance/guidance using haptic virtual fixtures in a shared control paradigm to aid in tactile diagram exploration. This work is part of a larger project intended to improve the ability of BVI users to explore tactile graphics on refreshable displays (particularly exploration time and cognitive load) through the use of robotic assistance/guidance. The particular focus of this paper is to share information related to the design and development of an affordable and compact device that may serve as a solution towards this overall goal. The proposed system uses a small omni-wheeled robot base to allow for smooth and unlimited movements in the 2D plane. Sufficient position and orientation accuracy is obtained by using a low-cost dead reckoning approach that combines data from an optical mouse sensor and inertial measurement unit. A low-cost force-sensing system and an admittance control model are used to allow shared control between the Cobot and the user, with the addition of guidance/virtual fixtures to aid in diagram exploration. Preliminary semi-structured interviews, with four blind or visually impaired participants who were allowed to use the Cobot, found that the system was easy to use and potentially useful for exploring virtual diagrams tactually.

A novel method to create long capture volumes for video tracking

This study examined a novel method to create a long and narrow calibrated capture volume for tracking objects. The methodology relies on the reflection of parallel distance-measuring lasers. Images of a board, blocking the lasers as it is moved through the field of interest, were assembled into a virtual calibration fixture. The method accommodates large calibration volumes and can be used with multiple cameras, providing a consistent absolute positional reference that is difficult to achieve with large mechanical calibration boards. This study considered a 17.4 m long tracking volume. A 0.9 m long rod was tracked throughout the calibrated volume where its average tracked length was within 0.2% of its measured length. The speed of balls traveling through the calibrated volume were within 0.1% of independent speed sensors. The average residual error of a ball’s tracked trajectory and a polynomial fit was within 1.5 mm. The method shows promise as an efficient means of calibrating large calibration volumes with multiple camera pairs.

Real-time vessel segmentation and reconstruction for virtual fixtures for an active handheld microneurosurgical instrument.

Complications related to vascular damage such as intra-operative bleeding may be avoided during neurosurgical procedures such as petroclival meningioma surgery. To address this and improve the patient's safety, we designed a real-time blood vessel avoidance strategy that enables operation on deformable tissue during petroclival meningioma surgery using Micron, a handheld surgical robotic tool. We integrated real-time intra-operative blood vessel segmentation of brain vasculature using deep learning, with a 3D reconstruction algorithm to obtain the vessel point cloud in real time. We then implemented a virtual-fixture-based strategy that prevented Micron's tooltip from entering a forbidden region around the vessel, thus avoiding damage to it. We achieved a median Dice similarity coefficient of 0.97, 0.86, 0.87 and 0.77 on datasets of phantom blood vessels, petrosal vein, internal carotid artery and superficial vessels, respectively. We conducted trials with deformable clay vessel phantoms, keeping the forbidden region 400 [Formula: see text]m outside and 400 [Formula: see text]m inside the vessel. Micron's tip entered the forbidden region with a median penetration of just 8.84 [Formula: see text]m and 9.63 [Formula: see text]m, compared to 148.74 [Formula: see text]m and 117.17 [Formula: see text]m without our strategy, for the former and latter trials, respectively. Real-time control of Micron was achieved at 33.3 fps. We achieved improvements in real-time segmentation of brain vasculature from intra-operative images and showed that our approach works even on non-stationary vessel phantoms. The results suggest that by enabling precise, real-time control, we are one step closer to using Micron in real neurosurgical procedures.

Cardiac X-ray image-based haptic guidance for robot-assisted coronary intervention: a feasibility study.

Effective and efficient haptic guidance is desirable for tele-operated robotic surgery because it has a potential to enhance surgeon's skills, especially in coronary interventions where surgeon loses both an eye-hand coordination and a direct sight to the organ. This paper proposes a novel haptic guidance procedure-both kinesthetic and cutaneous, which solely depends upon X-ray images, for tele-robotic system that assists an efficient navigation of the guidewire towards the target location during a coronary intervention. Proposed methodology requires cardiologists to draw virtual fixtures (VFs) on angiograms as a preoperative procedure. During an operation, these VFs direct the guidewire to the desired coronary vessel. For this, the position and orientation of guidewire tip are calculated with respect to VFs' anatomy, using image processing on the real-time 2D fluoroscopic images. The haptic feedbacks are then rendered on to the master device depending on the interaction with attractive and repulsive, guidance and forbidden region VFs. A feasibility study in the laboratory environment is performed by using a webcam as an image acquisition device and a phantom-based coronary vessel model. The subsequent statistical analysis shows that, on an average, a decrease of approx. 37% in task completion time is observed with haptic feedback. Moreover, haptic guidance is found effective for most difficult branch, whereas there is a minimal significance of such haptics for the easiest branch. The proposed haptic guidance procedure may assist cardiologists for an efficient and effective guidewire navigation during a surgical procedure. The cutaneous haptics (vibration feedback) is found more helpful in coronary interventions compared with kinesthetic haptics (force feedback).

Impedance controlled human–robot collaborative tooling for edge chamfering and polishing applications

Surface finishing, as the final stage in the manufacturing pipeline, is a key process in determining the quality and life span of a product. Such a task is characterized by low contact forces and minimal material removal from the object surface. Despite the advancements in machine learning and artificial intelligence, human workforce is still irreplaceable in performing such tasks due to superior dexterity and adaptability, but this is often prone to risks such as hand-arm vibration syndrome due to hand-held tools. Therefore, we propose a collaborative approach to assist the human in carrying out such tasks with the help of two case studies: Human–Robot-Collaborative edge chamfering and polishing tasks, based on an impedance controlled collaborative curve tracing technique. We propose a collaborative framework, where the robot assists an operator to guide the end-effector/tool along a pre-defined parametric curve. The algorithm is demonstrated in two scenarios. In the first case, we address a collaborative chamfering task whereas the second case focuses on a polishing application (for straight edges). For these kinds of tasks, the curve to be traced assumes the shape of a straight line along the edge. We make use of the compliant feature of a cobot, which allows the user to physically guide the robot in the task space, to generate a mathematical model for the tool path. From the end-user perspective, this is more intuitive than the classical programming-based path planning approaches. In the process of machining, to enhance the path tracking accuracy and to ensure constant tool-surface contact, we implement guidance virtual fixtures through impedance control. As a result, the machining error is reduced.

Mixed Reality-Enhanced Intuitive Teleoperation with Hybrid Virtual Fixtures for Intelligent Robotic Welding

This paper presents an integrated scheme based on a mixed reality (MR) and haptic feedback approach for intuitive and immersive teleoperation of robotic welding systems. By incorporating MR technology, the user is fully immersed in a virtual operating space augmented by real-time visual feedback from the robot working space. The proposed robotic tele-welding system features imitative motion mapping from the user’s hand movements to the welding robot motions, and it enables the spatial velocity-based control of the robot tool center point (TCP). The proposed mixed reality virtual fixture (MRVF) integration approach implements hybrid haptic constraints to guide the operator’s hand movements following the conical guidance to effectively align the welding torch for welding and constrain the welding operation within a collision-free area. Onsite welding and tele-welding experiments identify the operational differences between professional and unskilled welders and demonstrate the effectiveness of the proposed MRVF tele-welding framework for novice welders. The MRVF-integrated visual/haptic tele-welding scheme reduced the torch alignment times by 56% and 60% compared to the MRnoVF and baseline cases, with minimized cognitive workload and optimal usability. The MRVF scheme effectively stabilized welders’ hand movements and eliminated undesirable collisions while generating smooth welds.

A Unified Approach for Virtual Fixtures and Goal-Driven Variable Admittance Control in Manual Guidance Applications

In applications of human-robot physical interaction, it is important to guarantee high accuracy not only in terms of the position of the end-effector, but also in terms of its orientation. While translational movements are easily understandable for a human, rotational ones do not have an immediate interpretation. This does not allow the operator to achieve an adequate positioning accuracy when a target orientation is set. In this work, a suitable strategy that facilitates the human in accomplishing rotational motion is proposed through the formulation of rotational virtual fixtures. This is obtained thanks to a clever description of the orientation that allows to draw an equivalence between the formulation of translational virtual fixtures and their rotational counterparts. To further improve the human comprehension of the best motion direction in the rotational world, a variable admittance control is also designed. We finally propose to jointly exploit these techniques to achieve the best possible performance in terms of positioning accuracy. The performance are evaluated by means of rotational point to point cooperative motions involving several subjects and an ABB IRB-140 robot.

A Digital Twin Approach for Contextual Assistance for Surgeons During Surgical Robotics Training.

Minimally invasive robotic surgery copes with some disadvantages for the surgeon of minimally invasive surgery while preserving the advantages for the patient. Most commercially available robotic systems are telemanipulated with haptic input devices. The exploitation of the haptics channel, e.g., by means of Virtual Fixtures, would allow for an individualized enhancement of surgical performance with contextual assistance. However, it remains an open field of research as it is non-trivial to estimate the task context itself during a surgery. In contrast, surgical training allows to abstract away from a real operation and thus makes it possible to model the task accurately. The presented approach exploits this fact to parameterize Virtual Fixtures during surgical training, proposing a Shared Control Parametrization Engine that retrieves procedural context information from a Digital Twin. This approach accelerates a proficient use of the robotic system for novice surgeons by augmenting the surgeon’s performance through haptic assistance. With this our aim is to reduce the required skill level and cognitive load of a surgeon performing minimally invasive robotic surgery. A pilot study is performed on the DLR MiroSurge system to evaluate the presented approach. The participants are tasked with two benchmark scenarios of surgical training. The execution of the benchmark scenarios requires basic skills as pick, place and path following. The evaluation of the pilot study shows the promising trend that novel users profit from the haptic augmentation during training of certain tasks.

Volumetric Accuracy Analysis of Virtual Safety Barriers for Cooperative-Control Robotic Mastoidectomy.

Virtual fixtures can be enforced in cooperative-control robotic mastoidectomies with submillimeter accuracy. Otologic procedures are well-suited for robotic assistance due to consistent osseous landmarks. We have previously demonstrated the feasibility of cooperative-control robots (CCRs) for mastoidectomy. CCRs manipulate instruments simultaneously with the surgeon, allowing the surgeon to control instruments with robotic augmentation of motion. CCRs can also enforce virtual fixtures, which are safety barriers that prevent motion into undesired locations. Previous studies have validated the ability of CCRs to allow a novice surgeon to safely complete a cortical mastoidectomy. This study provides objective accuracy data for CCR-imposed safety barriers in cortical mastoidectomies. Temporal bone phantoms were registered to a CCR using preoperative computed tomography (CT) imaging. Virtual fixtures were created using 3D Slicer, with 2D planes placed along the external auditory canal, tegmen, and sigmoid, converging on the antrum. Five mastoidectomies were performed by a novice surgeon, moving the drill to the limit of the barriers. Postoperative CT scans were obtained, and Dice coefficients and Hausdorff distances were calculated. The average modified Hausdorff distance between drilled bone and the preplanned volume was 0.351 ± 0.093 mm. Compared with the preplanned volume of 0.947 cm3, the mean volume of bone removed was 1.045 cm3 (difference of 0.0982 cm3 or 10.36%), with an average Dice coefficient of 0.741 (range, 0.665-0.802). CCR virtual fixtures can be enforced with a high degree of accuracy. Future studies will focus on improving accuracy and developing 3D fixtures around relevant surgical anatomy.

Monocular Vision-Based Retinal Membrane Peeling With a Handheld Robot.

Retinal membrane peeling requires delicate manipulation. The presence of the surgeon's physiological tremor, the high variability and often low quality of the ophthalmic image, and excessive forces make the tasks more challenging. Preventing unintended movement caused by tremor and unintentional forces can reduce membrane injury. With the use of an actively stabilized handheld robot, we employ a monocular camera-based surface reconstruction method to estimate the retinal plane and we propose the use of a virtual fixture with the application of a hard stop and motion scaling to improve control of the tool tip during delaminating in a laboratory simulation of retinal membrane peeling. A hard stop helps to limit downward force exerted on the surface. Motion scaling also improves the user's control of contact force when delaminating. We demonstrate a reduction of maximum force and maximum surface-penetration distance from the estimated retinal plane using the proposed technique.

Virtual Fixture Generation for Task Planning With Complex Geometries

Abstract Many robotic processes require the system to maintain a tool’s orientation and distance from a surface. To do so, researchers often use virtual fixtures (VFs) to either guide the robot along a path or forbid it from leaving the workspace. Previous efforts relied on volumetric primitives (planes, cylinders, etc.) or raw sensor data to define VFs. However, those approaches only work for a small subset of real-world objects. Extending this approach is complicated not only by VF generation but also by generalizing user traversal of the VF to command a robot trajectory remotely. In this study, we present the concept of task VFs, which convert layers of point cloud-based Guidance VF into a bidirectional graph structure and pair it with a Forbidden Region VF. These VFs are hardware-agnostic and can be generated from virtually any source data, including from parametric objects (superellipsoids, supertoroids, etc.), meshes (including from computer-aided design (CAD)), and real-time sensor data for open-world scenarios. We address surface convexity and concavity since these and distance to the task surface determine the size and resolution of VF layers. This article then presents the manipulator-to-task transform tool for task VF visualization and to limit human–robot interaction ambiguities. Testing confirmed generation success and users performed spatially discrete experiments to evaluate task VF usability complex geometries, which showed their interpretability. The manipulator-to-task transform tool applies many robotic applications, including collision avoidance, process design, training, task definition, etc. for virtually any geometry.

Virtual-Fixture Based Drilling Control for Robot-Assisted Craniotomy: Learning From Demonstration

One of the promising solutions for drilling craniotomy is robot-assisted surgery with human guidance. The present study deals with a piecewise collaborative drilling task assisted by a robot while containing aligning and drilling. It can enable surgeons to complete the operation more efficiently and accurately. The switched virtual fixture (VF) between drilling and aligning can be addressed using intention recognition, which learns from demonstrating human-guided force during the collaborative drilling. The training of the switching condition is derived in terms of Gaussian mixture models (GMMs) and the intention recognition is achieved using the Kullback-Leibler (KL) divergence between the GMMs and human-guided real-time sampled forces. To evaluate the performance of aligning and drilling, two experiments are conducted corresponding to the steps of drilling tasks. The compliance, accuracy, and costing time are demonstrated in the experiments. The results indicate that the proposed method has better performance (0.78 ± 0.50 mm in collaborative drilling tasks for positioning accuracy and 38.65 ± 5.00 s for time-consuming) than the conventional method(2.96 ± 1.90 mm for positioning accuracy and 55.41 ± 13.70 s for time-consuming).

Collaborative Robotic Assistant Platform for Endonasal Surgery: Preliminary In-Vitro Trials.

Endonasal surgery is a minimally invasive approach for the removal of pituitary tumors (sarcomas). In this type of procedure, the surgeon has to complete the surgical maneuvers for sarcoma resection with extreme precision, as there are many vital structures in this area. Therefore, the use of robots for this type of intervention could increase the success of the intervention by providing accurate movements. Research has focused on the development of teleoperated robots to handle a surgical instrument, including the use of virtual fixtures to delimit the working area. This paper aims to go a step further with a platform that includes a teleoperated robot and an autonomous robot dedicated to secondary tasks. In this way, the aim is to reduce the surgeon’s workload so that he can concentrate on his main task. Thus, the article focuses on the description and implementation of a navigator that coordinates both robots via a force/position control. Finally, both the navigation and control scheme were validated by in-vitro tests.

A Task Space Virtual Fixture Architecture for Teleoperated Surgical System With Slave Joint Limit Constraints

Unintended motion is one of the major causes of intraoperative injuries in teleoperated surgeries. Due to the large workspace discrepancy between master and slave manipulators, the tip of the slave may deviate from the intended master command without prior notice, when the slave is moved beyond its joint limits. Conventional solutions such as the constrained-optimization-based virtual fixture (VF) method suffer from nonintuitive tip motion management and unnatural haptics in high-dimensional systems. To this end, we propose a task space virtual fixture (TSVF) architecture to systematically address those issues by forbidden-region VF design. It decomposes the high-dimensional task space into low-dimensional task subspaces according to its inherent topology. In each subspace, we design a human-centric TSVF geometry and controller to manage the tip behavior by exploiting the nonlinear kinematics mapping. This architecture builds a real-time TSVF system with natural and predictable haptics. To showcase this concept, we design and implement the proposed TSVF system for the state-of-the-art surgical system, da Vinci Research Kit. Simulations, experiments, and human-factor user study verify its effectiveness and intuitiveness. In the user study, our proposed TSVF system demonstrates the most easy-to-understand tip behavior and shows a significant positive effect over haptics likeability.

Promoting mutual adaptation in haptic negotiation using adaptive virtual fixture

PurposeTo promote the intuitiveness of collaborative tasks, the negotiation ability of humans with each other has inspired a large amount of studies aimed at reproducing the capacity in physical human-robot interaction (pHRI). This paper aims to promote mutual adaptation in negotiation when both parties possess incomplete information.Design/methodology/approachThis paper introduces virtual fixtures into the traditional negotiation mechanism, locally regulating tracking trajectory and impedance parameters in the negotiating phase until the final plan integrates bilateral intentions well. In the strategy, robots convey its task information to humans and offer groups of guide plans for them to choose, on the premise of maximizing the robot’s own profits.FindingsCompared with traditional negotiation strategies, humans adapt to robots easily and show lower cognitive load in the method, while the satisfied plan shows better performance for the whole human-robot system.Originality/valueIn this study, this paper proposes a novel negotiation strategy to facilitate the mutual adaptation of humans and robots in complicated shared tasks, especially when both parties possess incomplete information of tasks.

Telelocomotion—Remotely Operated Legged Robots

Teleoperated systems enable human control of robotic proxies and are particularly amenable to inaccessible environments unsuitable for autonomy. Examples include emergency response, underwater manipulation, and robot assisted minimally invasive surgery. However, teleoperation architectures have been predominantly employed in manipulation tasks, and are thus only useful when the robot is within reach of the task. This work introduces the idea of extending teleoperation to enable online human remote control of legged robots, or telelocomotion, to traverse challenging terrain. Traversing unpredictable terrain remains a challenge for autonomous legged locomotion, as demonstrated by robots commonly falling in high-profile robotics contests. Telelocomotion can reduce the risk of mission failure by leveraging the high-level understanding of human operators to command in real-time the gaits of legged robots. In this work, a haptic telelocomotion interface was developed. Two within-user studies validate the proof-of-concept interface: (i) The first compared basic interfaces with the haptic interface for control of a simulated hexapedal robot in various levels of traversal complexity; (ii) the second presents a physical implementation and investigated the efficacy of the proposed haptic virtual fixtures. Results are promising to the use of haptic feedback for telelocomotion for complex traversal tasks.

Evaluation of Hybrid Control and Palpation Assistance for Situational Awareness in Telemanipulated Task Execution

The use of intelligent feedback modalities to control and react to interaction forces during surgical procedures is an important factor in enabling safe and precise surgery. We explore the use of a model-mediated telemanipulation framework to enhance a user’s situational awareness using assistive virtual fixtures and semi-automated task execution for safe and intuitive environment interaction during robotic laparoscopic surgery. The framework allows stiffness mapping with semi-autonomous excitation, hybrid position-force control, and model updates during soft geometry contact. A 24-person study was carried out at 3 sites in simulated ablation and palpation of phantom anatomy. Compared to methods lacking intelligent feedback and guidance, the proposed framework improved task execution metrics (force regulation, completion time, path-following error) and reduced user effort.

Automatic Surgical Field of View Control in Robot-Assisted Nasal Surgery

In endoscopic nasal surgery, robots, rather than surgical assistants, can be introduced to hold endoscopes and act as the surgeon's third hand, which helps to reduce their operation burden. To address the problem of robot-assisted surgical field of view (FOV) acquisition in endoscopic nasal surgery, this letter proposes an automatic surgical FOV control method based on tracking motion for the entire surgical process to achieve efficient and safe surgical FOV acquisition. First, through the analysis of the spatial pose relationship between the nasal endoscope and the surgical instrument, the desired endoscope pose corresponding to the optimal surgical FOV during surgery is obtained. Then, an automatic surgical FOV control method based on tracking motion is proposed. For automatic FOV acquisition outside the nasal cavity, an unconstrained tracking motion control strategy is proposed. For automatic FOV acquisition inside the nasal cavity, a remote center of motion (RCM) constrained control strategy is proposed. Combined with the entire surgical process, the overall control framework of the surgical FOV during nasal surgery is established. After that, a virtual fixtures (VFs) model based on the CT data of the nasal cavity and a safety contact force threshold control method are proposed to improve the safety of robot motion. Finally, the robot-assisted maxillary sinus surgery experiment is carried out. The results show that the proposed automatic FOV control method can realize clear and safe automatic FOV acquisition during the entire surgical process so that the surgeon can realize the “one surgeon three hands” operation, which reduces the surgeon's operation burden.

Admittance‐Controlled Robotic Assistant for Fibula Osteotomies in Mandible Reconstruction Surgery

Herein, a semiautonomous robot control system for mandible reconstruction surgery is proposed. To reconstruct a segmental defect of the mandible caused by cancerous tissue, a piece of matched fibula bone is often segmented and used to replace the removed mandible section. Herein, to provide guidance to the surgeon during fibula segmentation according to the reconstruction surgical plan and improve the fibula bone cutting accuracy, an admittance‐controlled robotic assistant incorporating 3D augmented reality (AR) visualization and haptic virtual fixtures (VFs) is proposed. The admittance controller is used to reduce the surgeon's hand tremor. VF and AR are used to provide haptic and visual guidance to the surgeon, respectively. A feasibility study is conducted through a comparison of fibula osteotomies when performed with image‐guided surgery, AR‐guided surgery, VF‐guided robot‐assisted surgery, and AR‐ and VF‐guided robot‐assisted surgery. Experimental results show the effectiveness of the proposed admittance‐controlled robotic assistant with AR and VF compared with the other three methods. The proposed method is found to be able to increase precision of the osteotomized segments with a lower average linear variation of 1.04 ± 0.79 mm and a lower average angular variation of 1.83 ± 1.85° compared with the virtual preoperative plan.