Forbidden Region Virtual Fixtures Research Articles

Cardiac X-ray image-based haptic guidance for robot-assisted coronary intervention: a feasibility study.

Effective and efficient haptic guidance is desirable for tele-operated robotic surgery because it has a potential to enhance surgeon's skills, especially in coronary interventions where surgeon loses both an eye-hand coordination and a direct sight to the organ. This paper proposes a novel haptic guidance procedure-both kinesthetic and cutaneous, which solely depends upon X-ray images, for tele-robotic system that assists an efficient navigation of the guidewire towards the target location during a coronary intervention. Proposed methodology requires cardiologists to draw virtual fixtures (VFs) on angiograms as a preoperative procedure. During an operation, these VFs direct the guidewire to the desired coronary vessel. For this, the position and orientation of guidewire tip are calculated with respect to VFs' anatomy, using image processing on the real-time 2D fluoroscopic images. The haptic feedbacks are then rendered on to the master device depending on the interaction with attractive and repulsive, guidance and forbidden region VFs. A feasibility study in the laboratory environment is performed by using a webcam as an image acquisition device and a phantom-based coronary vessel model. The subsequent statistical analysis shows that, on an average, a decrease of approx. 37% in task completion time is observed with haptic feedback. Moreover, haptic guidance is found effective for most difficult branch, whereas there is a minimal significance of such haptics for the easiest branch. The proposed haptic guidance procedure may assist cardiologists for an efficient and effective guidewire navigation during a surgical procedure. The cutaneous haptics (vibration feedback) is found more helpful in coronary interventions compared with kinesthetic haptics (force feedback).

Virtual Fixture Generation for Task Planning With Complex Geometries

Abstract Many robotic processes require the system to maintain a tool’s orientation and distance from a surface. To do so, researchers often use virtual fixtures (VFs) to either guide the robot along a path or forbid it from leaving the workspace. Previous efforts relied on volumetric primitives (planes, cylinders, etc.) or raw sensor data to define VFs. However, those approaches only work for a small subset of real-world objects. Extending this approach is complicated not only by VF generation but also by generalizing user traversal of the VF to command a robot trajectory remotely. In this study, we present the concept of task VFs, which convert layers of point cloud-based Guidance VF into a bidirectional graph structure and pair it with a Forbidden Region VF. These VFs are hardware-agnostic and can be generated from virtually any source data, including from parametric objects (superellipsoids, supertoroids, etc.), meshes (including from computer-aided design (CAD)), and real-time sensor data for open-world scenarios. We address surface convexity and concavity since these and distance to the task surface determine the size and resolution of VF layers. This article then presents the manipulator-to-task transform tool for task VF visualization and to limit human–robot interaction ambiguities. Testing confirmed generation success and users performed spatially discrete experiments to evaluate task VF usability complex geometries, which showed their interpretability. The manipulator-to-task transform tool applies many robotic applications, including collision avoidance, process design, training, task definition, etc. for virtually any geometry.

Vision-Based Dynamic Virtual Fixtures for Tools Collision Avoidance in Robotic Surgery

In robot-aided surgery, during the execution of typical bimanual procedures such as dissection, surgical tools can collide and create serious damage to the robot or tissues. The da Vinci robot is one of the most advanced and certainly the most widespread robotic system dedicated to minimally invasive surgery. Although the procedures performed by da Vinci-like surgical robots are teleoperated, potential collisions between surgical tools are a very sensitive issue declared by surgeons. Shared control techniques based on Virtual Fixtures (VF) can be an effective way to help the surgeon prevent tools collision. This letter presents a surgical tools collision avoidance method that uses Forbidden Region Virtual Fixtures. Tool clashing is avoided by rendering a repulsive force to the surgeon. To ensure the correct definition of the VF, a marker-less tool tracking method, using deep neural network architecture for tool segmentation, is adopted. The use of direct kinematics for tools collision avoidance is affected by tools position error introduced by robot component elasticity during tools interaction with the environment. On the other hand, kinematics information can help in case of occlusions of the camera. Therefore, this work proposes an Extended Kalman Filter (EKF) for pose estimation which ensures a more robust application of VF on the tool, coupling vision and kinematics information. The entire pipeline is tested in different tasks using the da Vinci Research Kit system.

A Path/Surface Following Control Approach to Generate Virtual Fixtures

The workspace of a robot can be restricted by virtual fixtures to assist an operator in physical human–robot interaction tasks. This paper introduces a combination of surface following control (SFC) with compliance control and presents a path/SFC approach to systematically generate virtual fixtures. This approach allows implementation of numerous types of constraints like guidance and forbidden region virtual fixtures, hard and soft constraints, as well as static and dynamic virtual fixtures, and their combinations. Additionally, closed-loop stability proofs of the proposed control concepts are given. The flexibility of the presented approach is demonstrated by a series of measurement results from an industrial robot.

Haptic-Based Manipulation Scheme of Magnetic Nanoparticles in a Multi-Branch Blood Vessel for Targeted Drug Delivery.

Magnetic drug targeting is a promising technique that can deliver drugs to the diseased region, while keeping the drug away from healthy parts of body. Introducing a human in the control loop of a targeted drug delivery system and using inherent bilateralism of a haptic device at the same time can considerably improve the performance of targeted drug delivery systems. In this paper, we suggest a novel intelligent haptic guidance scheme for steering a number of magnetic nanoparticles (MNPs) using forbidden region virtual fixtures and a haptic rendering scheme with multi particles. Forbidden region virtual fixtures are a general class of guidance modes implemented in software, which help a human-machine collaborative system accomplish a specific task by constraining a movement into limited regions. To examine the effectiveness of our proposed scheme, we implemented a magnetic guided drug delivery system in a virtual environment using a physics-based model of targeted drug delivery including a multi-branch blood vessel and realistic blood dynamics. We performed user studies with different guidance modes: unguided, semi virtual fixture and full virtual fixture modes. We found out that the efficiency of targeting was significantly improved using the forbidden region virtual fixture and the proposed haptic rendering of MNPs. We can expect that using intelligent haptic feedback in real targeted drug delivery systems can improve the targeting efficiency of MNPs in multi-branch vessels.

Preliminary testing by adults of a haptics-assisted robot platform designed for children with physical impairments to access play

ABSTRACTDevelopment of children’s cognitive and perceptual skills depends heavily on object exploration and experience in their physical world. For children who have severe physical impairments, one of the biggest concerns is the loss of opportunities for meaningful play with objects, including physical contact and manipulation. Assistive robots can enable children to perform object manipulation through the control of simple interfaces. Touch sensations conveyed through haptic interfaces in the form of force reflection or force assistance can help a child to sense the environment and to control a robot. A robotic system with forbidden region virtual fixtures (VFs) was tested in an object sorting task. Three sorting tasks—by color, by shape, and by both color and shape—were performed by 10 adults without disability and one adult with cerebral palsy. Tasks performed with VFs were accomplished faster than tasks performed without VFs, and deviations of the motion area were smaller with VFs than without VFs. For the participant with physical impairments, two out of three tasks were slower with the VFs. This implies that forbidden region VFs are not always able to improve user task performance. Alignment with an individual’s unique motion characteristics can improve VF assistance.

Forbidden region virtual fixtures from streaming point clouds

Forbidden region virtual fixtures protect objects from unwanted contact with a robot. In this paper, we propose a method for creating forbidden region haptic virtual fixtures for teleoperation from streaming point clouds obtained by an RGB-D camera. Upon violating the protected area, the operator receives force feedback that opposes motion inside the forbidden region. Three architectures for creating virtual fixtures are presented and their advantages and disadvantages are described. The proposed methods have the ability to implement constraints and can handle dynamic environments in real-time. The effectiveness of the methods is demonstrated in experiments with a surgical robot.

Safety analysis and control of a robotic spinal surgical system

Screw path drilling is one of the most common and high-risk operations in many kinds of orthopedic surgery, especially in spinal surgeries. During spinal surgery, the bone screws are inserted into the vertebral body from the narrow vertebral pedicles. Any failures in this process will hurt important vessels and nerves of the patient. In this paper two aspects of the safety issues with using the Robotic Spinal Surgery System (RSSS) are analyzed: movement control and real-time operation control. For the safety motion control of the RSSS, two modes are developed: the cooperative control mode for positioning and the fine adjustment mode for precisely adjusting orientation. An automatic calibration algorithm for force/torque sensors is proposed to eliminate gravity effects. Guidance Virtual Fixtures (GVFs) and Forbidden Region Virtual Fixtures (FRVFs) are used to limit the movement of the RSSS. Damping Region Virtual Fixtures (DRVFs) are proposed to prevent the RSSS from crossing the constraint surface and harming the patient’s body. In the path drilling process, a state recognition algorithm is proposed to simulate the feeling in the hand of the surgeon during surgery. Based on force feature extraction and state recognition algorithm, 5 states in the drilling process are recognized, and the control point, which is the stop point of drilling, is found. Experiments are carried out to verify the DRVFs effects in the motion control of RSSS, the state recognition and safety control of the pedicle drilling.

Augmented reality and haptic interfaces for robot‐assisted surgery

Current teleoperated robot-assisted minimally invasive surgical systems do not take full advantage of the potential performance enhancements offered by various forms of haptic feedback to the surgeon. Direct and graphical haptic feedback systems can be integrated with vision and robot control systems in order to provide haptic feedback to improve safety and tissue mechanical property identification. An interoperable interface for teleoperated robot-assisted minimally invasive surgery was developed to provide haptic feedback and augmented visual feedback using three-dimensional (3D) graphical overlays. The software framework consists of control and command software, robot plug-ins, image processing plug-ins and 3D surface reconstructions. The feasibility of the interface was demonstrated in two tasks performed with artificial tissue: palpation to detect hard lumps and surface tracing, using vision-based forbidden-region virtual fixtures to prevent the patient-side manipulator from entering unwanted regions of the workspace. The interoperable interface enables fast development and successful implementation of effective haptic feedback methods in teleoperation.

Assessment of Environmental Effects on Collaborative Haptic Guidance

This paper investigates the effect of environmental factors on user performance in a dual-user haptic guidance system. The system under study allows for interaction between both users, the trainee and the trainer, to collaboratively perform a common task in a shared virtual environment. User studies are carried out to experimentally evaluate the users' performance while following square and circular trajectories with two viewpoints of the environment (top view and front view), while the virtual manipulator tool moves in free motion or against forbidden-region virtual fixtures. The performance is measured and statistically evaluated against task completion time, tracking accuracy, and user energy exchange. The studies revealed that changing the environment geometry from a square to a circle results in reduced task completion time and tracking error. Changing the environment viewpoint from top to front decreases the task completion time in both geometries. Forbidden-region virtual fixtures increase energy exchange by both users and decrease task completion time while compromising the tracking performance in the square-following task. However, when visual feedback is removed in the presence of the fixtures, the square tracking performance improves. The results also indicate a strong relationship between user dominance and tracking error only when the experiment is time-limited.

Stable Forbidden-Region Virtual Fixtures for Bilateral Telemanipulation

There has been recent interest in novel human-machine collaborative control laws, called “virtual fixtures,” which provide operator assistance for telemanipulation tasks. A forbidden-region virtual fixture is a constraint, implemented in software, that seeks to prevent the slave manipulator of a master/slave telemanipulation system from entering into a forbidden region of the workspace. In this paper, we consider the problem of unstable vibrations of the slave and/or master against forbidden-region virtual fixtures for a general class of telemanipulator control architectures, including those with haptic feedback. To the best of the authors’ knowledge, there has been no rigorous study of the stability of forbidden-region virtual fixtures in previous work. The system is evaluated around the master and slave equilibrium position resulting from a constant desired human input force, using a discrete state-space model. We present a method to analytically determine if instability is possible in the system. We thoroughly evaluate this method, experimentally, applying malicious user strategies that attempt to drive the system unstable. Our approach agrees with experimental results and can be used to design and analyze the stability and transient properties of a telemanipulator interacting with virtual fixtures. We show that the user can affect both slave- and master-side virtual fixture stability by modifying his or her impedance characteristics. However, the upper bound on stable slave-side virtual fixture stiffness does not depend on the particular user.