Robot-assisted Minimally Invasive Surgery Research Articles

Vision-based hand-eye calibration for robot-assisted minimally invasive surgery.

The knowledge of laparoscope vision can greatly improve the surgical operation room (OR) efficiency. For the vision-based computer-assisted surgery, the hand-eye calibration establishes the coordinate relationship between laparoscope and robot slave arm. While significant advances have been made for hand-eye calibration in recent years, efficient algorithm for minimally invasive surgical robot is still a major challenge. Removing the external calibration object in abdominal environment to estimate the hand-eye transformation is still a critical problem. We propose a novel hand-eye calibration algorithm to tackle the problem which relies purely on surgical instrument already in the operating scenario for robot-assisted minimally invasive surgery (RMIS). Our model is formed by the geometry information of the surgical instrument and the remote center-of-motion (RCM) constraint. We also enhance the algorithm with stereo laparoscope model. Promising validation of synthetic simulation and experimental surgical robot system have been conducted to evaluate the proposed method. We report results that the proposed method can exhibit the hand-eye calibration without calibration object. Vision-based hand-eye calibration is developed. We demonstrate the feasibility to perform hand-eye calibration by taking advantage of the components of surgical robot system, leading to the efficiency of surgical OR.

Master-slave control and evaluation of force sensing for robot-assisted minimally invasive surgery

PurposeExisting robot-assisted minimally invasive surgery (RMIS) system lacks of force feedback, and it cannot provide the surgeon with interaction forces between the surgical instruments and patient’s tissues. This paper aims to restore force sensation for the RMIS system and evaluate effect of force sensing in a master-slave manner.Design/methodology/approachThis paper presents a four-DOF surgical instrument with modular joints and six-axis force sensing capability and proposes an incremental position mode master–slave control strategy based on separated position and orientation to reflect motion of the end of master manipulator to the end of surgical instrument. Ex-vivo experiments including tissue palpation and blunt dissection are conducted to verify the effect of force sensing for the surgical instrument. An experiment of trajectory tracking is carried out to test precision of the control strategy.FindingsResults of trajectory tracking experiment show that this control strategy can precisely reflect the hand motion of the operator, and the results of the ex-vivo experiments including tissue palpation and blunt dissection illustrate that this surgical instrument can measure the six-axis interaction forces successfully for the RMIS.Originality/valueThis paper addresses the important role of force sensing and force feedback in RMIS, clarifies the feasibility to apply this instrument prototype in RMIS for force sensing and provides technical support of force feedback for further clinical application.

Vision-Based Framework of Single Master Dual Slave Semi-Autonomous Surgical Robot System

BackgroundIn the traditional Robot assisted minimally invasive surgery (RMIS) scenario, the instruments are fully controlled by the surgeon through tele-operation. Recent works have widely explored the surgical intelligence by integrating advanced approaches to enhance the surgical operating room (OR) outcomes. MethodsWe propose a novel single-master dual-slave framework for semi-autonomous suturing task, laparoscope information is introduced to feed back into the robotic control loop to guide the movement of surgical instrument. ResultsExperimental results demonstrate that the proposed method can handle the single-master dual-slave semi-autonomous suturing subtask. Furthermore, the framework exhibits compelling performance leading to the efficiency of surgical OR. ConclusionsAdding vision information into the robotic control loop can achieve the semi-autonomous operation, improve the surgical OR efficiency, this capability yields new level of intelligence for the RMIS.

Palpation-Based Multi-Tumor Detection Method Considering Moving Distance for Robot-assisted Minimally Invasive Surgery.

A novel palpation-based tumor detection method for robot-assisted minimally invasive surgery (RMIS) is proposed in this paper. A tactile stiffness sensor based on piezoelectric vibration is designed to detect tissue stiffness just by gentle contacting, without the risk of injuring the organ. A novel multi-tumor detection method is proposed to solve the problem that existing tumor detection methods can only detect a single tumor. In addition, the moving distance of the sensor during palpation is introduced into the sampling strategy function as a new factor, so that the total moving distance of sensor can be considered and optimized during the tumor palpation process. Simulation studies are carried out to compare the new method with existing methods in tumor detection. It shows that the proposed method can identify and locate multiple tumors successfully with the highest performance (F1 score > 0.99), and the total moving distance during palpation is reduced by 43% without any compromise in the detection accuracy.

Convolutional neural network-based surgical instrument detection.

BACKGROUND: Minimally invasive surgery (MIS), unlike open surgery in which surgeons can perform surgery directly, is performed using miniaturized instruments with indirect but careful observation of the surgical site.OBJECTIVE: Instrument detection is a crucial requirement in conventional and robot-assisted MIS, which can also be very useful during surgical training. In this paper, we propose a novel framework of using two three-layer convolutional neural networks (CNNs) in a series to detect surgical instrument in in-vivo video frames. METHODS: The two convolutional neural networks proposed in this paper have different tasks. (i) The former CNN is trained to detect the edges points of the instrument shaft directly from images patches. (ii) The latter is trained to locate the instrument tip also from images patches after the former detection finishes.RESULTS: We validated our method on the publicly available EndoVisSub dataset and a standard dataset, and it detected tools with an accuracy of 91.2% and 75% respectively.CONCLUSION: Our two-step detection method achieves better performance than other existing approaches in terms of detection accuracy.

An Evaluation of Inanimate and Virtual Reality Training for Psychomotor Skill Development in Robot-Assisted Minimally Invasive Surgery

Robot-assisted minimally invasive surgery (RAMIS) is gaining widespread adoption in many surgical specialties, despite the lack of a standardized training curriculum. Current training approaches rely heavily on virtual reality simulators, in particular for basic psychomotor and visuomotor skill development. It is not clear, however, whether training in virtual reality is equivalent to inanimate model training. In this manuscript, we seek to compare virtual reality training to inanimate model training, with regard to skill learning and skill transfer. Using a custom-developed needle-driving training task with inanimate and virtual analogs, we investigated the extent to which N=18 participants improved their skill on a given platform post-training, and transferred that skill to the opposite platform. Results indicate that the two approaches are not equivalent, with more salient skill transfer after inanimate training than virtual training. These findings support the claim that training with real physical models is the gold standard, and suggest more inanimate model training be incorporated into training curricula for early psychomotor skill development.

An Ergonomic Shared Workspace Analysis Framework for the Optimal Placement of a Compact Master Control Console

Master-Slave control is commonly used for Robot-Assisted Minimally Invasive Surgery (RAMIS). The configuration, as well as the placement of the master manipulators, can influence the remote control performance. An ergonomic shared workspace analysis framework is proposed in this letter. Combined with the workspace of the master manipulators and the human arms, the human-robot interaction workspace can be generated. The optimal master robot placement can be determined based on three criteria: 1) interaction workspace volume, 2) interaction workspace quality, and 3) intuitiveness for slave robot control. Experimental verification of the platform is conducted on a da Vinci Research Kit (dVRK). An in-house compact master manipulator (Hamlyn CRM) is used as the master robot and the da Vinci robot is used as the slave robot. Comparisons are made between with and without using design optimization to validate the effectiveness of the ergonomic shared workspace analysis technique. Results indicate that the proposed ergonomic shared workspace analysis can improve the performance of teleoperation in terms of task completion time and the number of clutching required during operation.

A Contact Force Sensor Based on S-Shaped Beams and Optoelectronic Sensors for Flexible Manipulators for Minimally Invasive Surgery (MIS)

Flexible, highly articulated robotic tools can greatly facilitate procedures in which the operator needs to access small openings and confined spaces. Particularly, in the context of robotic-assisted minimally invasive surgery (RMIS), the application of such manipulation tools can be significantly beneficial in preventing unnecessary interactions with sensitive body organs by which reducing patient’s recovery time when compared with conventional methods. However, these systems usually lack tactile feedback and are not able to perceive and quantify the interactions between themselves and soft body organs. This deficiency may result in damaging the organs due to unwanted excessive force applied. To this end, we introduce a contact force sensor based on three ’dyadic-S-shaped’ beams and three optoelectronic sensors. The modular design of a flexible manipulation system described as part of this paper allows ready integration of a series of the proposed sensors within its structure. The sensor uses our novel sensing principle for measuring contact forces. The strategic employment of custom sensor structure and the optoelectronic components fulfill our design objectives which have been focused on the creation of a modular, low-cost, low-noise (electrically) with large voltage variation, without the need for an amplifier, through a simple fabrication process for MIS. Our experimental results, following a very simple calibration processes show the average errors of ${F}_{x}$ (+19.37%±0.82, −18.32%±2.06) and ${F}_{y}$ (+18.56%±1.69, −17.00%±1.32), and the average RMS errors of ${F}_{x}$ (0.12N±0.0067) and ${F}_{y}$ (0.11N±0.0032) in the measurement of force values within the range of −4 to 4 N.

A Novel Grip Force Cognition Scheme for Robot-Assisted Minimally Invasive Surgery

Obtaining a grip force in robot-assisted minimally invasive surgery (RMIS) has always been a crucial research topic. A novel grip force cognition scheme is proposed in this article. We utilize the dynamic analysis of the cable-driven system to conduct feature engineering and add prior knowledge to the Gaussian process regression (GPR). Specifically, the feature and GPR model candidates are acquired by analyzing the dynamic characteristics first. After that, a wrapper searching method combined with exhaustive search (ES) and sequential backward selection (SBS) is proposed to determine the features and GPR model. The hyperparameters are optimized by differential evolution (DE) by minimizing the negative log marginal likelihood (NLML). Besides, the training set is extended by including several different objects to avoid the algorithm “learning” the object property. Our method is verified on a cable-driven platform to ensure the errors are all derived from the algorithm rather than the cable tension loss. Our method reduces the errors on our testing set by about 64% and outperforms several popular model-based and learning-based methods. This article illustrates the significance to integrate the system's characteristics when introducing machine learning techniques into robot systems.

Surgeon-Centered Analysis of Robot-Assisted Needle Driving Under Different Force Feedback Conditions.

Robotic assisted minimally invasive surgery (RAMIS) systems present many advantages to the surgeon and patient over open and standard laparoscopic surgery. However, haptic feedback, which is crucial for the success of many surgical procedures, is still an open challenge in RAMIS. Understanding the way that haptic feedback affects performance and learning can be useful in the development of haptic feedback algorithms and teleoperation control systems. In this study, we examined the performance and learning of inexperienced participants under different haptic feedback conditions in a task of surgical needle driving via a soft homogeneous deformable object—an artificial tissue. We designed an experimental setup to characterize their movement trajectories and the forces that they applied on the artificial tissue. Participants first performed the task in an open condition, with a standard surgical needle holder, followed by teleoperation in one of three feedback conditions: (1) no haptic feedback, (2) haptic feedback based on position exchange, and (3) haptic feedback based on direct recording from a force sensor, and then again with the open needle holder. To quantify the effect of different force feedback conditions on the quality of needle driving, we developed novel metrics that assess the kinematics of needle driving and the tissue interaction forces, and we combined our novel metrics with classical metrics. We analyzed the final teleoperated performance in each condition, the improvement during teleoperation, and the aftereffect of teleoperation on the performance when using the open needle driver. We found that there is no significant difference in the final performance and in the aftereffect between the 3 conditions. Only the two conditions with force feedback presented statistically significant improvement during teleoperation in several of the metrics, but when we compared directly between the improvements in the three different feedback conditions none of the effects reached statistical significance. We discuss possible explanations for the relative similarity in performance. We conclude that we developed several new metrics for the quality of surgical needle driving, but even with these detailed metrics, the advantage of state of the art force feedback methods to tasks that require interaction with homogeneous soft tissue is questionable.

Hamlyn CRM: a compact master manipulator for surgical robot remote control

PurposeCompact master manipulators have inherent advantages, since they can have practical deployment within the general surgical environments easily and bring benefits to surgical training. To assess the advantages of compact master manipulators for surgical skills training and the performance of general robot-assisted surgical tasks, Hamlyn Compact Robotic Master (Hamlyn CRM) is built up and evaluated in this paper.MethodsA compact structure for the master manipulator is proposed. A novel sensing system is designed while stable real-time motion tracking can be realized by fusing the information from multiple sensors. User studies were conducted based on a ring transfer task and a needle passing task to explore a suitable mapping strategy for the compact master manipulator to control a surgical robot remotely. The overall usability of the Hamlyn CRM is verified based on the da Vinci Research Kit (dVRK). The master manipulators of the dVRK control console are used as the referenceResultsMotion tracking experiments verified that the proposed system can track the operators’ hand motion precisely. As for the master–slave mapping strategy, user studies proved that the combination of the position relative mapping mode and the orientation absolute mapping mode is suitable for Robot-Assisted Minimally Invasive Surgery (RAMIS), while key parameters for mapping are selected.ConclusionResults indicated that the Hamlyn CRM can serve as a compact master manipulator for surgical training and has potential applications for RAMIS.

Tissue discrimination by bioelectrical impedance during PLL resection in anterior decompression surgery for treatment of cervical spondylotic myelopathy

BackgroundThe electrical properties of biological tissues differ depending on their physical properties. This study aimed to explore if bioelectrical impedance (modulus and phase) would discriminate tissues relevant to resection of the posterior longitudinal ligament (PLL) in anterior cervical decompression surgery.MethodsPLL resection via an anterior approach was performed on the C4/5 segments in six mini-pigs. The bioelectrical impedance measurements were performed for two tissue groups (annulus fibrosus, endplate cartilage, sub-endplate cortical bone, and PLL; PLL, dura mater, spinal cord, and nerve root) using a novel probe and a precision inductance-capacitance-resistance meter. For each group, impedance was analyzed in terms of modulus and phase along a broad spectrum of frequencies (200–3000 kHz) using a nonparametric statistical analysis (Kruskal-Wallis).ResultsThe analysis showed a clear difference among the tissues. The modulus and phase show the same changing trend with frequency and present lower values at higher frequencies. Among annulus fibrosus, endplate cartilage, sub-endplate cortical bone, and PLL, it was possible to discriminate each tissue at every frequency point, considering the phase (p < 0.05), while this was not always the case (i.e., annulus fibrosus vs PLL at frequency of 200 kHz, 400 kHz, and 3000 kHz, p > 0.05) for modulus. Among PLL, dura mater, spinal cord, and nerve root, for every comparison, a statistically significant difference was reported in the modulus, phase, or both (p < 0.05).ConclusionsThe results indicated the potential of bioelectrical impedance to provide real-time tissue differentiation and enhance safe PLL resection in anterior cervical decompression surgery, particularly in robot-assisted minimally invasive surgery (RMIS).

Robot-Assisted Minimally Invasive Surgical Skill Assessment—Manual and Automated Platforms

The practice of Robot-Assisted Minimally Invasive Surgery (RAMIS) requires extensive skills from the human surgeons due to the special input device control, such as moving the surgical instruments, use of buttons, knobs, foot pedals and so. The global popularity of RAMIS created the need to objectively assess surgical skills, not just for quality assurance reasons, but for training feedback as well. Nowadays, there is still no routine surgical skill assessment happening during RAMIS training and education in the clinical practice. In this paper, a review of the manual and automated RAMIS skill assessment techniques is provided, focusing on their general applicability, robustness and clinical relevance.

A Comparative Human-Centric Analysis of Virtual Reality and Dry Lab Training Tasks on the da Vinci Surgical Platform

There is a growing, widespread trend of adopting robot-assisted minimally invasive surgery (RMIS) in clinical care. Dry lab robot training and virtual reality simulation are commonly used to train surgical residents; however, it is unclear whether both types of training are equivalent or can be interchangeable and still achieve the same results in terms of training outcomes. In this paper, we take the first step in comparing the effects of physical and simulated surgical training tasks on human operator kinematics and physiological response to provide a richer understanding of exactly how the user interacts with the actual or simulated surgical robot. Four subjects, with expertise levels ranging from novice to expert surgeon, were recruited to perform three surgical tasks — Continuous Suture, Pick and Place, Tubes, with three repetitions — on two training platforms: (1) the da Vinci Si Skills Simulator and (2) da Vinci S robot, in a randomized order. We collected physiological response and kinematic movement data through body-worn sensors for a total of 72 individual experimental trials. A range of expertise was chosen for this experiment to wash out inherent differences based on expertise and only focus on inherent differences between the virtual reality and dry lab platforms. Our results show significant differences ([Formula: see text]-[Formula: see text]) between tasks done on the simulator and surgical robot. Specifically, robotic tasks resulted in significantly higher muscle activation and path length, and significantly lower economy of volume. The individual tasks also had significant differences in various kinematic and physiological metrics, leading to significant interaction effects between the task type and training platform. These results indicate that the presence of the robotic system may make surgical training tasks more difficult for the human operator. Thus, the potentially detrimental effects of virtual reality training alone are an important topic for future investigation.

The Repulsive Force Spectrum of Magnetorheological Fluids Based Tactile Devices Applicable to Robot Surgery

Objective: This paper presents controllable force ranges of tactile devices made of smart Magnetorheological Fluids (MRF) and porous sponges (MR sponges in short). Methods: In order to identify the wide controllable range of the field-dependent repulsive force, three MR sponge samples with three different MR fluids are fabricated using polyurethane foam and cling film. Then, the repulsive forces of the samples are measured using the motor-driven experimental apparatus and the results are presented with minimum and maximum values of the repulsive forces. On the other hand, in order to investigate the feasibility of the proposed tactile device for application to Robot-assisted Minimally Invasive Surgery (RMIS), pig’s organs such as liver, lung and heart, whose viscoelastic properties are very similar to those of human tissues, are tested under same conditions. Results: It is shown that the range of the repulsive spectrum of the pig’s organs can be achieved using the proposed samples by controlling the magnetic field intensity to be applied to MRF domain.

Compensating Uncertainties in Force Sensing for Robotic-Assisted Palpation

Force sensing in robotic-assisted minimally invasive surgery (RMIS) is crucial for performing dedicated surgical procedures, such as bilateral teleoperation and palpation. Due to the bio-compatibility and sterilization requirements, a specially designed surgical tool/shaft is normally attached to the sensor while contacting the organ targets. Through this design, the measured force from the sensor usually contains uncertainties, such as noise, inertial force etc., and thus cannot reflect the actual interaction force with the tissue environment. Motivated to provide the authentic contact force between a robotic tool and soft tissue, we proposed a data-driven force compensation scheme without intricate modeling to reduce the effects of force measurement uncertainties. In this paper, a neural-network-based approach is utilized to automatically model the inertial force subject to noise during the robotic palpation procedure, then the exact contact force can be obtained through the force compensation method which cancels the noise and inertial force. Following this approach, the genuine interaction force during the palpation task can be achieved furthermore to improve the appraisal of the tumor surrounded by the soft tissue. Experiments are conducted with robotic-assisted palpation tasks on a silicone-based soft tissue phantom and the results verify the effectiveness of the suggested method.

Sensorized Surgical Forceps for Robotic-Assisted Minimally Invasive Surgery

A novel force-sensing mechanism for surgical robotic systems is required for further innovation of robotic-assisted minimally invasive surgery (RMIS). In this paper, novel sensorized surgical forceps are presented with five-degree-of-freedom (5-DOF) force/torque (F/T) sensing capability: a grasping force, a 3-DOF manipulating force, and a rotational torque acting at the forceps. Two compact 3-DOF force sensors are integrated into the proximal region on two jaws of the forceps to measure 5-DOF F/T. The forceps also present important advantages for actual use in RMIS, such as miniaturization, compatible configuration to various shapes of surgical forceps, and packaged sensor electronics. In this study, the forceps are designed and manufactured at a low cost and are even disposable. To evaluate the forceps in a surgical robotic environment, the instrument is installed in a surgical robot, named S-Surge. In the environment, the 5-DOF F/T sensing capability is evaluated through comparison with reference sensors.

Design of a novel 6-DOF haptic master mechanism using MR clutches and gravity compensator

This work presents design and dynamic analysis results of a novel 6-degrees-of-freedom (6-DOF) haptic master mechanism featured by magneto-rheological (MR) devices and gravity compensator. The design target of the haptic master mechanism is determined on the basis of the possible application to the robotic-assisted minimally invasive surgery (RMIS) system in which a certain magnitude of the force or moment is required to feel by the surgeon. The dynamic torque/force models associated with MR clutches and brakes are derived considering the geometry and Bingham characteristics of MR Fluid. In the mechanism geometry, the wrist part is designed with a symmetrical structure such that the center of gravity is at the center of the handle. This makes the dynamic model of the proposed mechanism to be decoupled between the wrist motion and the translation motion. In addition, a gravity compensator is designed to enhance tracking performances of the desired repulsive forces/torques. It is demonstrated that the proposed 6-DOF MR haptic master can sufficiently generate desired rotational and translational forces/torques required in RMIS system.

Human-Centered Transparency of Grasping via a Robot-Assisted Minimally Invasive Surgery System

In this paper, we investigate the grasping of rigid objects in a unilateral robot-assisted minimally invasive surgery (RAMIS). We define a human-centered transparency that quantifies the natural action and perception in RAMIS. We demonstrate this human-centered transparency analysis for different values of gripper scaling—the scaling between the grasp aperture of the surgeon-side manipulator and the aperture of the surgical instrument grasper. A total of 31 participants performed teleoperated grasping and perceptual assessment of rigid objects in one of three gripper scaling conditions ( fine , normal , and quick , trading off precision and responsiveness). A psychophysical analysis of the variability of the maximal grasping aperture during prehension and of the reported size of the object revealed that under normal and quick (but not under the fine ) gripper scaling conditions, the teleoperated grasping with our system was similar to natural grasping and, therefore, human-centered transparent. We anticipate that using motor control and psychophysics for human-centered optimization of teleoperation control will eventually improve the usability of RAMIS.

Robotics-Assisted Surgical Skills Evaluation based on Electrocortical Activity.

Skills assessment in Robotics-Assisted Minimally Invasive Surgery (RAMIS) is mainly performed based on temporal, motion-based and outcome-based metrics. While these components are essential for the proper assessment of skills in RAMIS, they do not suffice for full representation of all underlying aspects of skilled performance. Besides such commonplace components of skills, there exist other elements to be taken into account for comprehensive skills assessment. Among such elements are cognitive states (such as levels of stress, attention, concentration) that can directly affect performance. Investigating the impact of electrocortical activity and cognitive states of RAMIS surgeons over their performance has, however, received little attention in the literature. Therefore, in this paper, novel performance metrics based on electroencephalography (EEG) signals are studied for potential augmentation into RAMIS training and its assessment platform. For this purpose, a user study was conducted involving 23 novices and 9 expert RAMIS surgeons. The participants were asked to perform two tasks on the dv-Trainer®, (Mimic Technologies) RAMIS simulator, while their brain EEG signals were being measured using the Muse EEG headband (InteraXon Inc.). The performance metrics were defined as mean values of band powers of EEG signals over various ranges of frequency. Statistical analysis was performed to evaluate metrics over 5 different ranges of frequency for 4 electrode locations and during 2 RAMIS training tasks. The results indicated statistically significant differences in electrocortical activity between novices and experts in temporoparietal and left frontal regions of their brain for mid to high-frequency ranges. Overall, RAMIS experts showed lower levels of electrocortical activity in those regions compared to novices. The results indicate that electrocortical activity measured by EEG signals have the potential to provide useful information for skills assessment in RAMIS.