Intraoperative Scene Research Articles

Towards multimodal graph neural networks for surgical instrument anticipation

PurposeDecision support systems and context-aware assistance in the operating room have emerged as the key clinical applications supporting surgeons in their daily work and are generally based on single modalities. The model- and knowledge-based integration of multimodal data as a basis for decision support systems that can dynamically adapt to the surgical workflow has not yet been established. Therefore, we propose a knowledge-enhanced method for fusing multimodal data for anticipation tasks.MethodsWe developed a holistic, multimodal graph-based approach combining imaging and non-imaging information in a knowledge graph representing the intraoperative scene of a surgery. Node and edge features of the knowledge graph are extracted from suitable data sources in the operating room using machine learning. A spatiotemporal graph neural network architecture subsequently allows for interpretation of relational and temporal patterns within the knowledge graph. We apply our approach to the downstream task of instrument anticipation while presenting a suitable modeling and evaluation strategy for this task.ResultsOur approach achieves an F1 score of 66.86% in terms of instrument anticipation, allowing for a seamless surgical workflow and adding a valuable impact for surgical decision support systems. A resting recall of 63.33% indicates the non-prematurity of the anticipations.ConclusionThis work shows how multimodal data can be combined with the topological properties of an operating room in a graph-based approach. Our multimodal graph architecture serves as a basis for context-sensitive decision support systems in laparoscopic surgery considering a comprehensive intraoperative operating scene.

FRSR: Framework for real-time scene reconstruction in robot-assisted minimally invasive surgery

3D reconstruction of the intra-operative scenes provides precise position information which is the foundation of various safety related applications in robot-assisted surgery, such as augmented reality. Herein, a framework integrated into a known surgical system is proposed to enhance the safety of robotic surgery. In this paper, we present a scene reconstruction framework to restore the 3D information of the surgical site in real time. In particular, a lightweight encoder–decoder network is designed to perform disparity estimation, which is the key component of the scene reconstruction framework. The stereo endoscope of da Vinci Research Kit (dVRK) is adopted to explore the feasibility of the proposed approach, and it provides the possibility for the migration to other Robot Operating System (ROS) based robot platforms due to the strong independence on hardware. The framework is evaluated using three different scenarios, including a public dataset (3018 pairs of endoscopic images), the scene from the dVRK endoscope in our lab as well as a self-made clinical dataset captured from an oncology hospital. Experimental results show that the proposed framework can reconstruct 3D surgical scenes in real time (25 FPS), and achieve high accuracy (2.69 ± 1.48 mm in MAE, 5.47 ± 1.34 mm in RMSE and 0.41 ± 0.23 in SRE, respectively). It demonstrates that our framework can reconstruct intra-operative scenes with high reliability of both accuracy and speed, and the validation of clinical data also shows its potential in surgery. This work enhances the state of art in 3D intra-operative scene reconstruction based on medical robot platforms. The clinical dataset has been released to promote the development of scene reconstruction in the medical image community.

Learning feature descriptors for pre- and intra-operative point cloud matching for laparoscopic liver registration.

In laparoscopic liver surgery, preoperative information can be overlaid onto the intra-operative scene by registering a 3D preoperative model to the intra-operative partial surface reconstructed from the laparoscopic video. To assist with this task, we explore the use of learning-based feature descriptors, which, to our best knowledge, have not been explored for use in laparoscopic liver registration. Furthermore, a dataset to train and evaluate the use of learning-based descriptors does not exist. We present the LiverMatch dataset consisting of 16 preoperative models and their simulated intra-operative 3D surfaces. We also propose the LiverMatch network designed for this task, which outputs per-point feature descriptors, visibility scores, and matched points. We compare the proposed LiverMatch network with a network closest to LiverMatch and a histogram-based 3D descriptor on the testing split of the LiverMatch dataset, which includes two unseen preoperative models and 1400 intra-operative surfaces. Results suggest that our LiverMatch network can predict more accurate and dense matches than the other two methods and can be seamlessly integrated with a RANSAC-ICP-based registration algorithm to achieve an accurate initial alignment. The use of learning-based feature descriptors in laparoscopic liver registration (LLR) is promising, as it can help achieve an accurate initial rigid alignment, which, in turn, serves as an initialization for subsequent non-rigid registration.

Real-time stereo reconstruction of intraoperative scene and registration to preoperative 3D models for augmenting surgeons' view during RAMIS.

During minimally invansive surgery (MIS) procedures, there exists an ever-growing/apparent need for providing computer generated visual feedback to the surgeon(s), through a visualization device. While multiple solutions have been proposed in the literature, there is limited evidence of such a system performing reliably in practice, and when it does, it is often tailored to a specific operation type. Another important aspect is regarding the usability of such systems, which typically include complicated and time-consuming steps, and often require the assistance of specialized personnel. In this study, we propose an auxiliary visualization system for surgeons, which includes streamlined process to use preoperative data of the patient, and apply it to two different MIS cases, namely, robot-assisted partial nephrectomy and robot-assisted partial lateral meniscectomy. The visualization and processing pipeline consists of an intraoperative 3D reconstruction of the surgical area, using an optimized version of the quasi-dense method, aimed to perform with good accuracy while maintaining real-time speed. A set of preprocessing and postprocessing techniques further contribute to the result by providing a smoother and more dense point cloud. DynamicFusion is used for the registration of the preoperative model to the intraoperative scene. Two silicon kidney phantoms and an ex-vivo porcine meniscus are used for evaluation, representing subjects for the examined surgicalcases. Performance is evaluated qualitatively using the two datasets. The preoperative model of the subject is projected on top of the actual 2D image and also in 3D space. The model is superimposed on top of the actual physical structure it represents, and remains in the correct position throughout the experiments, even when abrupt camera movements are taking place. Finally, when deformation is introduced, the model is deformed as well, resembling the real subject's structure. Results demonstrate and validate the use of the presented algorithms for each separate task of the pipeline. A complete methodology to provide surgeon(s) with visual information during surgery is presented. Its operation is evaluated over two different surgical scenarios, paving the way for a single visualization methodology that can adapt and perform robustly for multiple cases, with minimaleffort.

Augmented reality-based autostereoscopic surgical visualization system for telesurgery.

The visualization of remote surgical scenes is the key to realizing the remote operation of surgical robots. However, current non-endoscopic surgical robot systems lack an effective visualization tool to offer sufficient surgical scene information and depth perception. We propose a novel autostereoscopic surgical visualization system integrating 3D intraoperative scene reconstruction, autostereoscopic 3D display, and augmented reality-based image fusion. The preoperative organ structure and the intraoperative surface point cloud are obtained from medical imaging and the RGB-D camera, respectively, and aligned by an automatic marker-free intraoperative registration algorithm. After registration, preoperative meshes with precalculated illumination and intraoperative textured point cloud are blended in real time. Finally, the fused image is shown on a 3D autostereoscopic display device to achieve depth perception. A prototype of the autostereoscopic surgical visualization system was built. The system had a horizontal image resolution of 1.31mm, a vertical image resolution of 0.82mm, an average rendering rate of 33.1 FPS, an average registration rate of 20.5 FPS, and average registration errors of approximately 3mm. A telesurgical robot prototype based on 3D autostereoscopic display was built. The quantitative evaluation experiments showed that our system achieved similar operational accuracy (1.79 ± 0.87mm) as the conventional system (1.95 ± 0.71mm), while having advantages in terms of completion time (with 34.11% reduction) and path length (with 35.87% reduction). Post-experimental questionnaires indicated that the system was user-friendly for novices and experts. We propose a 3D surgical visualization system with augmented instruction and depth perception for telesurgery. The qualitative and quantitative evaluation results illustrate the accuracy and efficiency of the proposed system. Therefore, it shows great prospects in robotic surgery and telesurgery.

Global rigid registration of CT to video in laparoscopic liver surgery

PurposeImage-guidance systems have the potential to aid in laparoscopic interventions by providing sub-surface structure information and tumour localisation. The registration of a preoperative 3D image with the intraoperative laparoscopic video feed is an important component of image guidance, which should be fast, robust and cause minimal disruption to the surgical procedure. Most methods for rigid and non-rigid registration require a good initial alignment. However, in most research systems for abdominal surgery, the user has to manually rotate and translate the models, which is usually difficult to perform quickly and intuitively.MethodsWe propose a fast, global method for the initial rigid alignment between a 3D mesh derived from a preoperative CT of the liver and a surface reconstruction of the intraoperative scene. We formulate the shape matching problem as a quadratic assignment problem which minimises the dissimilarity between feature descriptors while enforcing geometrical consistency between all the feature points. We incorporate a novel constraint based on the liver contours which deals specifically with the challenges introduced by laparoscopic data.ResultsWe validate our proposed method on synthetic data, on a liver phantom and on retrospective clinical data acquired during a laparoscopic liver resection. We show robustness over reduced partial size and increasing levels of deformation. Our results on the phantom and on the real data show good initial alignment, which can successfully converge to the correct position using fine alignment techniques. Furthermore, since we can pre-process the CT scan before surgery, the proposed method runs faster than current algorithms.ConclusionThe proposed shape matching method can provide a fast, global initial registration, which can be further refined by fine alignment methods. This approach will lead to a more usable and intuitive image-guidance system for laparoscopic liver surgery.

Minimally disruptive needle insertion: a biologically inspired solution.

The mobility of soft tissue can cause inaccurate needle insertions. Particularly in steering applications that employ thin and flexible needles, large deviations can occur between pre-operative images of the patient, from which a procedure is planned, and the intra-operative scene, where a procedure is executed. Although many approaches for reducing tissue motion focus on external constraining or manipulation, little attention has been paid to the way the needle is inserted and actuated within soft tissue. Using our biologically inspired steerable needle, we present a method of reducing the disruptiveness of insertions by mimicking the burrowing mechanism of ovipositing wasps. Internal displacements and strains in three dimensions within a soft tissue phantom are measured at the needle interface, using a scanning laser-based image correlation technique. Compared to a conventional insertion method with an equally sized needle, overall displacements and strains in the needle vicinity are reduced by 30% and 41%, respectively. The results show that, for a given net speed, needle insertion can be made significantly less disruptive with respect to its surroundings by employing our biologically inspired solution. This will have significant impact on both the safety and targeting accuracy of percutaneous interventions along both straight and curved trajectories.

Intraoperative Visualization of Hepatocellular Carcinoma with Indocyanine Green: Revealing the Mechanisms behind the Glowing Tumor

Reliable intraoperative detection of tumors is important in the surgical management of malignant disease. In hepatocellular carcinoma (HCC), for instance, it is well known that failure to achieve R0 resections results in recurrence and reduced survival. Beyond the tactile and visual senses of the surgeon, additional tools are needed to reveal the extent of tumors and their involvement with nearby anatomy in order to achieve safe, curative resections. However, the technologies used to provide such anatomical and functional information for guiding resections have not changed considerably over the years and are based largely on conventional imaging modalities used preoperatively. Of these, intraoperative ultrasound (IOUS) remains the most common form of imaging available to visualize intra-abdominal tumors, although other modalities like intraoperative magnetic resonance and computed tomography have been tried in neurosurgery. To exploit fully the pre-operative advantages of these modalities inside the operating room, many scientific, engineering, and logistical challenges remain to be solved. One problem is that these imaging systems typically discriminate malignant from benign tissues on the basis of their physical properties (e.g. density, vascular flow, etc.) and generally not on their actual biology (e.g. tumor-specific proteins, genetic mutations, etc.). Another problem is that the images seen on a two-dimensional monitor are presented without the three-dimensional context of the actual patient and must be reoriented back in the operative field in order to be useful. Although systems exist to map pre-operative scans onto the intra-operative scene, these technologies are still being evaluated. Without addressing these problems, such images would only serve to suggest a possible tumor in a possible location, requiring the surgeon to explore and sample the tissue to confirm a tumor. In this issue, Ishizawa et al. have explored the use of indocyanine green (ICG) in vivo to make HCCs fluorescent for direct visualization intraoperatively. Unlike the imaging methods described above, the authors demonstrate that the tumor retention of ICG is based, in part, on its biology and that the tumor fluorescence is directly visible intraoperatively in nearly every patient of a fairly large series. As a historical note, ICG was first described in liver surgery, not as a tumor marker but as a means of assessing peri-operative liver function via the clearance of the marker by the organ. ICG rapidly binds albumin upon administration and is then transported to the liver, where it is excreted into the biliary system. Later, ICG was also shown to fluoresce in the near-infrared (NIR) wavelength and was often used as an indicator dye in angiography. This fluorescent property makes ICG and other similar dyes attractive for intra-operative use because its detection is not based on ionizing radiation. Furthermore, at the wavelength at which ICG is excited, the surrounding tissues do not contribute to background auto-fluorescence, allowing ICG to be detected with a high signal-to-noise ratio. Another property of ICG that engendered its clinical use is that it is relatively non-toxic, with few side effects, although the application of ICG as an imaging agent is still considered off-label use. It was against this historical backdrop that the preferential retention of ICG in HCC was accidentally discovered. Nevertheless, the mechanism by which this retention occurred remains unknown. It was hypothesized that ICG retention was a product of disruption in biliary excretion and from leaky portal vasculature near the tumor. Despite these uncertainties, multiple applications of ICG in oncologic surgeries have since been reported. Society of Surgical Oncology 2013

Deformable image registration for cone-beam CT guided transoral robotic base-of-tongue surgery

Transoral robotic surgery (TORS) offers a minimally invasive approach to resection of base-of-tongue tumors. However, precise localization of the surgical target and adjacent critical structures can be challenged by the highly deformed intraoperative setup. We propose a deformable registration method using intraoperative cone-beam computed tomography (CBCT) to accurately align preoperative CT or MR images with the intraoperative scene. The registration method combines a Gaussian mixture (GM) model followed by a variation of the Demons algorithm. First, following segmentation of the volume of interest (i.e. volume of the tongue extending to the hyoid), a GM model is applied to surface point clouds for rigid initialization (GM rigid) followed by nonrigid deformation (GM nonrigid). Second, the registration is refined using the Demons algorithm applied to distance map transforms of the (GM-registered) preoperative image and intraoperative CBCT. Performance was evaluated in repeat cadaver studies (25 image pairs) in terms of target registration error (TRE), entropy correlation coefficient (ECC) and normalized pointwise mutual information (NPMI). Retraction of the tongue in the TORS operative setup induced gross deformation >30 mm. The mean TRE following the GM rigid, GM nonrigid and Demons steps was 4.6, 2.1 and 1.7 mm, respectively. The respective ECC was 0.57, 0.70 and 0.73, and NPMI was 0.46, 0.57 and 0.60. Registration accuracy was best across the superior aspect of the tongue and in proximity to the hyoid (by virtue of GM registration of surface points on these structures). The Demons step refined registration primarily in deeper portions of the tongue further from the surface and hyoid bone. Since the method does not use image intensities directly, it is suitable to multi-modality registration of preoperative CT or MR with intraoperative CBCT. Extending the 3D image registration to the fusion of image and planning data in stereo-endoscopic video is anticipated to support safer, high-precision base-of-tongue robotic surgery.

Fusion and visualization of intraoperative cortical images with preoperative models for epilepsy surgical planning and guidance

Objective: During epilepsy surgery it is important for the surgeon to correlate the preoperative cortical morphology (from preoperative images) with the intraoperative environment. Augmented Reality (AR) provides a solution for combining the real environment with virtual models. However, AR usually requires the use of specialized displays, and its effectiveness in the surgery still needs to be evaluated. The objective of this research was to develop an alternative approach to provide enhanced visualization by fusing a direct (photographic) view of the surgical field with the 3D patient model during image guided epilepsy surgery.Materials and Methods: We correlated the preoperative plan with the intraoperative surgical scene, first by a manual landmark-based registration and then by an intensity-based perspective 3D-2D registration for camera pose estimation. The 2D photographic image was then texture-mapped onto the 3D preoperative model using the solved camera pose. In the proposed method, we employ direct volume rendering to obtain a perspective view of the brain image using GPU-accelerated ray-casting. The algorithm was validated by a phantom study and also in the clinical environment with a neuronavigation system.Results: In the phantom experiment, the 3D Mean Registration Error (MRE) was 2.43 ± 0.32 mm with a success rate of 100%. In the clinical experiment, the 3D MRE was 5.15 ± 0.49 mm with 2D in-plane error of 3.30 ± 1.41 mm. A clinical application of our fusion method for enhanced and augmented visualization for integrated image and functional guidance during neurosurgery is also presented.Conclusions: This paper presents an alternative approach to a sophisticated AR environment for assisting in epilepsy surgery, whereby a real intraoperative scene is mapped onto the surface model of the brain. In contrast to the AR approach, this method needs no specialized display equipment. Moreover, it requires minimal changes to existing systems and workflow, and is therefore well suited to the OR environment. In the phantom and in vivo clinical experiments, we demonstrate that the fusion method can achieve a level of accuracy sufficient for the requirements of epilepsy surgery.

Cortical Shift Tracking Using a Laser Range Scanner and Deformable Registration Methods.

A novel brain shift tracking protocol is introduced in this paper which utilizes laser range scan (LRS) data and 2D deformable image registration. This protocol builds on previous efforts to incorporate intra-operative LRS data into a model-updated image guided surgery paradigm for brain shift compensation. The shift tracking method employs the use of a LRS system capable of capturing textures of the intra-operative scene during range data acquisition. Textures from serial range images are then registered using a 2D deformable registration approach that uses local support radial basis functions and mutual information. Given the deformation field provided by the registration, 3D points in serial LRS datasets can then be tracked. Results from this paper indicate that the error associated with tracking brain movement is 1.1mm on average given brain shifts of approximately 20.5mm. Equally important, a strategy is presented to rapidly acquire intra-operative measurements of shift which are compatible with model-based strategies for brain deformation compensation.

Deforming a Preoperative Volume to Represent the Intraoperative Scene

Soft-tissue deformation can be a problem if a preoperative modality is used to help guide a surgical or interventional procedure. We present a method that can-warp a preoperative CT image to represent the intraoperative scene shown by an interventional fluoroscopy image. The method is a novel combination of a 2D-3D image registration algorithm and a deformation algorithm that allows rigid bodies to be incorporated into a nonlinear deformation based on radial basis functions. The 2D-3D registration algorithm is used to obtain information on relative vertebral movements between preoperative and intraoperative images. The deformation algorithm uses this information to warp the preoperative image to represent the intraoperative scene more accurately. Images from an aortic stenting procedure were used. The observed deformation in our experiment was 5° flexion and 5 mm lengthening of the lumbar spine over a distance of four vertebrae. The vertebral positions in the warped CT volume represent the intraoperative scene more accurately than in the preoperative CT volume. Although we had no gold standard with which to assess the registration accuracy of soft-tissue structures, the position of such structures within the warped CT volume appeared visually realistic.

Deforming a preoperative volume to represent the intraoperative scene.

Soft-tissue deformation can be a problem if a preoperative modality is used to help guide a surgical or interventional procedure. We present a method that can warp a preoperative CT image to represent the intraoperative scene shown by an interventional fluoroscopy image. The method is a novel combination of a 2D-3D image registration algorithm and a deformation algorithm that allows rigid bodies to be incorporated into a nonlinear deformation based on radial basis functions. The 2D-3D registration algorithm is used to obtain information on relative vertebral movements between preoperative and intraoperative images. The deformation algorithm uses this information to warp the preoperative image to represent the intraoperative scene more accurately. Images from an aortic stenting procedure were used. The observed deformation in our experiment was 5 degrees flexion and 5 mm lengthening of the lumbar spine over a distance of four vertebrae. The vertebral positions in the warped CT volume represent the intraoperative scene more accurately than in the preoperative CT volume. Although we had no gold standard with which to assess the registration accuracy of soft-tissue structures, the position of such structures within the warped CT volume appeared visually realistic.

Validation of a two- to three-dimensional registration algorithm for aligning preoperative CT images and intraoperative fluoroscopy images.

We present a validation of an intensity based two- to three-dimensional image registration algorithm. The algorithm can register a CT volume to a single-plane fluoroscopy image. Four routinely acquired clinical data sets from patients who underwent endovascular treatment for an abdominal aortic aneurysm were used. Each data set was comprised of two intraoperative fluoroscopy images and a preoperative CT image. Regions of interest (ROI) were drawn around each vertebra in the CT and fluoroscopy images. Each CT image ROI was individually registered to the corresponding ROI in the fluoroscopy images. A cross validation approach was used to obtain a measure of registration consistency. Spinal movement between the preoperative and intraoperative scene was accounted for by using two fluoroscopy images. The consistency and robustness of the algorithm when using two similarity measures, pattern intensity and gradient difference, was investigated. Both similarity measures produced similar results. The consistency values were rotational errors below 0.74 degree and in-plane translational errors below 0.90 mm. These errors approximately relate to a two-dimensional projection error of 1.3 mm. The failure rate was less than 8.3% for three of the four data sets. However, for one of the data sets a much larger failure rate (28.5%) occurred.