Endoscopic Navigation Research Articles

Toward Human-Out-of-the-Loop Endoscope Navigation Based on Context Awareness for Enhanced Autonomy in Robotic Surgery

Toward Human-Out-of-the-Loop Endoscope Navigation Based on Context Awareness for Enhanced Autonomy in Robotic Surgery

Multisource Differential Fusion Driven Monocular Endoscope Hybrid 3-D Tracking for Advanced Endoscopic Navigation Surgery.

Surgical navigation systems involve various technologies of segmentation, calibration, registration, tracking, and visualization. These systems aim to superimpose multisource information in the surgical field and provide surgeons with a composite overlay (augmented-reality) view, improving the operative precision and experience. Surgical 3-D tracking is the key to build these systems. Unfortunately, surgical 3-D tracking is still a challenge to endoscopic and robotic navigation systems and easily gets trapped in image artifacts, tissue deformation, and inaccurate positional (e.g., electromagnetic) sensor measurements. This work explores a new monocular endoscope hybrid 3-D tracking method called spatially constrained adaptive differential evolution that combines two spatial constraints with observation-recall adaptive propagation and observation-based fitness computing for stochastic optimization. Specifically, we spatially constraint inaccurate electromagnetic sensor measurements to the centerline of anatomical tubular structures to keep them physically locating inside the tubes, as well as interpolate these measurements to reduce jitter errors for smooth 3-D tracking. We then propose observation-recall adaptive propagation with fitness computing to precisely fuse the constrained sensor measurements, preoperative images, and endoscopic video sequences for accurate hybrid 3-D tracking. Additionally, we also propose a new marker-free hybrid registration strategy to precisely align positional sensor measurements to preoperative images. Our new framework was evaluated on a large amount of clinical data acquired from various surgical endoscopic procedures, with the experimental results showing that it certainly outperforms current surgical 3-D approaches. In particular, the position and rotation errors were significantly reduced from (6.55, 11.4) to (3.02 mm, 8.54 °).

Automatic endoscopic navigation based on attention-based network for Nasotracheal Intubation

Nasotracheal intubation (NTI) is one of the most commonly performed procedures in anesthesia and is considered the gold standard for securing the airway of patients. Endoscope operation is critical to the success of NTI. However, this operation remains challenging as it requires the surgeon to classify anatomical landmarks and detect heading targets of the endoscope tip in a sequence of monocular images. To alleviate this problem, this study presents a learning-based navigation method that automatically classifies four different anatomical landmarks and detects the heading target of the endoscope tip from endoscopic images. First, an end-to-end multitask network is introduced that consists of a branch for anatomical landmark classification and another for heading target detection. In addition, a convolutional attention module is designed to improve network performance by combining spatial and channel attention. Second, an endoscopic dataset named intuNav is built for network training. The trained network calculates navigation information without image prior knowledge. Finally, extensive experiments on the built dataset and endoscopic videos demonstrate the high performance of our method, achieving classification (94%) and detection (79.4%) accuracies. The results also indicate that the proposed method is effective and efficient in navigation generation during NTI.

Advancing Education in Endoscopic Spinal Navigation: Novel Methods and Technical Note

This report aims to demonstrate how to teach anatomy and understanding of spinal endoscopic vision and navigation using mnemonics. The authors present a new surgical technique for teaching endoscopic spinal navigation in a didactic manner with tips such as the "rule of the hand" and decomposition of the endoscopic navigation movement. We demonstrate how the surgery is seen and illustrate how images are projected onto the screen, then divide the navigation into spatial orientation and self-navigation. The article describes the proper puncture technique, how to introduce the working portal, and how to assimilate this new anatomical vision using the "rule of the hand." The surgeon projects their hand on the video screen to guide themselves when starting the navigation and uses the same technique to localize regions of interest during surgery. Finally, the authors break down the navigational movement into three components: forceps positioning, triangulation, and joystick motion. One of the biggest challenges when learning spinal endoscopic surgery is understanding the anatomy seen through the endoscope. By decomposing movements required for navigation, one can understand how to make proper use of the equipment as well as improve their knowledge of this "new anatomy." The learning methods taught in this article have the potential to decrease the learning curve and radiation exposure to those that are still acquainting themselves to spinal endoscopic navigation. We recommend that further studies measure and quantify the impact of these methods on surgical practice.

A geometry-aware deep network for depth estimation in monocular endoscopy

Monocular depth estimation is critical for endoscopists to perform spatial perception and 3D navigation of surgical sites. However, most of the existing methods ignore the important geometric structural consistency, which inevitably leads to performance degradation and distortion of 3D reconstruction. To address this issue, we introduce a gradient loss to penalize edge fluctuations ambiguous around stepped edge structures and a normal loss to explicitly express the sensitivity to frequently small structures, and propose a geometric consistency loss to spreads the spatial information across the sample grids to constrain the global geometric anatomy structures. In addition, we develop a synthetic RGB-Depth dataset that captures the anatomical structures under reflections and illumination variations. The proposed method is extensively validated across different datasets and clinical images and achieves mean RMSE values of 0.066 (stomach), 0.029 (small intestine), and 0.139 (colon) on the EndoSLAM dataset. The generalizability of the proposed method achieves mean RMSE values of 12.604 (T1-L1), 9.930 (T2-L2), and 13.893 (T3-L3) on the ColonDepth dataset. The experimental results show that our method exceeds previous state-of-the-art competitors and generates more consistent depth maps and reasonable anatomical structures. The quality of intraoperative 3D structure perception from endoscopic videos of the proposed method meets the accuracy requirements of video-CT registration algorithms for endoscopic navigation. The dataset and the source code will be available at https://github.com/YYM-SIA/LINGMI-MR.

EndoTrainer: a novel hybrid training platform for endoscopic surgery

PurposeEndoscopy implies high demanding procedures, and their practice requires structured formation curricula supported by adequate training platforms. Physical platforms are the most standardised solution for surgical training, but over the last few years, virtual platforms have been progressively introduced. This research work presents a new hybrid, physic-virtual, endoscopic training platform that exploits the benefits of the two kind of platforms combining realistic tools and phantoms together with the capacity of measuring all relevant parameters along the execution of the exercises and of providing an objective assessment performance.MethodsThe developed platform, EndoTrainer, has been designed to train and assess surgical skills in hysteroscopy and cystoscopy following a structured curricula. The initial development and validation is focused on hysteroscopic exercises proposed in the Gynaecological Endoscopic Surgical Education and Assessment (GESEA) Certification Programme from The Academy and European Society for Gynaecological Endoscopy (ESGE) and analyses the obtained results of an extensive study with 80 gynaecologists executing 30 trials of the standard 30 degree endoscope navigation exercise.ResultsThe experiments demonstrate the benefits of the presented hybrid platform. Multi-variable statistical analysis points out that all subjects have obtained statistically significant improvement in all relevant parameters: shorter and safer trajectories, improved 30-degree endoscope navigation, accurate positioning over the targets and reduction of the execution time.ConclusionThis paper presents a new hybrid approach for training, and evaluating whether it provides an objectivable improvement of camera navigation endoscopic basic skills. The obtained results demonstrate the initial hypothesis: all subjects have improved their camera handling and navigation skills.

Endoscope-navigation-assisted orbital decompression for graves' orbitopathy.

To compare the surgical outcomes of endoscope-navigation (EN)-assisted orbital decompression and non-EN-assisted orbital decompression for Graves' orbitopathy (GO) and to assess the potential clinical advantage of EN in orbital decompression surgery. This retrospective cohort study was performed on 227 orbits of 147 GO patients who underwent EN-assisted orbital decompression (185 orbits) or non-EN-assisted orbital decompression (42 orbits). Assessment included proptosis reduction, best-corrected visual acuity (BCVA), diplopia, ocular restriction and surgical complications. The proptosis reduction in the EN group was 0.9 mm greater than that in the non-EN group in the entire cohort (p = 0.004) and 1.0 mm greater than that in the non-EN group in the propensity score matching cohort (p = 0.025) at 2 years postoperatively. In all, 78.2% of orbits with sight-threatening GO in the EN group and 52.6% of orbits in the non-EN group showed BCVA improvement (p = 0.026). The proportion of patients with improvement in diplopia was significantly greater in the EN group than in the non-EN group (p = 0.026). EN offers anatomical localization and deep-seated tissue visualization in orbital decompression and significantly improves the surgical outcomes for GO.

Study on impact of flexible endoscopy training course for surgeons in India.

Competence in flexible endoscopy is essential for all surgeons during this era of minimal access surgery. However, fewer surgeons have expertise in endoscopy due to a lack of training and interest. The Indian Association of Gastrointestinal Endo Surgeons devised a short-structured training course in the art and science of endoscopy. This study aimed to find the impact of the endoscopy training course (Endoscopic Fellowship of Indian Association of Gastrointestinal Endo Surgeons [EFIAGES]) in improving the endoscopic skill of surgeons. Twenty-two-part electronic survey forms were sent to all 375 candidates who took the course between 2016 and 2019 for this retrospective observational study. The following outcome measures were noted, namely technical competence in endoscopy before the course, delegate feedback about the course modules, volume of endoscopies before and after the course and quality indicators such as reaching up to duodenum (D2) and caecum before and after the course. Statistical analysis of the impact of the course was done using Chi-square test. Responses from 262 out of a total of 375 candidates were received. Seventy-seven per cent of trainees were pleased with content and mode of conduct of the course. The quality indicator of gastroscopy with the ability to reach D2 in 90% of the caseload was achieved by only 28% of trainees before the EFIAGES. This increased to 72% of candidates after the course and similar results were seen with colonoscopy also. Most of the candidates noted a distinct improvement in their endoscopic navigation skills subsequent to the course. Endoscopy skill transfer was possible with a short-structured endoscopy course. The surgical fraternity should realise the importance of endoscopy skills in the current era of surgical practice.

Anatomy aware-based 2.5D bronchoscope tracking for image-guided bronchoscopic navigation

ABSTRACT Physicians use an endoscopic navigation system during bronchoscopy to decrease the risk of getting lost in complex tree-structure like bronchus. Most existing navigation systems based on the camera pose estimated from bronchoscope tracking and/or deep learning. However, bronchoscope tracking-based method exists tracking error, and the pre-training of the model needs massive data. This paper describes an improved bronchoscope tracking procedure by adopting image domain translation technique to improve tracking performance. Specifically, our scheme consists of three modules, an RGB-D image domain translation module, an anatomical structure classification module and a structure-aware bronchoscope tracking module. The RGB-D image domain translation module translates a real bronchoscope (RB) image to its corresponding virtual bronchoscope image and depth image. The anatomical dependency module classifies the current scene into two categories: structureless and rich structure. The bronchoscope tracking module uses a modified video-CT bronchoscope tracking approach to estimate camera pose. Experimental results showed that the proposed method achieved higher tracking accuracy than the current state-of-the-art bronchoscope tracking methods.

An incremental registration method for endoscopic sinus and skull base surgery navigation: From phantom study to clinical trials.

Surface-based image-to-patient registration in current surgical navigation is mainly achieved by a 3D scanner, which has several limitations in clinical practice such as uncontrollable scanning range, complicated operation, and even high failure rate. An accurate, robust, and easy-to-perform image-to-patient registration method is urgently required. An incremental point cloud registration method was proposed for surface-based image-to-patient registration. The point cloud in image space was extracted from the computed tomography (CT) image, and a template matching method was applied to remove the redundant points. The corresponding point cloud in patient space was incrementally collected by an optically tracked pointer, while the nearest point distance (NPD) constraint was applied to ensure the uniformity of the collected points. A coarse-to-fine registration method under the constraints of coverage ratio (CR) and outliers ratio (OR) was then proposed to obtain the optimal rigid transformation from image to patient space. The proposed method was integrated in the recently developed endoscopic navigation system, and phantom study and clinical trials were conducted to evaluate the performance of the proposedmethod. The results of the phantom study revealed that the proposed constraints greatly improved the accuracy and robustness of registration. The comparative experimental results revealed that the proposed registration method significantly outperform the scanner-based method, and achieved comparable accuracy to the fiducial-based method. In the clinical trials, the average registration duration was 1.24 ± 0.43 min, the target registration error (TRE) of 294 marker points (59 patients) was 1.25 ± 0.40 mm, and the lower 97.5% confidence limit of the success rate of positioning marker points exceeds the expected value (97.56% vs. 95.00%), revealed that the accuracy of the proposed method significantly met the clinical requirements (TRE ⩽ 2 mm, p<0.05). The proposed method has both the advantages of high accuracy and convenience, which were absent in the scanner-based method and the fiducial-based method. Our findings will help improve the quality of endoscopic sinus and skull basesurgery.

Electromagnetic Tracking of Elongated Sensors for Endoscopic Navigation

As the prevalence of image-guided interventions increases, electromagnetic tracking (EMT) systems play an important role in modern patient care, as they enable real-time instrument positioning and navigation inside the human body without line-of-sight restrictions. Miniature-size inductive coils are the gold standard in clinical settings, as they provide accurate, passive sensing of the magnetic field. To compensate for their small dimensions, such sensors are designed with an elongated shape, where the coil length is usually 10 to 20 times larger than the diameter. In this article, the benefits of a field model based on the magnetic scalar potential formulation are demonstrated for EMT applications where elongated tracking sensors are used. The novel method resolves the single-point approximation error when the coil length is not negligible, and demonstrates improvements in terms of speed and storage requirements. A detailed analysis is proposed where alternative formulations of the magnetic model used in the tracking algorithm are compared. Although this work does not resolve any substantial limits of EMT used in a clinical environment, which are mainly caused by the presence of magnetic distortions, the proposed method is an improvement over existing EMT systems because it enables more accurate and faster tracking. The method might facilitate the use of longer tracking sensor coils which can achieve high sensitivities without the requirement of a magnetic core. In the envisioned application, such coils may be wound around flexible instruments, such as endoscopes or catheters.

Robotic Autonomy for Magnetic Endoscope Biopsy.

Magnetically actuated endoscopes are currently transitioning in to clinical use for procedures such as colonoscopy, presenting numerous benefits over their conventional counterparts. Intelligent and easy-to-use control strategies are an essential part of their clinical effectiveness due to the un-intuitive nature of magnetic field interaction. However, work on developing intelligent control for these devices has mainly been focused on general purpose endoscope navigation. In this work, we investigate the use of autonomous robotic control for magnetic colonoscope intervention via biopsy, another major component of clinical viability. We have developed control strategies with varying levels of robotic autonomy, including semi-autonomous routines for identifying and performing targeted biopsy, as well as random quadrant biopsy. We present and compare the performance of these approaches to magnetic endoscope biopsy against the use of a standard flexible endoscope on bench-top using a colonoscopy training simulator and silicone colon model. The semi-autonomous routines for targeted and random quadrant biopsy were shown to reduce user workload with comparable times to using a standard flexible endoscope.

The Orbital Apex Cell as a Landmark for Endoscopic Navigation to the Sphenoid Sinus.

The Orbital Apex Cell as a Landmark for Endoscopic Navigation to the Sphenoid Sinus.

Topographic and anatomical landmarks for endoscopic navigation in surgery for complicated gastric cancer

To develop an algorithm for sequential visual navigation and imaging of the most permanent topographic and anatomical landmarks for safe laparoscopic surgery of complicated gastric cancer. We analyzed 42 laparoscopic surgeries for complicated locally advanced gastric cancer. Anatomical navigational landmarks and technical aspects of their safe isolation during laparoscopic surgery for gastric cancer are recommended. The topographic-anatomical navigation system based on the most constant anatomical landmarks ensures safe laparoscopic interventions for complicated locally advanced gastric cancer.

Automating Endoscope Motion in Robotic Surgery: A Usability Study on da Vinci-Assisted Ex Vivo Neobladder Reconstruction.

Robots for minimally invasive surgery introduce many advantages, but still require the surgeon to alternatively control the surgical instruments and the endoscope. This work aims at providing autonomous navigation of the endoscope during a surgical procedure. The autonomous endoscope motion was based on kinematic tracking of the surgical instruments and integrated with the da Vinci Research Kit. A preclinical usability study was conducted by 10 urologists. They carried out an ex vivo orthotopic neobladder reconstruction twice, using both traditional and autonomous endoscope control. The usability of the system was tested by asking participants to fill standard system usability scales. Moreover, the effectiveness of the method was assessed by analyzing the total procedure time and the time spent with the instruments out of the field of view. The average system usability score overcame the threshold usually identified as the limit to assess good usability (average score = 73.25 > 68). The average total procedure time with the autonomous endoscope navigation was comparable with the classic control (p = 0.85 > 0.05), yet it significantly reduced the time out of the field of view (p = 0.022 < 0.05). Based on our findings, the autonomous endoscope improves the usability of the surgical system, and it has the potential to be an additional and customizable tool for the surgeon that can always take control of the endoscope or leave it to move autonomously.

Learning-Based Depth and Pose Estimation for Monocular Endoscope with Loss Generalization.

Gastroendoscopy has been a clinical standard for diagnosing and treating conditions that affect a part of a patient's digestive system, such as the stomach. Despite the fact that gastroendoscopy has a lot of advantages for patients, there exist some challenges for practitioners, such as the lack of 3D perception, including the depth and the endoscope pose information. Such challenges make navigating the endoscope and localizing any found lesion in a digestive tract difficult. To tackle these problems, deep learning-based approaches have been proposed to provide monocular gastroendoscopy with additional yet important depth and pose information. In this paper, we propose a novel supervised approach to train depth and pose estimation networks using consecutive endoscopy images to assist the endoscope navigation in the stomach. We firstly generate real depth and pose training data using our previously proposed whole stomach 3D reconstruction pipeline to avoid poor generalization ability between computer-generated (CG) models and real data for the stomach. In addition, we propose a novel generalized photometric loss function to avoid the complicated process of finding proper weights for balancing the depth and the pose loss terms, which is required for existing direct depth and pose supervision approaches. We then experimentally show that our proposed generalized loss performs better than existing direct supervision losses.

Homography-based robust pose compensation and fusion imaging for augmented reality based endoscopic navigation system

BackgroundAugmented reality (AR) based fusion imaging in endoscopic surgeries rely on the quality of image-to-patient registration and camera calibration, and these two offline steps are usually performed independently to get the target transformation separately. The optimal solution can be obtained under independent conditions but may not be globally optimal. All residual errors will be accumulated and eventually lead to inaccurate AR fusion. MethodsAfter a careful analysis of the principle of AR imaging, a robust online calibration framework was proposed for an endoscopic camera to enable accurate AR fusion. A 2D checkerboard-based homography estimation algorithm was proposed to estimate the local pose of the endoscopic camera, and the least square method was used to calculate the compensation matrix in combination with the optical tracking system. ResultsIn comparison with conventional methods, the proposed compensation method improved the performance of AR fusion, which reduced physical error by up to 82%, reduced pixel error by up to 83%, and improved target coverage by up to 6%. Experimental results of simulating mechanical noise revealed that the proposed compensation method effectively corrected the fusion errors caused by the rotation of the endoscopic tube without recalibrating the camera. Furthermore, the simulation results revealed the robustness of the proposed compensation method to noises. ConclusionsOverall, the experiment results proved the effectiveness of the proposed compensation method and online calibration framework, and revealed a considerable potential in clinical practice.

Gastrointestinal Residue Removal Using a Balloon Overtube under Ultrathin Endoscopic Navigation: Ex Vivo and In Vivo Experimental Studies.

Pooled gastric residues involving blood clots and food interrupt appropriate endoscopic intervention, leading to poor outcomes in endoscopic hemostasis and lifesaving. However, procedures and devices that enable the effective removal of gastrointestinal residues remain unsatisfactory. This study aimed to evaluate the efficacy and safety of our developed suction method in ex vivo and in vivo studies. We created a hand-made device with a large suction diameter, consisting of a balloon overtube and an ultrathin endoscope for navigation. In the ex vivo study, we compared the success rate and the suctioning time for four types of simulated residue between a standard endoscope and our device. Our device had s significantly higher suction ability and a shorter procedure time than the standard endoscope. The subsequent in vivo animal study involved five beagle dogs that were administered with food jelly to mimic gastric residue. Suction was performed twice for five dogs (ten attempts). The outcome measure was the successful procedure rate; secondary outcomes were procedure-associated complications and procedure time. The procedure was successful in all attempts, without any complications. The mean procedure time was 5 min. This novel method enabled the efficient and safe removal of gastric residue, and our findings will likely lead to the development of the instrument.

Development and Testing of an Anatomic in vitro Kidney Model for Measuring Intrapelvic Pressure During Ureteroscopy

ObjectivesTo create an in vitro anatomic bench model that can mimic in vivo intrapelvic pressure (IPP) during ureteroscopy (URS) and compare it against existing in vivo and ex vivo data. MethodsA silicone kidney model (Simagine Health) that permits intrarenal endoscopic navigation was engineered to have a fluid-tight seal for the ureteral opening and a Tuohy-Borst valve in the renal pelvis incorporating a 0.2 mm pressure sensor (Opsens). To calibrate the model, a Cobra ureteroscope (Wolf) was inserted to the pelvis with 200cmH2O irrigation, and the valve adjusted until an IPP of 54cmH2O was obtained (prior human data). All experiments were conducted with a laser fiber in the working channel, with and without ureteral access sheaths (UAS) (11/13F, 13/15F) at irrigation setting of 61, 102, 153, and 193cmH2O using an automated system (Rocamed). Study outcome was mean steady-state IPP for each UAS/irrigation condition. ResultsFluid leakage through the Tuohy-Borst valve, which could be adjusted, was critical to simulate ureteric outflow during URS. IPP values for each condition corresponded with data from in vivo and ex vivo models. In the no UAS condition, IPP increased with increasing irrigation pressures, and surpassed 40cmH2O when ≥153cmH2O. When using a UAS, IPP was below 40cmH2O for all irrigation pressures. ConclusionsAn in vitro kidney model can simulate in vivo and ex vivo IPP profiles, and be tailored to different conditions by controlling fluid outflow. This bench model can be useful for testing of new technologies and their impact on IPP.

The Future of Endoscopic Navigation: A Review of Advanced Endoscopic Vision Technology

Minimally invasive medicine has become mainstream because of its crucial clinical significance in providing a low risk of postoperative complications, limited blood loss, short postoperative recovery time, and small sizes of associated physiological tissue wounds. Endoscopic navigation systems comprise a research hot spot in medical science and technology and are an essential means to achieve precision medicine and improve surgical operation safety. As a core component in endoscopic navigation during minimally invasive surgery, endoscopes play a critical role in disease diagnosis and treatment. The development of endoscopic vision technologies has resulted in a renewed drive to further develop endoscopic navigation systems. Multiple endoscopic optical imaging modalities provide data sources for endoscopic vision technology, including white-light endoscopy, contrast-enhanced imaging and technologies involving magnified observation. Endoscopic vision is a specific application of computer vision involving the use of endoscopes that include instrument tracking, endoscopic view expansion, and suspicious lesion tracking in the application of endoscopic navigation. These techniques help surgeons or surgical robots locate instruments and lesions and expand the field of view of the endoscope. Although these technologies have been applied to various clinical and pre-clinical diagnoses and treatments, the use and combination of these advanced technologies in endoscopic navigation system for specific clinical requirements remains challenging. This review performs a broad survey of advanced endoscopic vision technologies and their application in endoscopic navigation systems. Finally, we discuss the challenges and future directions in implementing and developing endoscopic navigation systems.