Monocular Endoscope Research Articles

Gastrointestinal image stitching based on improved unsupervised algorithm.

Image stitching is a traditional but challenging computer vision task. The goal is to stitch together multiple images with overlapping areas into a single, natural-looking, high-resolution image without ghosts or seams. This article aims to increase the field of view of gastroenteroscopy and reduce the missed detection rate. To this end, an improved depth framework based on unsupervised panoramic image stitching of the gastrointestinal tract is proposed. In addition, preprocessing for aberration correction of monocular endoscope images is introduced, and a C2f module is added to the image reconstruction network to improve the network's ability to extract features. A comprehensive real image data set, GASE-Dataset, is proposed to establish an evaluation benchmark and training learning framework for unsupervised deep gastrointestinal image splicing. Experimental results show that the MSE, RMSE, PSNR, SSIM and RMSE_SW indicators are improved, while the splicing time remains within an acceptable range. Compared with traditional image stitching methods, the performance of this method is enhanced. In addition, improvements are proposed to address the problems of lack of annotated data, insufficient generalization ability and insufficient comprehensive performance in image stitching schemes based on supervised learning. These improvements provide valuable aids in gastrointestinal examination.

Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane.

Intraoperative imaging techniques for reconstructing deformable tissues in vivo are pivotal for advanced surgical systems. Existing methods either compromise on rendering quality or are excessively computationally intensive, often demanding dozens of hours to perform, which significantly hinders their practical application. In this paper, we introduce Fast Orthogonal Plane (Forplane), a novel, efficient framework based on neural radiance fields (NeRF) for the reconstruction of deformable tissues. We conceptualize surgical procedures as 4D volumes, and break them down into static and dynamic fields comprised of orthogonal neural planes. This factorization discretizes the four-dimensional space, leading to a decreased memory usage and faster optimization. A spatiotemporal importance sampling scheme is introduced to improve performance in regions with tool occlusion as well as large motions and accelerate training. An efficient ray marching method is applied to skip sampling among empty regions, significantly improving inference speed. Forplane accommodates both binocular and monocular endoscopy videos, demonstrating its extensive applicability and flexibility. Our experiments, carried out on two in vivo datasets, the EndoNeRF and Hamlyn datasets, demonstrate the effectiveness of our framework. In all cases, Forplane substantially accelerates both the optimization process (by over 100 times) and the inference process (by over 15 times) while maintaining or even improving the quality across a variety of non-rigid deformations. This significant performance improvement promises to be a valuable asset for future intraoperative surgical applications. The code of our project is now available at https://github.com/Loping151/ForPlane.

Depth estimation from monocular endoscopy using simulation and image transfer approach

Obtaining accurate distance or depth information in endoscopy is crucial for the effective utilization of navigation systems. However, due to space constraints, incorporating depth cameras into endoscopic systems is often impractical. Our goal is to estimate depth images directly from endoscopic images using deep learning. This study presents a three-step methodology for training a depth-estimation network model. Initially, simulated endoscopy images and corresponding depth maps are generated using Unity based on a colon surface model obtained from segmented computed tomography colonography data. Subsequently, a cycle generative adversarial network model is employed to enhance the realism of the simulated endoscopy images. Finally, a deep learning model is trained using the synthesized endoscopy images and depth maps to estimate depths accurately. The performance of the proposed approach is evaluated and compared against prior studies utilizing unsupervised training methods. The results demonstrate the superior precision of the proposed technique in estimating depth images within endoscopy. The proposed depth estimation method holds promise for advancing the field by enabling enhanced navigation, improved lesion marking capabilities, and ultimately leading to better clinical outcomes.

Taking measurement in every direction: Implicit scene representation for accurately estimating target dimensions under monocular endoscope

Background and objectives:In endoscopy, measurement of target size can assist medical diagnosis. However, limited operating space, low image quality, and irregular target shape pose great challenges to traditional vision-based measurement methods. Methods:In this paper, we propose a novel approach to measure irregular target size under monocular endoscope using image rendering. Firstly synthesize virtual poses on the same main optical axis as known camera poses, and use implicit neural representation module that considers brightness and target boundaries to render images corresponding to virtual poses. Then, Swin-Unet and rotating calipers are utilized to obtain maximum pixel length of the target in image pairs with the same main optical axis. Finally, the similarity triangle relationship of the endoscopic imaging model is used to measure the size of the target. Results:The evaluation is conducted using renal stone fragments of patients which are placed in the kidney model and the isolated porcine kidney. The mean error of measurement is 0.12 mm. Conclusions:The approached method can automatically measure object size within narrow body cavities in any visible direction. It improves the effectiveness and accuracy of measurement in limited endoscopic space.

Non-rigid scene reconstruction of deformable soft tissue with monocular endoscopy in minimally invasive surgery.

The utilization of image-guided surgery has demonstrated its ability to improve the precision and safety of minimally invasive surgery (MIS). Non-rigid scene reconstruction is a challenge in image-guided system duo to uniform texture, smoke, and instrument occlusion, etc. METHODS: In this paper, we introduced an algorithm for 3D reconstruction aimed at non-rigid surgery scenes. The proposed method comprises two main components: firstly, the front-end process involves the initial reconstruction of 3D information for deformable soft tissues using embedded deformation graph (EDG) on the basis of dual quaternions, enabling the reconstruction without the need for prior knowledge of the target. Secondly, the EDG is integrated with isometric nonrigid structure from motion (Iso-NRSFM) to facilitate centralized optimization of the observed map points and camera motion across different time instances in deformable scenes. For the quantitative evaluation of the proposed method, we conducted comparative experiments with both synthetic datasets and publicly available datasets against the state-of-the-art 3D reconstruction method, DefSLAM. The test results show that our proposed method achieved a maximum reduction of 1.6mm in average reconstruction error compared to method DefSLAM across all datasets. Additionally, qualitative experiments were performed on video scene datasets involving surgical instrument occlusions. Our method proved to outperform DefSLAM on both synthetic datasets and public datasets through experiments, demonstrating its robustness and accuracy in the reconstruction of soft tissues in dynamic surgical scenes. This success highlights the potential clinical application of our method in delivering surgeons with critical shape and depth information for MIS.

Image Intrinsic-Based Unsupervised Monocular Depth Estimation in Endoscopy.

Unsupervised monocular depth estimation plays a vital role for endoscopy-based minimally invasive surgery (MIS). However, it remains challenging due to the distinctive imaging characteristics of endoscopy which disrupt the assumption of photometric consistency, a foundation relied upon by conventional methods. Distinct from recent approaches taking image pre-processing strategy, this paper introduces a pioneering solution through intrinsic image decomposition (IID) theory. Specifically, we propose a novel end-to-end intrinsic-based unsupervised monocular depth learning framework that is comprised of an image intrinsic decomposition module and a synthesis reconstruction module. This framework seamlessly integrates IID with unsupervised monocular depth estimation, and dedicated losses are meticulously designed to offer robust supervision for network training based on this novel integration. Noteworthy, we rely on the favorable property of the resulting albedo map of IID to circumvent the challenging images characteristics instead of pre-processing the input frames. The proposed method is extensively validated on SCARED and Hamlyn datasets, and better results are obtained than state-of-the-art techniques. Beside, its generalization ability and the effectiveness of the proposed components are also validated. This innovative method has the potential to elevate the quality of 3D reconstruction in monocular endoscopy, thereby enhancing the accuracy and robustness of augmented reality navigation technology in MIS. Our code will be available at: https://github.com/bobo909/IID-SfmLearner.

Multisource Differential Fusion Driven Monocular Endoscope Hybrid 3-D Tracking for Advanced Endoscopic Navigation Surgery.

Surgical navigation systems involve various technologies of segmentation, calibration, registration, tracking, and visualization. These systems aim to superimpose multisource information in the surgical field and provide surgeons with a composite overlay (augmented-reality) view, improving the operative precision and experience. Surgical 3-D tracking is the key to build these systems. Unfortunately, surgical 3-D tracking is still a challenge to endoscopic and robotic navigation systems and easily gets trapped in image artifacts, tissue deformation, and inaccurate positional (e.g., electromagnetic) sensor measurements. This work explores a new monocular endoscope hybrid 3-D tracking method called spatially constrained adaptive differential evolution that combines two spatial constraints with observation-recall adaptive propagation and observation-based fitness computing for stochastic optimization. Specifically, we spatially constraint inaccurate electromagnetic sensor measurements to the centerline of anatomical tubular structures to keep them physically locating inside the tubes, as well as interpolate these measurements to reduce jitter errors for smooth 3-D tracking. We then propose observation-recall adaptive propagation with fitness computing to precisely fuse the constrained sensor measurements, preoperative images, and endoscopic video sequences for accurate hybrid 3-D tracking. Additionally, we also propose a new marker-free hybrid registration strategy to precisely align positional sensor measurements to preoperative images. Our new framework was evaluated on a large amount of clinical data acquired from various surgical endoscopic procedures, with the experimental results showing that it certainly outperforms current surgical 3-D approaches. In particular, the position and rotation errors were significantly reduced from (6.55, 11.4) to (3.02 mm, 8.54 °).

Pose estimation via structure-depth information from monocular endoscopy images sequence.

Image-based endoscopy pose estimation has been shown to significantly improve the visualization and accuracy of minimally invasive surgery (MIS). This paper proposes a method for pose estimation based on structure-depth information from a monocular endoscopy image sequence. Firstly, the initial frame location is constrained using the image structure difference (ISD) network. Secondly, endoscopy image depth information is used to estimate the pose of sequence frames. Finally, adaptive boundary constraints are used to optimize continuous frame endoscopy pose estimation, resulting in more accurate intraoperative endoscopy pose estimation. Evaluations were conducted on publicly available datasets, with the pose estimation error in bronchoscopy and colonoscopy datasets reaching 1.43 mm and 3.64 mm, respectively. These results meet the real-time requirements of various scenarios, demonstrating the capability of this method to generate reliable pose estimation results for endoscopy images and its meaningful applications in clinical practice. This method enables accurate localization of endoscopy images during surgery, assisting physicians in performing safer and more effective procedures.

Monocular endoscopy images depth estimation with multi-scale residual fusion

BackgroundMonocular depth estimation plays a fundamental role in clinical endoscopy surgery. However, the coherent illumination, smooth surfaces, and texture-less nature of endoscopy images present significant challenges to traditional depth estimation methods. Existing approaches struggle to accurately perceive depth in such settings. MethodTo overcome these challenges, this paper proposes a novel multi-scale residual fusion method for estimating the depth of monocular endoscopy images. Specifically, we address the issue of coherent illumination by leveraging image frequency domain component space transformation, thereby enhancing the stability of the scene's light source. Moreover, we employ an image radiation intensity attenuation model to estimate the initial depth map. Finally, to refine the accuracy of depth estimation, we utilize a multi-scale residual fusion optimization technique. ResultsTo evaluate the performance of our proposed method, extensive experiments were conducted on public datasets. The structural similarity measures for continuous frames in three distinct clinical data scenes reached impressive values of 0.94, 0.82, and 0.84, respectively. These results demonstrate the effectiveness of our approach in capturing the intricate details of endoscopy images. Furthermore, the depth estimation accuracy achieved remarkable levels of 89.3 % and 91.2 % for the two models’ data, respectively, underscoring the robustness of our method. ConclusionsOverall, the promising results obtained on public datasets highlight the significant potential of our method for clinical applications, facilitating reliable depth estimation and enhancing the quality of endoscopy surgical procedures.

Scale-preserving shape reconstruction from monocular endoscope image sequences by supervised depth learning.

Reconstructing 3D shapes from images are becoming popular, but such methods usually estimate relative depth maps with ambiguous scales. A method for reconstructing a scale-preserving 3D shape from monocular endoscope image sequences through training an absolute depth prediction network is proposed. First, a dataset of synchronized sequences of RGB images and depth maps is created using an endoscope simulator. Then, a supervised depth prediction network is trained that estimates a depth map from a RGB image minimizing the loss compared to the ground-truth depth map. The predicted depth map sequence is aligned to reconstruct a 3D shape. Finally, the proposed method is applied to a real endoscope imagesequence.

Monocular endoscope 6-DoF tracking with constrained evolutionary stochastic filtering.

Monocular endoscopic 6-DoF camera tracking plays a vital role in surgical navigation that involves multimodal images to build augmented or virtual reality surgery. Such a 6-DoF camera tracking generally can be formulated as a nonlinear optimization problem. To resolve this nonlinear problem, this work proposes a new pipeline of constrained evolutionary stochastic filtering that originally introduces spatial constraints and evolutionary stochastic diffusion to deal with particle degeneracy and impoverishment in current stochastic filtering methods. With its application to endoscope 6-DoF tracking and validation on clinical data including more than 59,000 endoscopic video frames acquired from various surgical procedures, the experimental results demonstrate the effectiveness of the new pipeline that works much better than state-of-the-art tracking methods. In particular, it can significantly improve the accuracy of current monocular endoscope tracking approaches from (4.83 mm, 10.2∘) to (2.78 mm, 7.44∘).

Object dimension measurement based on monocular endoscope and 5-DOF sensor

BackgroundDuring endoscopy, dimension measurement of the object is quintessential in diagnosis, e.g., the necessity and extent of further kidney stone fragmentation. The monocular endoscope lacks enough visual cues to measure accurately in the narrow cavity. MethodsFirstly, a tiny five-degree-of-freedom (5-DOF) sensor is embedded into the tip of the endoscope to obtain the pose of the lens. Then, a semantic segmentation network is designed to segment the object masks automatically. Finally, the masks' dimension and the lens's pose are fused to compute the object dimension in physical units, e.g., mm. ResultsTo mimic an endoscopic scenario, the authors placed kidney stone fragments of known size in the isolated porcine kidney. The mean absolute percentage error (MAPE) of measurement was 7.30%. ConclusionThe proposed approach is safe and feasible for endoscopy. It enables automatic object dimension measurements in the limited cavity without changing existing endoscopy procedures.

Joint estimation of depth and motion from a monocular endoscopy image sequence using a multi-loss rebalancing network.

Building an in vivo three-dimensional (3D) surface model from a monocular endoscopy is an effective technology to improve the intuitiveness and precision of clinical laparoscopic surgery. This paper proposes a multi-loss rebalancing-based method for joint estimation of depth and motion from a monocular endoscopy image sequence. The feature descriptors are used to provide monitoring signals for the depth estimation network and motion estimation network. The epipolar constraints of the sequence frame is considered in the neighborhood spatial information by depth estimation network to enhance the accuracy of depth estimation. The reprojection information of depth estimation is used to reconstruct the camera motion by motion estimation network with a multi-view relative pose fusion mechanism. The relative response loss, feature consistency loss, and epipolar consistency loss function are defined to improve the robustness and accuracy of the proposed unsupervised learning-based method. Evaluations are implemented on public datasets. The error of motion estimation in three scenes decreased by 42.1%,53.6%, and 50.2%, respectively. And the average error of 3D reconstruction is 6.456 ± 1.798mm. This demonstrates its capability to generate reliable depth estimation and trajectory reconstruction results for endoscopy images and meaningful applications in clinical.

Object Dimension Measurement Based on Monocular Endoscope and 5-Dof Sensor

Object Dimension Measurement Based on Monocular Endoscope and 5-Dof Sensor

Depth estimation from single-shot monocular endoscope image using image domain adaptation and edge-aware depth estimation

ABSTRACT We propose a depth estimation method from a single-shot monocular endoscopic image using Lambertian surface translation by domain adaptation and depth estimation using multi-scale edge loss. We employ a two-step estimation process including Lambertian surface translation from unpaired data and depth estimation. The texture and specular reflection on the surface of an organ reduce the accuracy of depth estimations. We apply Lambertian surface translation to an endoscopic image to remove these texture and reflections. Then, we estimate the depth by using a fully convolutional network (FCN). During the training of the FCN, improvement of the object edge similarity between an estimated image and a ground truth depth image is important for getting better results. We introduced a muti-scale edge loss function to improve the accuracy of depth estimation. We quantitatively evaluated the proposed method using real colonoscopic images. The estimated depth values were proportional to the real depth values. Furthermore, we applied the estimated depth images to automated anatomical location identification of colonoscopic images using a convolutional neural network. The identification accuracy of the network improved from 69.2% to 74.1% by using the estimated depth images.

Learning-Based Depth and Pose Estimation for Monocular Endoscope with Loss Generalization.

Gastroendoscopy has been a clinical standard for diagnosing and treating conditions that affect a part of a patient's digestive system, such as the stomach. Despite the fact that gastroendoscopy has a lot of advantages for patients, there exist some challenges for practitioners, such as the lack of 3D perception, including the depth and the endoscope pose information. Such challenges make navigating the endoscope and localizing any found lesion in a digestive tract difficult. To tackle these problems, deep learning-based approaches have been proposed to provide monocular gastroendoscopy with additional yet important depth and pose information. In this paper, we propose a novel supervised approach to train depth and pose estimation networks using consecutive endoscopy images to assist the endoscope navigation in the stomach. We firstly generate real depth and pose training data using our previously proposed whole stomach 3D reconstruction pipeline to avoid poor generalization ability between computer-generated (CG) models and real data for the stomach. In addition, we propose a novel generalized photometric loss function to avoid the complicated process of finding proper weights for balancing the depth and the pose loss terms, which is required for existing direct depth and pose supervision approaches. We then experimentally show that our proposed generalized loss performs better than existing direct supervision losses.

An Endoscopic Transcanal Transpromontorial Approach for Vestibular Schwannomas using a Computer-Based Three-Dimensional Imaging System.

A 2D monocular endoscope has been used in transcanal transpromontory vestibular schwannoma surgery instead of craniotomy. However, the absence of depth perception is the limitation of this approach. With the loss of depth perception, the surgeon will be not able to perform delicate and particularly complicated surgery. A binocular endoscope has been developed to provide stereoscopic vision with better depth perception for complicated anatomic structures and has been applied in some endoscopic surgeries. However, the diameter of the endoscope is a limitation in the performance of transcanal otologic surgeries. A small diameter endoscope facilitates easier surgery in a restricted space. A computer-based 3D imaging system can obtain 3D images in real-time using a small monocular endoscope. In this study, to evaluate the feasibility of a computer-based 3D imaging system for endoscopic lateral skull base surgery, we applied this 3D imaging system in a transcanal transpromontorial approach in two patients with vestibular schwannomas. The surgical procedure was completed without complication in these two cases. There was no mortality, perioperative complications, nor notable postoperative complications. Using this computer-based 3D imaging system, a better depth perception and stereoscopic vision was observed compared to a conventional 2D endoscope. The improvement in depth perception offers superior management of the complicated surgical anatomy.

A Small and High-Speed Driving Mechanism for 3D Shape Measurement in Monocular Endoscopy.

Three-dimensional (3D) shape acquisition has been widely introduced to enrich quantitative analysis with the combination of object shape and texture, for example, surface roughness evaluation in industry and gastrointestinal endoscopy in medicine. Shape from focus is a promising technique to measure substance surfaces in 3D space because no occlusion problem appears in principle, as does with stereo shape measurement, which is another commonly used option. We have been developing endoscopic shape measurement devices and shape reconstruction algorithms. In this paper, we propose a mechanism for driving an image sensor reciprocated for the shape from focus of 3D shape measurement in monocular endoscopy. It uses a stepping motor and a planar-end cam, which transforms the motor rotation to imaging sensor reciprocation, to implement the shape from focus of 3D shape measurement in endoscopy. We test and discuss the device in terms of its driving accuracy and application feasibility for endoscopic 3D shape measurement.

Self-supervised Dense Depth Prediction in Monocular Endoscope Video for 3D Liver Surface Reconstruction

In this paper, we propose a self-supervised monocular depth prediction method which combines traditional multi-view stereo method and fully convolutional network to predict the depth map in monocular endoscopic video and achieve 3D dense reconstruction of liver surface. We adopt the sparse data generated by COLMAP supervision signal as the training data, and integrate the attention model in the fully convolutional network to effectively extract the channel features to improve the accuracy of depth prediction. Taking into account the problem of insufficient supervision ability of sparse data, the projection transformation of two images within a certain range is carried out to make up for the missing supervision points. Experimental results show that this method has achieved good results in the depth prediction of monocular endoscopic video and has good applicability to the whole liver.

Cost-Efficient Video Synthesis and Evaluation for Development of Virtual 3D Endoscopy.

Objective: 3D reconstruction of the shape and texture of hollow organs captured by endoscopy is important for the diagnosis and surveillance of early and recurrent cancers. Better evaluation of 3D reconstruction pipelines developed for such applications requires easy access to extensive datasets and associated ground truths, cost-efficient and scalable simulations of a range of possible clinical scenarios, and more reliable and insightful metrics to assess performance. Methods: We present a computer-aided simulation platform for cost-effective synthesis of monocular endoscope videos and corresponding ground truths that mimic a range of potential settings and situations one might encounter during acquisition of clinical endoscopy videos. Using cystoscopy of the bladder as model case, we generated an extensive dataset comprising several synthesized videos of a bladder phantom. We then introduce a novel evaluation procedure to reliably assess an individual 3D reconstruction pipeline or to compare different pipelines. Results: To illustrate the use of the proposed platform and evaluation procedure, we use the aforementioned dataset and ground truths to evaluate a proprietary 3D reconstruction pipeline (CYSTO3D) for bladder cystoscopy videos and compared it with a general-purpose 3D reconstruction pipeline (COLMAP). The evaluation results provide insight into the suggested clinical acquisition protocol and several potential areas for refinement of the pipeline to improve future performance. Conclusion: Our work proposes an endoscope video synthesis and reconstruction evaluation toolset and presents experimental results that illustrate usage of the toolset to efficiently assess performance and reveal possible problems of any given 3D reconstruction pipeline, to compare different pipelines, and to provide technically or clinically actionable insights.