Laparoscopic Images Research Articles

Towards Liver Segmentation in Laparoscopic Images by Training U-Net With Synthetic Data

Abstract The lack of labeled, intraoperative patient data in medical scenarios poses a relevant challenge for machine learning applications. Given the apparent power of machine learning, this study examines how synthetically-generated data can help to reduce the amount of clinical data needed for robust liver surface segmentation in laparoscopic images. Here, we report the results of three experiments, using 525 annotated clinical images from 5 patients alongside 20,000 synthetic photo-realistic images from 10 patient models. The effectiveness of the use of synthetic data is compared to the use of data augmentation, a traditional performance-enhancing technique. For training, a supervised approach employing the U-Net architecture was chosen. The results of these experiments show a progressive increase in accuracy. Our base experiment on clinical data yielded an F1 score of 0.72. Applying data augmentation to this model increased the F1 score to 0.76. Our model pre-trained on synthetic data and fine-tuned with augmented data achieved an F1 score of 0.80, a 4% increase. Additionally, a model evaluation involving k-fold cross validation highlighted the dependency of the result on the test set. These results demonstrate that leveraging synthetic data has the ability of limiting the need for more patient data to increase the segmentation performance.

Visual-Numeric Endometriosis Scoring System (VNESS) for mapping surgical findings: A validation study

Background: Several endometriosis classification systems have been proposed and published but the search for a universal language that communicates the complexity, laterality and severity of this disease continues. The authors introduce the Visual-Numeric Endometriosis Scoring System. VNESS is a novel system for describing surgical findings in each compartment of the pelvis in a way that is simple to use, visually intuitive and mirrors a laparoscopic image of the pelvis. Objective: The aim of this study was to assess inter-rater reliability for components of VNESS. Materials and Methods: The project took the format of a validation study using short surgical laparoscopic video clips. Anonymised video clips of endometriosis procedures were scored by 50 Gynaecologists of varying levels of experience from 12 different countries. The clips were collated from a series of procedures performed between 2012 and 2022. Each participant scored 93 short surgical clips using VNESS. 4650 scores were compared against a reference score and analysis was performed to assess inter-rater reliability. Main outcome measures: The outcome measures were percentage agreement between given and reference scores, as well as intra-class correlation coefficients (ICC), Cohen Kappa and Quadratic Weighted Kappa Coefficients calculated to evaluate inter-rater reliability. Results: The highest and lowest percentage agreement with the reference score was seen in VNESS 4 (full thickness disease, 97% perfect agreement) and VNESS 1 (superficial disease, 53% perfect agreement) respectively. The intraclass correlation coefficient showed strong inter-rater reliability for all VNESS compartments except the vagina. Conclusions: This study suggests that VNESS has excellent reliability between observers. Correlation is stronger with more severe disease.

Laparoscopic stereo matching using 3-Dimensional Fourier transform with full multi-scale features

3-Dimensional (3D) reconstruction of laparoscopic surgical scenes is a key task for future surgical navigation and automated robotic minimally invasive surgery. Binocular laparoscopy with stereo matching enables 3D reconstruction. Stereo matching models used for natural images such as autopilot tend to be less suitable for laparoscopic environments due to the constraints of small samples of laparoscopic images, complex textures, and uneven illumination. In addition, current stereo matching modules use 3D convolutions and transformers in the spatial domain as the base module, which is limited by the ability to learn in the spatial domain. In this paper, we propose a model for laparoscopic stereo matching using 3D Fourier Transform combined with Full Multi-scale Features (FT-FMF Net). Specifically, the proposed Full Multi-scale Fusion Module (FMFM) is able to fuse the full multi-scale feature information from the feature extractor into the stereo matching block, which densely learns the feature information with parallax and FMFM fusion information in the frequency domain using the proposed Dense Fourier Transform Module (DFTM). We validated the proposed method in both the laparoscopic dataset (SCARED) and the endoscopic dataset (SERV-CT). In comparison with other popular and advanced deep learning models available at present, FT-FMF Net achieves the most advanced stereo matching performance available. In the SCARED and SERV-CT public datasets, the End-Point-Error (EPE) was 0.7265 and 2.3119, and the Root Mean Square Error Depth (RMSE Depth) was 4.00 mm and 3.69 mm, respectively. In addition, the inference time is only 0.17s. Our project code is available on https://github.com/wurenkai/FT-FMF.

An objective comparison of methods for augmented reality in laparoscopic liver resection by preoperative-to-intraoperative image fusion from the MICCAI2022 challenge

Augmented reality for laparoscopic liver resection is a visualisation mode that allows a surgeon to localise tumours and vessels embedded within the liver by projecting them on top of a laparoscopic image. Preoperative 3D models extracted from Computed Tomography (CT) or Magnetic Resonance (MR) imaging data are registered to the intraoperative laparoscopic images during this process. Regarding 3D–2D fusion, most algorithms use anatomical landmarks to guide registration, such as the liver’s inferior ridge, the falciform ligament, and the occluding contours. These are usually marked by hand in both the laparoscopic image and the 3D model, which is time-consuming and prone to error. Therefore, there is a need to automate this process so that augmented reality can be used effectively in the operating room. We present the Preoperative-to-Intraoperative Laparoscopic Fusion challenge (P2ILF), held during the Medical Image Computing and Computer Assisted Intervention (MICCAI 2022) conference, which investigates the possibilities of detecting these landmarks automatically and using them in registration. The challenge was divided into two tasks: (1) A 2D and 3D landmark segmentation task and (2) a 3D–2D registration task. The teams were provided with training data consisting of 167 laparoscopic images and 9 preoperative 3D models from 9 patients, with the corresponding 2D and 3D landmark annotations. A total of 6 teams from 4 countries participated in the challenge, whose results were assessed for each task independently. All the teams proposed deep learning-based methods for the 2D and 3D landmark segmentation tasks and differentiable rendering-based methods for the registration task. The proposed methods were evaluated on 16 test images and 2 preoperative 3D models from 2 patients. In Task 1, the teams were able to segment most of the 2D landmarks, while the 3D landmarks showed to be more challenging to segment. In Task 2, only one team obtained acceptable qualitative and quantitative registration results. Based on the experimental outcomes, we propose three key hypotheses that determine current limitations and future directions for research in this domain.

Navigating the Future of 3D Laparoscopic Liver Surgeries: Visualization of Internal Anatomy on Laparoscopic Images With Augmented Reality.

Liver tumor resection requires precise localization of tumors and blood vessels. Despite advancements in 3-dimensional (3D) visualization for laparoscopic surgeries, challenges persist. We developed and evaluated an augmented reality (AR) system that overlays preoperative 3D models onto laparoscopic images, offering crucial support for 3D visualization during laparoscopic liver surgeries. Anatomic liver structures from preoperative computed tomography scans were segmented using open-source software including 3D Slicer and Maya 2022 for 3D model editing. A registration system was created with 3D visualization software utilizing a stereo registration input system to overlay the virtual liver onto laparoscopic images during surgical procedures. A controller was customized using a modified keyboard to facilitate manual alignment of the virtual liver with the laparoscopic image. The AR system was evaluated by 3 experienced surgeons who performed manual registration for a total of 27 images from 7 clinical cases. The evaluation criteria included registration time; measured in minutes, and accuracy; measured using the Dice similarity coefficient. The overall mean registration time was 2.4±1.7 minutes (range: 0.3 to 9.5min), and the overall mean registration accuracy was 93.8%±4.9% (range: 80.9% to 99.7%). Our validated AR system has the potential to effectively enable the prediction of internal hepatic anatomic structures during 3D laparoscopic liver resection, and may enhance 3D visualization for select laparoscopic liver surgeries.

Fast high quality highlight removal using a simplified matrix iteration method

Recently, a global non-smooth optimization method was developed for efficiently eliminating specular highlights from single color image. Although this method was reported to be effective on both natural and medical images, its algorithm’s speed is slow due to the non-smoothness of the objective function. This paper proposes a global smooth optimization method, where two gradient regularization terms are used to describe diffuse and specular component features for texture details. A simplified optimality condition is established and thus a new iteration algorithm for specular highlight removal is proposed. By employing the gradient regularization of diffuse reflection, image texture detail can be preserved during specular highlight removal. Due to the smoothness of the objective function, the proposed smooth optimization algorithm has lower computational complexity and less storage requirement than the global non-smooth optimization algorithm. Furthermore, the proposed iteration algorithm is guaranteed to converge to a global optimal solution under a fixed step length. Experimental results demonstrate that the proposed smooth optimization algorithm is much faster than the non-smooth optimization algorithm. Moreover, the proposed highlight removal algorithm is more effective on benchmark natural images and real laparoscopic images than conventional highlight removal algorithms.

Correction Compensation and Adaptive Cost Aggregation for Deep Laparoscopic Stereo Matching

Perception of digitized depth is a prerequisite for enabling the intelligence of three-dimensional (3D) laparoscopic systems. In this context, stereo matching of laparoscopic stereoscopic images presents a promising solution. However, the current research in this field still faces challenges. First, the acquisition of accurate depth labels in a laparoscopic environment proves to be a difficult task. Second, errors in the correction of laparoscopic images are prevalent. Finally, laparoscopic image registration suffers from ill-posed regions such as specular highlights and textureless areas. In this paper, we make significant contributions by developing (1) a correction compensation module to overcome correction errors; (2) an adaptive cost aggregation module to improve prediction performance in ill-posed regions; (3) a novel self-supervised stereo matching framework based on these two modules. Specifically, our framework rectifies features and images based on learned pixel offsets, and performs differentiated aggregation on cost volumes based on their value. The experimental results demonstrate the effectiveness of the proposed modules. On the SCARED dataset, our model reduces the mean depth error by 12.6% compared to the baseline model and outperforms the state-of-the-art unsupervised methods and well-generalized models.

Extra-abdominal trocar and instrument detection for enhanced surgical workflow understanding.

Video-based intra-abdominal instrument tracking for laparoscopic surgeries is a common research area. However, the tracking can only be done with instruments that are actually visible in the laparoscopic image. By using extra-abdominal cameras to detect trocars and classify their occupancy state, additional information about the instrument location, whether an instrument is still in the abdomen or not, can be obtained. This can enhance laparoscopic workflow understanding and enrich already existing intra-abdominal solutions. A data set of four laparoscopic surgeries recorded with two time-synchronized extra-abdominal 2D cameras was generated. The preprocessed and annotated data were used to train a deep learning-based network architecture consisting of a trocar detection, a centroid tracker and a temporal model to provide the occupancy state of all trocars during the surgery. The trocar detection model achieves an F1 score of . The prediction of the occupancy state yields an F1 score of , providing a first step towards enhanced surgical workflow understanding. The current method shows promising results for the extra-abdominal tracking of trocars and their occupancy state. Future advancements include the enlargement of the data set and incorporation of intra-abdominal imaging to facilitate accurate assignment of instruments to trocars.

Improving Surgical Scene Semantic Segmentation through a Deep Learning Architecture with Attention to Class Imbalance.

This article addresses the semantic segmentation of laparoscopic surgery images, placing special emphasis on the segmentation of structures with a smaller number of observations. As a result of this study, adjustment parameters are proposed for deep neural network architectures, enabling a robust segmentation of all structures in the surgical scene. The U-Net architecture with five encoder-decoders (U-Net5ed), SegNet-VGG19, and DeepLabv3+ employing different backbones are implemented. Three main experiments are conducted, working with Rectified Linear Unit (ReLU), Gaussian Error Linear Unit (GELU), and Swish activation functions. The applied loss functions include Cross Entropy (CE), Focal Loss (FL), Tversky Loss (TL), Dice Loss (DiL), Cross Entropy Dice Loss (CEDL), and Cross Entropy Tversky Loss (CETL). The performance of Stochastic Gradient Descent with momentum (SGDM) and Adaptive Moment Estimation (Adam) optimizers is compared. It is qualitatively and quantitatively confirmed that DeepLabv3+ and U-Net5ed architectures yield the best results. The DeepLabv3+ architecture with the ResNet-50 backbone, Swish activation function, and CETL loss function reports a Mean Accuracy (MAcc) of 0.976 and Mean Intersection over Union (MIoU) of 0.977. The semantic segmentation of structures with a smaller number of observations, such as the hepatic vein, cystic duct, Liver Ligament, and blood, verifies that the obtained results are very competitive and promising compared to the consulted literature. The proposed selected parameters were validated in the YOLOv9 architecture, which showed an improvement in semantic segmentation compared to the results obtained with the original architecture.

Advancements in Deep Learning for Minimally Invasive Surgery: A Journey through Surgical System Evolution

The surge in artificial intelligence (AI) applications across diverse fields owes much to advancements in deep learning and computational processing speed. In medicine, AI's reach extends to medical image analysis and genomic data interpretation. More recently, AI's role in analyzing minimally invasive surgery (MIS) videos has gained traction, with a growing body of research focusing on organ and anatomy identification, instrument recognition, procedure recognition, surgical phase delineation, surgery duration prediction, optimal incision line identification, and surgical education. Concurrently, the development of autonomous surgical robots, exemplified by the Smart Tissue Autonomous Robot (STAR) and RAVEN systems, has shown promising strides. Notably, STAR is currently employed in laparoscopic imaging to discern the surgical site from laparoscopic images and is undergoing trials for an automated suturing system, albeit in animal models. This review contemplates the prospect of fully autonomous surgical robots in the future.

Advancements in Deep Learning for Minimally Invasive Surgery: A Journey through Surgical System Evolution

The surge in artificial intelligence (AI) applications across diverse fields owes much to advancements in deep learning and computational processing speed. In medicine, AI's reach extends to medical image analysis and genomic data interpretation. More recently, AI's role in analyzing minimally invasive surgery (MIS) videos has gained traction, with a growing body of research focusing on organ and anatomy identification, instrument recognition, procedure recognition, surgical phase delineation, surgery duration prediction, optimal incision line identification, and surgical education. Concurrently, the development of autonomous surgical robots, exemplified by the Smart Tissue Autonomous Robot (STAR) and RAVEN systems, has shown promising strides. Notably, STAR is currently employed in laparoscopic imaging to discern the surgical site from laparoscopic images and is undergoing trials for an automated suturing system, albeit in animal models. This review contemplates the prospect of fully autonomous surgical robots in the future.

Distinguishing the Uterine Artery, the Ureter, and Nerves in Laparoscopic Surgical Images Using Ensembles of Binary Semantic Segmentation Networks.

Performing a minimally invasive surgery comes with a significant advantage regarding rehabilitating the patient after the operation. But it also causes difficulties, mainly for the surgeon or expert who performs the surgical intervention, since only visual information is available and they cannot use their tactile senses during keyhole surgeries. This is the case with laparoscopic hysterectomy since some organs are also difficult to distinguish based on visual information, making laparoscope-based hysterectomy challenging. In this paper, we propose a solution based on semantic segmentation, which can create pixel-accurate predictions of surgical images and differentiate the uterine arteries, ureters, and nerves. We trained three binary semantic segmentation models based on the U-Net architecture with the EfficientNet-b3 encoder; then, we developed two ensemble techniques that enhanced the segmentation performance. Our pixel-wise ensemble examines the segmentation map of the binary networks on the lowest level of pixels. The other algorithm developed is a region-based ensemble technique that takes this examination to a higher level and makes the ensemble based on every connected component detected by the binary segmentation networks. We also introduced and trained a classic multi-class semantic segmentation model as a reference and compared it to the ensemble-based approaches. We used 586 manually annotated images from 38 surgical videos for this research and published this dataset.

LARLUS: laparoscopic augmented reality from laparoscopic ultrasound.

This research endeavors to improve tumor localization in minimally invasive surgeries, a challenging task primarily attributable to the absence of tactile feedback and limited visibility. The conventional solution uses laparoscopic ultrasound (LUS) which has a long learning curve and is operator-dependent. The proposed approach involves augmenting LUS images onto laparoscopic images to improve the surgeon's ability to estimate tumor and internal organ anatomy. This augmentation relies on LUS pose estimation and filtering. Experiments conducted with clinical data exhibit successful outcomes in both the registration and augmentation of LUS images onto laparoscopic images. Additionally, noteworthy results are observed in filtering, leading to reduced flickering in augmentations. The outcomes reveal promising results, suggesting the potential of LUS augmentation in surgical images to assist surgeons and serve as a training tool. We have used the LUS probe's shaft to disambiguate the rotational symmetry. However, in the long run, it would be desirable to find more convenient solutions.

Vision-based estimation of manipulation forces by deep learning of laparoscopic surgical images obtained in a porcine excised kidney experiment

In robot-assisted surgery, in which haptics should be absent, surgeons experience haptics-like sensations as “pseudo-haptic feedback”. As surgeons who routinely perform robot-assisted laparoscopic surgery, we wondered if we could make these “pseudo-haptics” explicit to surgeons. Therefore, we created a simulation model that estimates manipulation forces using only visual images in surgery. This study aimed to achieve vision-based estimations of the magnitude of forces during forceps manipulation of organs. We also attempted to detect over-force, exceeding the threshold of safe manipulation. We created a sensor forceps that can detect precise pressure at the tips with three vectors. Using an endoscopic system that is used in actual surgery, images of the manipulation of excised pig kidneys were recorded with synchronized force data. A force estimation model was then created using deep learning. Effective detection of over-force was achieved if the region of the visual images was restricted by the region of interest around the tips of the forceps. In this paper, we emphasize the importance of limiting the region of interest in vision-based force estimation tasks.

Encoding laparoscopic image to words using vision transformer for distortion classification and ranking in laparoscopic videos

AbstractLaparoscopic videos are tools used by surgeons to insert narrow tubes into the abdomen and keep the skin without large incisions. The videos captured by a camera are prone to numerous distortions such as uneven illumination, motion blur, defocus blur, smoke, and noise which have impact on visual quality. Automatic detection and identification of distortions are significant to enhance the quality of laparoscopic videos to avoid errors during surgery. The video quality assessment includes two stages: classification of distortions affecting the video frames to identify their types and ranking of distortions to estimate the intensity levels. The dataset generated in ICIP2020 challenge including laparoscopic videos was utilized for training, validation, and testing the proposed solution. The difficulty of this dataset is caused by having five categories of distortions and four levels of severity. Additionally, the availability of multiple distortion categories in one video is considered the most challenging part of this dataset. The work presented in this paper contributes to solve the multi-label distortion classification and ranking problem. This paper aims to enhance the performance of distortion classification solutions. Vision transformer which is a deep learning model was used to extract informative features by transferring learning and representation from the general domain to the medical domain (laparoscopic videos). Additionally, six parallel multilayer perceptron (MLP) classifiers were added and attached to vision transformer for distortion classification and ranking. The experiment showed that the proposed solution outperforms existing distortion classification methods in terms of average accuracy (89.7%), average single distortion F1 score (94.18%), and average of both single and multiple distortions F1 score (96.86%). Moreover, it can also rank the distortions with an average accuracy of 79.22% and average F1 score of 78.44%. Hence, the high performance of the method proposed in this paper opens the door to integrate our solution in the intelligent video enhancement system.

SwinPA-Net: Swin Transformer-Based Multiscale Feature Pyramid Aggregation Network for Medical Image Segmentation.

The precise segmentation of medical images is one of the key challenges in pathology research and clinical practice. However, many medical image segmentation tasks have problems such as large differences between different types of lesions and similar shapes as well as colors between lesions and surrounding tissues, which seriously affects the improvement of segmentation accuracy. In this article, a novel method called Swin Pyramid Aggregation network (SwinPA-Net) is proposed by combining two designed modules with Swin Transformer to learn more powerful and robust features. The two modules, named dense multiplicative connection (DMC) module and local pyramid attention (LPA) module, are proposed to aggregate the multiscale context information of medical images. The DMC module cascades the multiscale semantic feature information through dense multiplicative feature fusion, which minimizes the interference of shallow background noise to improve the feature expression and solves the problem of excessive variation in lesion size and type. Moreover, the LPA module guides the network to focus on the region of interest by merging the global attention and the local attention, which helps to solve similar problems. The proposed network is evaluated on two public benchmark datasets for polyp segmentation task and skin lesion segmentation task as well as a clinical private dataset for laparoscopic image segmentation task. Compared with existing state-of-the-art (SOTA) methods, the SwinPA-Net achieves the most advanced performance and can outperform the second-best method on the mean Dice score by 1.68%, 0.8%, and 1.2% on the three tasks, respectively.

Multispectral Imaging Method in Laparoscopy

Introduction. At present, video data acquired in narrow spectral bands are widely used to improve the efficiency of diagnostics in various medical fields, laparoscopy in particular. Conventional laparoscopy uses images obtained in the white light. Images obtained in the visible range suitably depict the color and textural features of tissues. However, it is difficult for a physician to use visible images for distinguishing between lesion areas and normal tissues, largely due to their proximity in color and texture. The efficiency of lesion detection can be improved using fluorescence images, which clearly differentiate lesion areas from normal tissues. However, the use multispectral data implies the need to present the images obtained both in the white and fluorescence light to the physician. In this paper, we propose an image composition method based on visible and fluorescence images, which facilitates their analysis by physicians. A distinctive feature of the method is the use of CIEDE 2000 metric for image fusion, which takes the properties of human vision into account.Aim. Development of a method for multispectral data visualization, which provides a physician with an image that combines white light data and a color-highlighted area of lesions.Materials and methods. The proposed method consists of the following steps: preprocessing of images obtained in visible and fluorescence light; segmentation of the lesion area in the fluorescence images; generation of a pseudo-color image of the segmented lesion area; and fusion of the pseudo-color image with the image obtained in the white light.Results. The proposed method forms an image that includes an image of the operation area obtained in the white light and a separated lesion area based on fluorescence information in the near infrared range. The image composite takes the properties of human vision into account. An experimental study of the method was carried out on actual laparoscopic images, involving endoscopists who were experts in subjective evaluation of video images. The method of paired comparisons was used to evaluated the presented images. The majority of experts preferred the fused image formed by the proposed method to those visualized simultaneously in the white and fluorescence light.Conclusion. The developed method ensures generation of images with an increased diagnostic value.

Stereo matching of binocular laparoscopic images with improved densely connected neural architecture search.

Stereo matching is a crucial technology in the binocular laparoscopic-based surgical navigation systems. In recent years, neural networks have been widely applied to stereo matching and demonstrated outstanding performance. however, this method heavily relies on manual feature engineering meaning that professionals must be involved in the feature extraction and matching. This process is both time-consuming and demands specific expertise. This paper introduces a novel stereo matching framework DCStereo that realizes a fully automatic neural architecture design for the stereo matching of binocular laparoscopic images. The proposed framework utilizes a densely connected search space which enables a more flexible and diverse architecture composition. Furthermore, the proposed algorithm leverages the channel and path sampling strategies to reduce memory consumption during searching. Empirically, our searched DCStereo on the SCARED training dataset achieves a mean absolute error of 3.589mm on the test dataset, which outperforms hand-crafted stereo matching methods and other approaches. Furthermore, when directly testing on the SERV-CT dataset, our DCStereo demonstrates better generalization ability than other methods. Our proposed approach leverages the neural architecture search technique and a densely connected search space for automatic neural architecture design in stereo matching of binocular laparoscopic images. Our method delivers advanced performance on the SCARED dataset and promising results on the SERV-CT dataset. These findings demonstrate the potential of our approach for improving clinical surgical navigation systems.

Navigating the future of laparoscopic liver surgeries: visualization of internal anatomy on laparoscopic images with augmented reality

Navigating the future of laparoscopic liver surgeries: visualization of internal anatomy on laparoscopic images with augmented reality

Laparoscopic Tool Classification in Gynaecological Images Using Convolutional Neural Network and Attention Modules

Laparoscopic image analysis plays a crucial role in surgical workflow, as it enables the recognition of surgical tools and anatomical structures. While deep learning approaches demonstrate remarkable recognition performance, they demand large amount of labelled data for training. To date, studies were carried out mainly using data from cholecystectomy procedures due to availability of labelled data. In this work, the scope is extended to more intricate gynaecological surgeries. Laparoscopic videos of gynaecological procedures were recorded and surgical tools labelled. A convolutional neural network (CNN) model incorporated attention modules performed a tool classification task. The base-CNN model achieved a mean average precision of 63% over all tools. Experimental results showed a high classification performance for several surgical tools such as the grasper with an average precision of 93%. Furthermore, results revealed higher performance when attention modules were added to the base-CNN model, resulting in ∼7% improvement in the classification performance over all tools. This study demonstrates the CNN's capability in handling gynaecological images and highlights the potential of incorporating attention modules to enhance the model performance.