3D Reconstruction of Laparoscope Images With Contrastive Learning Methods

Nannan Chong

doi:10.1109/access.2022.3140334

Abstract

The use of laparoscopic images and videos to reconstruct abdominal tissue structure overcomes the visual limitations of human eyes and provides great convenience for the detection and diagnosis of medical diseases. The method described in this paper is based on contrast learning and ORB-SLAM design. The contributions of this study are as follows. (i) A data preprocessing thread is introduced, which includes data augmentation and input frame evaluation. (ii) Encoding global information to avoid semantic information loss and eliminate mismatch by designing an improved U-NET network. (iii) The improved U-NET network was used to introduce a dual-branched Siamese network and GPU-accelerated intensive reconstruction thread. The dual-branched Siamese network structure was used to compute the depth information of the feature points and optical flow information in parallel, and the dense depth estimation was obtained without interrupting the original sparse reconstruction. (iv) An experimental system was established to conduct three-dimensional reconstruction tests on laparoscopic/endoscope /UBE videos of 186 patients. To effectively provide more accurate feature detection and matching support for the application of combat scenes, the influence of lens distortion was considered. Compared with the current mainstream three-dimension reconstruction and deep learning algorithms, the practicability and superiority of laparoscopic three-dimension reconstruction based on contrastive learning are demonstrated in clinical scenarios such as surgical navigation, auxiliary diagnosis, and surgical simulation.

Highlights

The emergence and development of laparoscopy breaks through the visual limitations of the human eyes and provides a lot of convenience for the detection and diagnosis of medical diseases
In Minimally Invasive Surgery (MIS) systems, estimating depth information from laparoscopic images, restoring dense 3D information and accurately calculating laparoscopic position is the basis for realizing computer-aided guidance, which is a research hotspot in the medical field
PROPOSED APPROACH This paper proposes a kind of three-dimensional (3D) reconstruction method for abdominal tissue, which is based on the contrast from the supervised learning by laparoscopically obtaining two-dimensional image reconstruction

Summary

INTRODUCTION

The emergence and development of laparoscopy breaks through the visual limitations of the human eyes and provides a lot of convenience for the detection and diagnosis of medical diseases. We introduce a new method based on contrastive self-supervised learning to extract dense features, calculate depth information with stronger robustness to sparse textures, lighting, etc., and perform dense three-dimensional (3D) reconstruction. The researchers used VR in flexible digestive endoscopy by implementing different image sources in the virtual laparoscopy operation room. A dual-branch Siamese network on the basis of the traditional framework of ORB-SLAM, the addition of dense depth estimation network DepthNet and the use of optical flow to estimate camera motion FlowNet, to overcome the inconsistency of color luminance between different frames at the same position in medical images, occlusion and reflection areas in images. Qualitative and quantitative analyses of medical images from 186 patients were performed using modern deep learning methods as well as modern 3d reconstruction of medical images. Cross-patient studies, respectively, to prove the superiority of our method

RELATED WORK

LIVE ALIGNMENT OF KEYFRAME DEPTHMAPS AND BUNDLE ADJUSTMENT

Findings

CONCLUSION