Med-3D: 3D Reconstruction of Medical Images based on Structure-from-Motion via Transfer Learning

Abstract

The problem of 3D reconstruction of medical images is very challenging due to the disadvantages of less texture and serious speckle noise. In this paper, we propose a medical image 3D reconstruction framework based on transfer learning and Structure-from-Motion. Unlike some existing methods, which need special hardware devices, or be effective for a particular modal of images, our unified framework does not depend on any special equipment and can be applied to diverse modalities of medical images, such as ultrasonic image, computed tomography (CT) etc. Its contribution covers three aspects: First, it uses Structure-From-Motion(SFM) to estimate the camera structure and pose of consecutive frames end-to-end, which is suitable for clinical auxiliary diagnosis. Second, strong and robust photogrammetry knowledge, including depth, the camera structure and pose, can be pre-learned from the natural images with the richer texture, and then, are transferred to the medical images learning. Besides, Med-3D is proposed, a GAN model based on self-supervised learning framework, in the generator, the spatial coplanar characteristics to constrain the cloud points reconstructed from a single captured frame or adjacent frames are considered for learning more precise features. Experimental results on the medical data of ultrasound images, and CT in the open clinical library, demonstrate that our method can reconstruct geometric structures from 2D slices and has excellent accuracy in terms of reconstruction effects.