3D Reconstruction and Augmented Reality Display of Bone Structure Based on Ultrasound Segmentation with Feature-Reweighted U-Net

Abstract

<p indent=0mm>During the minimally invasive orthopedic surgery, surgeons insert surgical instruments under the guidance of two-dimensional (2D) X-ray images of bone structures that displayed on a screen. This method not only lacks of intuitive three-dimensional (3D) information, but also exposes patients and surgeons to significant amounts of ionizing radiations. To solve these problems, this paper proposes 3D reconstruction and augmented reality display of bone structure based on Free-hand ultrasound segmentation with feature-reweighted U-Net, and provides doctors with 3D bone structure for radiation-free surgical navigation. Firstly, a feature-reweighted U-Net is developed for accurate segmentation of bone surfaces from Free-hand ultrasound scans. Then, the segmented bone surfaces are reconstructed to acquire 3D bone structure according to the locations of ultrasound images. Finally, the bone structure is further rendered using integral photography techniques for augmented reality guidance. The experiments are finished on ultrasound data set of phantom and patients. The bone segmentation accuracy of (88.51±1.44)% and bone reconstruction error of <sc>(1.29±0.11) mm</sc> are acquired during the experiments. The proposed method can provide surgeons with intuitive intraoperative 3D navigation information.