3D voxel reconstruction from single-view image based on cross-domain feature fusion

Abstract

The single-view 3D voxel reconstruction approach that relies on deep learning is inherently constrained by its single-input nature, which fails to account for the disparities in data distribution between the image and voxel domains. This leads to insufficient model details in the reconstruction. Currently, there is an absence of an end-to-end single-view 3D voxel reconstruction technique that effectively addresses the data distribution variances across domains while streamlining the training and inference processes. To mitigate the challenges arising from the disparities in data distribution across various domains, we introduce a novel single-view 3D voxel reconstruction model that leverages cross-domain feature fusion, termed SV3D-CDFF. SV3D-CDFF utilizes cross-domain feature clustering to mitigate the impact of data distribution disparities and employ a feature supervision method to learns voxel features . Additionally, it incorporates attention mechanism to fuse image and voxel features and utilizes residual network for 3D voxel reconstruction. Quantitative and qualitative experimental results demonstrate that SV3D-CDFF effectively integrates image and voxel features, eliminates data distribution disparities across various domains, and improves the quality of reconstructed voxel models in terms of overall structure and local detail. SV3D-CDFF outperforms existing state-of-the-arts methods in terms of the IoU and F-Score, with improvements of 0.005 and 0.045, respectively.