Attention aware cost volume pyramid based multi-view stereo network for 3D reconstruction

Abstract

We present an efficient multi-view stereo (MVS) network for 3D reconstruction from multi-view images. While previous learning based reconstruction approaches performed quite well, most of them estimate depth maps at a fixed resolution using plane sweep volumes with a fixed depth hypothesis at each plane, which requires densely sampled planes for desired accuracy and therefore is difficult to achieve high resolution depth maps. In this paper we introduce a coarse-to-fine depth inference strategy to achieve high resolution depth. This strategy first estimates the depth map at coarsest level, and the depth maps at finer levels are considered as the upsampled depth map from previous level with pixel-wise depth residual. Thus, we narrow the depth searching range with the priori information from previous level and construct new cost volumes from the pixel-wise depth residual to perform depth map refinement. Then the final depth map can be achieved iteratively since all the parameters are shared among different levels. At each level, the self-attention layer is introduced to the feature extraction block for capturing the important information in depth inference task, and the cost volume is generated using similarity measurement instead of the variance based methods used in previous work. Experiments were conducted on three diverse datasets including the DTU benchmark dataset, BlendedMVS dataset and the Tanks and Temples dataset. The results demonstrated that our proposed approach could outperform most state-of-the-arts (SOTA) methods.22The codebase of this project is at https://github.com/ArthasMil/AACVP-MVSNet.