Multi-view Stereo Datasets Research Articles

Multiview Textured Mesh Recovery by Differentiable Rendering

Although having achieved the promising results on shape and color recovery through self-supervision, the multi-layer perceptrons-based methods usually suffer from heavy computational cost on learning the deep implicit surface representation. Since rendering each pixel requires a forward network inference, it is very computational intensive to synthesize a whole image. To tackle these challenges, we propose an effective coarse-to-fine approach to recover the textured mesh from multi-views in this paper. Specifically, a differentiable Poisson Solver is employed to represent the object's shape, which is able to produce topology-agnostic and watertight surfaces. To account for depth information, we optimize the shape geometry by minimizing the differences between the rendered mesh and the predicted depth from multi-view stereo. In contrast to the implicit neural representation on shape and color, we introduce a physically based inverse rendering scheme to jointly estimate the environment lighting and object's reflectance, which is able to render the high resolution image at real-time. The texture of the reconstructed mesh is interpolated from a learnable dense texture grid. We have conducted the extensive experiments on several multi-view stereo datasets, whose promising results demonstrate the efficacy of our proposed approach. The code is available at https://github.com/l1346792580123/diff.

High accuracy and geometry-consistent confidence prediction network for multi-view stereo

Confidence prediction task attempts to infer the correctness of estimated depth hypotheseshich has gained popularity recently in stereo matching and boosts the accuracy of disparity estimation. However, less attention is paid on confidence prediction of multi-view stereo (MVS), where multi-view depth estimation is a key step for high-quality reconstruction. In this work, we propose a Geometry-consistent Confidence prediction Network (GeoConfNet), where the correctness of a depth hypothesis is accurately predicted via a deep neural network that explores both spatial coherence and cross-view consistency. The proposed deep network consists of a feature extraction module, a U-Net-based fusion module and a confidence refinement module. Furthermore, we demonstrate that truncated signed distance field (TSDF) is a powerful cross-view feature which can be an effective complement to spatial features, thereby remarkably boosting confidence prediction accuracy of MVS. Exhaustive experiments on a variety of MVS datasets as well as stereo matching datasets clearly demonstrate that our method achieves significantly better performance than state-of-the-art methods in terms of area under the curve (AUC).

Depth-map completion for large indoor scene reconstruction

Traditional Multi View Stereo (MVS) algorithms are often difficult to deal with large-scale indoor scene reconstruction, due to the photo-consistency measurement errors in weak textured regions, which are commonly exist in indoor scenes. To solve this limitation, in this paper we proposed a point cloud completion strategy that combines learning-based depth-map completion and geometry-based consistency filtering to fill large-area missing in depth-maps. The proposed method takes nonuniform and noisy MVS depth-map as input, and completes each depth-map individually. In the completion process, we first complete depth-maps using learning based method, and then filter each depth-map using depth consistency validation with its neighboring depth-maps. This depth-map completion and geometric filtering steps are performed iteratively until the number of depth points is converged. Experiments on large-scale indoor scenes and benchmark MVS datasets demonstrate the effectiveness of the proposed methods.

A Point-Cloud-Based Multiview Stereo Algorithm for Free-Viewpoint Video

This paper presents a robust multiview stereo (MVS) algorithm for free-viewpoint video. Our MVS scheme is totally point-cloud-based and consists of three stages: point cloud extraction, merging, and meshing. To guarantee reconstruction accuracy, point clouds are first extracted according to a stereo matching metric which is robust to noise, occlusion, and lack of texture. Visual hull information, frontier points, and implicit points are then detected and fused with point fidelity information in the merging and meshing steps. All aspects of our method are designed to counteract potential challenges in MVS data sets for accurate and complete model reconstruction. Experimental results demonstrate that our technique produces the most competitive performance among current algorithms under sparse viewpoint setups according to both static and motion MVS data sets.