Comparative Analysis of 3D Reconstruction Results Using MVS and NeRF: Insights from PCA

There are three main approaches for reconstructing 3D models of buildings. Laser scanning is accurate but expensive and limited by the laser's range. Structure-from-motion (SfM) and multi-view stereo (MVS) recover 3D point clouds from multiple views of a building. MVS methods, especially patch-based MVS, can achieve higher density than do SfM methods. Sophisticated algorithms need to be applied to the point clouds to construct mesh surfaces. The recovered point clouds can be sparse in areas that lack features for accurate reconstruction, making recovery of complete surfaces difficult. Moreover, segmentation of the building's surfaces from surrounding surfaces almost always requires some form of manual inputs, diminishing the ease of practical application of automatic 3D reconstruction algorithms. This paper presents an alternative approach for reconstructing textured mesh surfaces from point cloud recovered by patch-based MVS method. To a good first approximation, a building's surfaces can be modeled by planes or curve surfaces which are fitted to the point cloud. 3D points are resampled on the fitted surfaces in an orderly pattern, whose colors are obtained from the input images. This approach is simple, inexpensive, and effective for reconstructing textured mesh surfaces of large buildings. Test results show that the reconstructed 3D models are sufficiently accurate and realistic for 3D visualization in various applications.

This research presents a novel method for automated construction progress monitoring. Using the proposed method, an accurate and complete 3D point cloud is generated for automatic outdoor and indoor progress monitoring throughout the project duration. In this method, Structured-from-Motion (SFM) and Multi-View-Stereo (MVS) algorithms coupled with photogrammetric principles for the coded targets’ detection are exploited to generate as-built 3D point clouds. The coded targets are utilized to automatically resolve the scale and increase the accuracy of the point cloud generated using SFM and MVS methods. Having generated the point cloud, the CAD model is generated from the as-built point cloud and compared with the as-planned model. Finally, the quantity of the performed work is determined in two real case study projects. The proposed method is compared to the Structured-from-Motion (SFM)/Clustering Multi-Views Stereo (CMVS)/Patch-based Multi-View Stereo (PMVS) algorithm, as a common method for generating 3D point cloud models. The proposed photogrammetric Multi-View Stereo method reveals an accuracy of around 99 percent and the generated noises are less compared to the SFM/CMVS/PMVS algorithm. It is observed that the proposed method has extensively improved the accuracy of generated points cloud compared to the SFM/CMVS/PMVS algorithm. It is believed that the proposed method may present a novel and robust tool for automated progress monitoring in construction projects.

Abstract. Nowadays, photogrammetrically derived point clouds are widely used in many civilian applications due to their low cost and flexibility in acquisition. Typically, photogrammetric point clouds are assessed through reference data such as LiDAR point clouds. However, when reference data are not available, the assessment of photogrammetric point clouds may be challenging. Since these point clouds are algorithmically derived, their accuracies and precisions are highly varying with the camera networks, scene complexity, and dense image matching (DIM) algorithms, and there is no standard error metric to determine per-point errors. The theory of internal reliability of camera networks has been well studied through first-order error estimation of Bundle Adjustment (BA), which is used to understand the errors of 3D points assuming known measurement errors. However, the measurement errors of the DIM algorithms are intricate to an extent that every single point may have its error function determined by factors such as pixel intensity, texture entropy, and surface smoothness. Despite the complexity, there exist a few common metrics that may aid the process of estimating the posterior reliability of the derived points, especially in a multi-view stereo (MVS) setup when redundancies are present. In this paper, by using an aerial oblique photogrammetric block with LiDAR reference data, we analyze several internal matching metrics within a common MVS framework, including statistics in ray convergence, intersection angles, DIM energy, etc. We associate these metrics to the per-point errors evaluated through LiDAR reference data and discuss their potential contributions in estimating internal reliabilities of point clouds derived from DIM algorithms. The experimental results show that ray convergence and DIM energy are relevant indicators for the accuracy of the generated point clouds. Initial investigation shows that these two indicators could be further utilized to infer the measurement errors without reference data, which could potentially estimate the reliabilities of point clouds through error propagation.

Hyperspectral images 3D reconstruction based on structure-from-motion and multi-view stereo

35 - Determination of Principal Components in Data

The learning-based multiview stereo (MVS) methods for three-dimensional (3D) reconstruction generally use 3D volumes for depth inference. The quality of the reconstructed depth maps and the corresponding point clouds is directly influenced by the spatial resolution of the 3D volume. Consequently, these methods produce point clouds with sparse local regions because of the lack of the memory required to encode a high volume of information. Here, we apply the atrous spatial pyramid pooling (ASPP) module in MVS methods to obtain dense feature maps with multiscale, long-range, contextual information using high receptive fields. For a given 3D volume with the same spatial resolution as that in the MVS methods, the dense feature maps from the ASPP module encoded with superior information can produce dense point clouds without a high memory footprint. Furthermore, we propose a 3D loss for training the MVS networks, which improves the predicted depth values by 24.44%. The ASPP module provides state-of-the-art qualitative results by constructing relatively dense point clouds, which improves the DTU MVS dataset benchmarks by 2.25% compared with those achieved in the previous MVS methods.

Improving completeness and accuracy of 3D point clouds by using deep learning for applications of digital twins to civil structures

Multi-view based reconstruction is one of the major approaches to create dense 3D point clouds from series of images, and many excellent algorithms have been published recently. Especially, most of the algorithms are capable to achieve complete reconstruction results when dealing with the highly-textured regions of the objects and scenes. However, due to the lack of textures within some regions, surface reflection and occlusions, certain regions of objects or scenes can usually not be reconstructed correctly, leading to undesirable holes in the uncomplete reconstructed point clouds. In this work, a new strategy on hole-filling for 3D reconstructed models from multi-view stereo is proposed. The holes in the point clouds can be detected automatically by examining the geometric properties of the original point clouds, and then the detected holes will be filled by using moving least squares in an iterative manner. The proposed algorithm has been demonstrated to improve the quality and the completeness of reconstructed point clouds testing on DTU dataset in the experiments.

We address multiview stereo (MVS), an important 3D vision task that reconstructs a 3D model such as a dense point cloud from multiple calibrated images. We propose CER-MVS (Cascaded Epipolar RAFT Multiview Stereo), a new approach based on the RAFT (Recurrent All-Pairs Field Transforms) architecture developed for optical flow. CER-MVS introduces five new changes to RAFT: epipolar cost volumes, cost volume cascading, multiview fusion of cost volumes, dynamic supervision, and multiresolution fusion of depth maps. CER-MVS is significantly different from prior work in multiview stereo. Unlike prior work, which operates by updating a 3D cost volume, CER-MVS operates by updating a disparity field. Furthermore, we propose an adaptive thresholding method to balance the completeness and accuracy of the reconstructed point clouds. Experiments show that our approach achieves state-of-the-art performance on the DTU and Tanks-and-Temples benchmarks (both intermediate and advanced set). Code is available at https://github.com/princeton-vl/CER-MVS.KeywordMultiview stereo

This paper presents a fast and reliable approach for detecting 3D line segment on the unorganized point clouds from multiview stereo. The core idea is to discover weak matching of line segments by re-projecting 3D point to 2D image plane and infer 3D line segment by spatial constraints. On the basis of 2D line segment detector and multiview stereo, the proposed algorithm firstly re-projects the spatial point clouds into planar set on different camera matrices; then finds the best re-projection line from tentative matched points. Finally, 3D line segment is produced by back-projection after outlier removal. In order to remove the matching errors caused by re-projection, a plane clustering method is implemented. Experimental results show that the approach can obtain satisfactory 3D line detection visually as well as high computational efficiency. The proposed fast line detection can be extended in the application of 3D sketch for large-scale scenes from multiple images.

Abstract. Despite the success of multi-view stereo (MVS) reconstruction from massive oblique images in city scale, only point clouds and triangulated meshes are available from existing MVS pipelines, which are topologically defect laden, free of semantical information and hard to edit and manipulate interactively in further applications. On the other hand, 2D polygons and polygonal models are still the industrial standard. However, extraction of the 2D polygons from MVS point clouds is still a non-trivial task, given the fact that the boundaries of the detected planes are zigzagged and regularities, such as parallel and orthogonal, cannot preserve. Aiming to solve these issues, this paper proposes a hierarchical polygon regularization method for the photogrammetric point clouds from existing MVS pipelines, which comprises of local and global levels. After boundary points extraction, e.g. using alpha shapes, the local level is used to consolidate the original points, by refining the orientation and position of the points using linear priors. The points are then grouped into local segments by forward searching. In the global level, regularities are enforced through a labeling process, which encourage the segments share the same label and the same label represents segments are parallel or orthogonal. This is formulated as Markov Random Field and solved efficiently. Preliminary results are made with point clouds from aerial oblique images and compared with two classical regularization methods, which have revealed that the proposed method are more powerful in abstracting a single building and is promising for further 3D polygonal model reconstruction and GIS applications.

Abstract. We propose to use a discriminative classifier for outlier detection in large-scale point clouds of cities generated via multi-view stereo (MVS) from densely acquired images. What makes outlier removal hard are varying distributions of inliers and outliers across a scene. Heuristic outlier removal using a specific feature that encodes point distribution often delivers unsatisfying results. Although most outliers can be identified correctly (high recall), many inliers are erroneously removed (low precision), too. This aggravates object 3D reconstruction due to missing data. We thus propose to discriminatively learn class-specific distributions directly from the data to achieve high precision. We apply a standard Random Forest classifier that infers a binary label (inlier or outlier) for each 3D point in the raw, unfiltered point cloud and test two approaches for training. In the first, non-semantic approach, features are extracted without considering the semantic interpretation of the 3D points. The trained model approximates the average distribution of inliers and outliers across all semantic classes. Second, semantic interpretation is incorporated into the learning process, i.e. we train separate inlieroutlier classifiers per semantic class (building facades, roof, ground, vegetation, fields, and water). Performance of learned filtering is evaluated on several large SfM point clouds of cities. We find that results confirm our underlying assumption that discriminatively learning inlier-outlier distributions does improve precision over global heuristics by up to ≈ 12 percent points. Moreover, semantically informed filtering that models class-specific distributions further improves precision by up to ≈ 10 percent points, being able to remove very isolated building, roof, and water points while preserving inliers on building facades and vegetation.

SfM-MVS (Structure from Motion -Multi View Stereo) photogrammetry is a cost-effective and versatile technique used, among other applications, for the three-dimensional (3D) modelling of archaeological heritage sites and artistic objects. Traditional photogrammetry primarily derives the calibration parameters of the camera and the exterior orientation variables from ground control points (GCPs), obtained by means of a GPS, and tie points (TPs). The SfM-MVS approach simultaneously computes both the relative projection geometry and a set of sparse 3D points. However, when these techniques must be used with difficult-to-reach objects, the process is very complex and implies the use of expensive pieces of equipment. For these reasons, this study develops a new procedure to obtain 3D scale models using of a low-cost Unmanned Aerial System (UAS). To achieve good dimensional accuracy, a scaling tool has been designed which can be carried by the aircraft and deployed on the top of object. By means of this tool it is possible to determine the object actual size, scale the 3D model and then verify the operability and quality of the measurements obtained. The new method has been validated by comparing point clouds and distances. The results showed an error of less than 1 mm in almost 90% of the point cloud and a mean relative error of 1.49%. The procedure is therefore simple, effective and allows the use of SfM-MVS photogrammetry techniques indoors or outdoors on objects which due to their complex location, it is impossible to use scaling references and their reconstruction is only feasible with sophisticated and expensive technical means.

Abstract. This study presents a methodology for dense point cloud fusion based on Multi-Criteria Decision-Making (MCDM) techniques, applied to heritage documentation. Photogrammetric reconstruction was conducted using both classical Multi-View Stereo (MVS) algorithms, including Agisoft Metashape, RealityScan, and OpenMVS, as well as a learning-based method (VIS-MVS Net). UAV imagery of historical buildings from the Museum of the Kielce Countryside served as input data, while terrestrial laser scanning (TLS) provided reference datasets. Point cloud quality was evaluated based on completeness, density, and geometric accuracy, with additional metrics assessing surface roughness, planarity, and variance. The proposed fusion approach employed the CRITIC method to assign objective weights to geometric descriptors and used TOPSIS and OWA algorithms to compute point quality scores and merge multiple datasets. The MCDM-based fusion method effectively integrated point clouds of varying origins, preserving structural fidelity and surface smoothness while compensating for missing data. The developed methodology provides a systematic and objective framework for integrating multi-source point clouds, supporting advanced heritage documentation and metrological applications.

Abstract. 3D Gaussian Splatting (3DGS) is an innovative solution for explicit point-based 3D representation where each point is represented as a Gaussian distribution. Using calibrated images and sparse point cloud for initialization, the scene is reconstructed by optimizing the Gaussian position, orientation, shape and appearance. In this contribution, we present a comprehensive overview of 3DGS methods available on complimentary radiance field reconstruction platforms namely, the original 3DGS implementation as reference, Splatfacto, 3DGS-MCMC and 3DGS-LumaAI to address a broader audience in industry and non-technical users as well. Being an indispensable part of our environment, we are particularly interested in vegetation since the irregular and complex shape of plants and trees, especially dense foliage can challenge the 3D reconstruction. We evaluate the geometric accuracy and completeness in two real-world scenarios, one occlusion-free indoor and one outdoor scenario where the object of interest is placed behind vegetation to investigate how the methods can reconstruct the underlying geometry behind occlusion. To investigate if 3DGS methods can challenge traditional and state-of-the-art 3D reconstruction approaches we compare the results with Multi-View Stereo (MVS) and Neural Radiance Fields (NeRFs). The evaluation is based on point cloud comparison against a ground truth mesh. Just behind MVS, the original 3DGS implementation achieves second best accuracy results outperforming NeRFs in both scenarios, making it the most accurate 3DGS method. 3DGS-MCMC achieves the best and third best completeness for each scenario respectively, making it competitive with MVS and NeRFs in real-world setting. Moreover, we demonstrate 3DGS ability to reliably reconstruct the geometry behind vegetation occlusion indicating the potential for large-scale forestry applications, allowing canopy reconstruction, biomass estimation and agricultural monitoring.