Multi-view Stereo Reconstruction Research Articles

Multi-View Jujube Tree Trunks Stereo Reconstruction Based on UAV Remote Sensing Imaging Acquisition System

High-quality agricultural multi-view stereo reconstruction technology is the key to precision and informatization in agriculture. Multi-view stereo reconstruction methods are an important part of 3D vision technology. In the multi-view stereo 3D reconstruction method based on deep learning, the effect of feature extraction directly affects the accuracy of reconstruction. Aiming at the actual problems in orchard fruit tree reconstruction, this paper designs an improved multi-view stereo structure based on the combination of remote sensing and artificial intelligence to realize the accurate reconstruction of jujube tree trunks. Firstly, an automatic key frame extraction method is proposed for the DSST target tracking algorithm to quickly recognize and extract high-quality data. Secondly, a composite U-Net feature extraction network is designed to enhance the reconstruction accuracy, while the DRE-Net feature extraction enhancement network improved by the parallel self-attention mechanism enhances the reconstruction completeness. Comparison tests show different levels of improvement on the Technical University of Denmark (DTU) dataset compared to other deep learning-based methods. Ablation test on the self-constructed dataset, the MVSNet + Co U-Net + DRE-Net_SA method proposed in this paper improves 20.4% in Accuracy, 12.8% in Completion, and 16.8% in Overall compared to the base model, which verifies the real effectiveness of the scheme.

Evaluation of Asphalt Pavement Texture Using Multiview Stereo Reconstruction Based on Deep Learning

To quickly obtain texture information and accurately evaluate the skid resistance of asphalt pavement, a multiview stereo reconstruction method based on deep learning is proposed for evaluating the texture depth of asphalt pavement. First, a depth camera and a digital camera are used to collect RGB-D (i.e., Red, Green, Blue, Depth) and RGB (i.e., Red, Green, Blue) multiview image datasets on different types of pavements for model training and validation respectively. Then, the model precision is validated by calculating the overlap (intersection over union, IoU) between the ground truth point cloud and the reconstructed point cloud. The model performances under different training strategies are compared to obtain the best model, and the effects of image resolution, number of views, and types of pavement material on the model precision are analyzed. Finally, image processing methods and texture depth characterization indexes are proposed to obtain pavement texture information from the reconstructed depth map, thus validating the effectiveness of the model. The results show that the trained model using the transfer learning strategy achieves a reconstruction precision of 0.77 when the image resolution is 3024×3024 and the number of views is 7. Furthermore, the reconstruction performance remains stable across different pavement materials, suggesting the model is suitable for accurately reconstructing depth maps for asphalt pavements. The errors between the predicted texture depth value calculated based on the volume-based index (MTDp) and the profile-based index (MPDp) using the depth maps after image processing and the measured texture depth value (MTDe) using the sand patch method are 11.72% and 18.85%, respectively. It is believed that the pavement texture depth can be effectively evaluated from the reconstructed depth map using the volume-based metric (MTDp). In contrast to traditional testing methods, this approach requires using only a digital camera and a personal computer, making it a lightweight and intelligent analysis method for obtaining pavement texture depth information.

Multiview Stereo Reconstruction with Feature Aggregation Transformer

Multiview Stereo Reconstruction with Feature Aggregation Transformer

SA-MVSNet: Self-attention-based multi-view stereo network for 3D reconstruction of images with weak texture

Multi-view stereo (MVS) reconstruction is a key task of image-based 3D reconstruction, and deep learning-based methods can achieve better results than traditional algorithms. However, most of the current deep learning-based MVS methods use convolutional neural networks (CNNs) to extract image features, which cannot achieve the aggregation of long-distance context information and capture robust global information. In addition, in the process of fusing depth maps into point clouds, the confidence filters will filter out the depth values with low confidence in weak texture areas. These problems will lead to the low completeness of 3D reconstruction of weak texture and texture-less areas. To address the above problems, this paper proposes SA-MVSNet based on the PatchmatchNet with a self-attentive mechanism. First, we design a coarse-to-fine network framework to advance depth map estimation. In the feature extraction network, a module with a pyramid structure based on Swin Transformer Block is used to replace the original Feature Pyramid Network (FPN), and the self-correlation between weak texture areas is enhanced by applying a global self-attention mechanism. Then, we also propose a self-attention-based adaptive propagation module (SA-AP), which applies a self-attention calculation within depth value propagation window to obtain the relative weight values of current pixel and others, and then adaptively samples the depth values of neighbors on the same surface for propagation. Experiments show that SA-MVSNet has significantly improved the completeness of 3D reconstruction for the images with weak texture on DTU (provided by Danish Technical University), BlendedMVS, and Tanks and Temple datasets. Our code is available at https://github.com/miaowang525/SA-MVSNet.

A Multitask Network for Multiview Stereo Reconstruction: When Semantic Consistency-Based Clustering Meets Depth Estimation Optimization

A Multitask Network for Multiview Stereo Reconstruction: When Semantic Consistency-Based Clustering Meets Depth Estimation Optimization

EI-MVSNet: Epipolar-Guided Multi-View Stereo Network With Interval-Aware Label.

Recent learning-based methods demonstrate their strong ability to estimate depth for multi-view stereo reconstruction. However, most of these methods directly extract features via regular or deformable convolutions, and few works consider the alignment of the receptive fields between views while constructing the cost volume. Through analyzing the constraint and inference of previous MVS networks, we find that there are still some shortcomings that hinder the performance. To deal with the above issues, we propose an Epipolar-Guided Multi-View Stereo Network with Interval-Aware Label (EI-MVSNet), which includes an epipolar-guided volume construction module and an interval-aware depth estimation module in a unified architecture for MVS. The proposed EI-MVSNet enjoys several merits. First, in the epipolar-guided volume construction module, we construct cost volume with features from aligned receptive fields between different pairs of reference and source images via epipolar-guided convolutions, which take rotation and scale changes into account. Second, in the interval-aware depth estimation module, we attempt to supervise the cost volume directly and make depth estimation independent of extraneous values by perceiving the upper and lower boundaries, which can achieve fine-grained predictions and enhance the reasoning ability of the network. Extensive experimental results on two standard benchmarks demonstrate that our EI-MVSNet performs favorably against state-of-the-art MVS methods. Specifically, our EI-MVSNet ranks 1st on both intermediate and advanced subsets of the Tanks and Temples benchmark, which verifies the high precision and strong robustness of our model.

3D-based precise evaluation pipeline for maize ear rot using multi-view stereo reconstruction and point cloud semantic segmentation

Maize ear rot poses a severe threat to maize yield and quality. Breeding and cultivating highly resistant maize varieties is a crucial approach for preventing and controlling maize ear rot. However, traditional methods of visually grading the severity of maize ear infection and resistance lack objectivity and repeatability. To meet the requirement of precise breeding and resistance assessment scenarios, a novel pipeline based on three-dimensional (3D) point clouds of maize ear was developed for ear rot precise evaluation. First, multi-view stereo (MVS) reconstruction was employed to obtain high-precision dense point clouds of maize ears. And the coordinate correction and circular sampling approaches were proposed to optimize the data structure of the input maize ear samples. Next, a specialized network called the ear rot segmentation network (ERSegNet) was proposed to detect the infected area of maize ears. This network incorporated an orientation-encoding (OE) module and point transformer (PT) attention, which effectively boosted the performance of PointNet++. The proposed ERSegNet achieved impressive results, including a mean intersection over union (mIoU) of 85.83%, a mean precision (mPrec) of 92.34%, a mean recall (mRec) of 92.23%, a mean F1-score of 92.28%, and an overall accuracy (OA) of 93.76%. This demonstrated the feasibility of using semantic segmentation algorithms to predict 3D point clouds of maize ears. Furthermore, a point cloud resampling method was suggested to enhance the spatial uniformity of maize ear point clouds and a point-level quantitative assessment approach based on the 3D point cloud data was provided for evaluating the severity of ear rot. The results showed an average evaluation error of 1.55% in the testing set, indicating the accuracy of the proposed method. This study provides a reliable and objective method for maize ear rot precise assessment, offering potential and valuable support for the identification of resistant varieties in breeding programs.

Digital twin models of replicative ground stones: insight into simulating usage of Upper Paleolithic tools

This work presents the first attempt to create a physics-based digital twin model for predictive analysis of damage evolution during the use of ground stone tools (GSTs) in transformative tasks, encompassing the processing of raw resources for nutritional and non-alimentary purposes. The proposed methodology introduces a digital twin of the GSTs developed from 3D models generated using a photogrammetric technique based on Structure-from-Motion and Multi-View Stereo reconstruction. These models serve as the foundation for the development of the finite element (FE)-based digital twin model of the GSTs that exploits a contact formulation and the phase-field approach to simulate tool damage during pounding and grinding tasks. Defining the initial relative positions of the stones, their mechanical behaviour, and controlling the movement of the active stone in a way as close as possible to the real one, the digital twin model has been devised to evaluate how the surface damage is affected by perturbations in the loading conditions. The simulated damage is compared with the surface traces observed from experiments. The developed digital twin model aims at demonstrating its potentials for the GSTs investigations, as a supporting tool for experiments and for simulated tests on the archaeological records.

Surface defects 3D localization for fluorescent magnetic particle inspection via regional reconstruction and partial-in-complete point clouds registration

Accurate three-dimensional localization of defects plays a crucial role in the automated defect marking and grinding process. However, this task becomes increasingly intricate when precisely recovering the three-dimensional information of small defects due to the presence of noise and damage in point clouds generated through three-dimensional reconstruction. Simultaneously, separately reconstructing defects at different locations on the component would significantly impact efficiency. This study focuses on fluorescent magnetic particle inspection, with the aim of enhancing the accuracy of fine crack localization and multi-defects reconstruction efficiency. Its basic idea is to align the precise complete point cloud with the regionally reconstructed noisy point cloud of a component, achieving its spatial positioning within the visual system while eliminating noise and imperfections. We commence by discussing various methods for acquiring offline complete point clouds and their respective advantages and disadvantages. To enhance flexibility and minimize regional reconstruction noise, we have devised a multi-camera vision system and optimized a multi-view stereo reconstruction algorithm. We introduce a filtering function for culling back-facing points during the sparse point cloud projection process. The feasibility of our approach is empirically validated, with a particular focus on its precision and efficiency. Experimental results indicate that our method can execute within 5 s, with a relative error of less than 2 millimeters for 95% of the points and a root mean square error less than 1 millimeter. Our approach exhibits significant advantages in addressing noise issues associated with point cloud data generated through image-based 3D reconstruction processes.

A large scale training sample database system for intelligent interpretation of remote sensing imagery

ABSTRACT Artificial Intelligence (AI) Machine Learning (ML) technologies, particularly Deep Learning (DL), have demonstrated significant potential in the interpretation of Remote Sensing (RS) imagery, covering tasks such as scene classification, object detection, land-cover/land-use classification, change detection, and multi-view stereo reconstruction. Large-scale training samples are essential for ML/DL models to achieve optimal performance. However, the current organization of training samples is ad-hoc and vendor-specific, lacking an integrated approach that can effectively manage training samples from different vendors to meet the demands of various RS AI tasks. This article proposes a solution to address these challenges by designing and implementing LuoJiaSET, a large-scale training sample database system for intelligent interpretation of RS imagery. LuoJiaSET accommodates over five million training samples, providing support for cross-dataset queries and serving as a comprehensive training data store for RS AI model training and calibration. It overcomes challenges related to label semantic categories, structural heterogeneity in label representation, and interoperable data access.

Miper-MVS: Multi-scale iterative probability estimation with refinement for efficient multi-view stereo

Multi-view stereo reconstruction aims to construct 3D scenes from multiple 2D images. In recent years, learning-based multi-view stereo methods have achieved significant results in depth estimation for multi-view stereo reconstruction. However, the current popular multi-stage processing method cannot solve the low-efficiency problem satisfactorily owing to the use of 3D convolution and still involves significant amounts of calculation. Therefore, to further balance the efficiency and generalization performance, this study proposed a multi-scale iterative probability estimation with refinement, which is a highly efficient method for multi-view stereo reconstruction. It comprises three main modules: 1) a high-precision probability estimator, dilated-LSTM that encodes the pixel probability distribution of depth in the hidden state, 2) an efficient interactive multi-scale update module that fully integrates multi-scale information and improves parallelism by interacting information between adjacent scales, and 3) a Pi-error Refinement module that converts the depth error between views into a grayscale error map and refines the edges of objects in the depth map. Simultaneously, we introduced a large amount of high-frequency information to ensure the accuracy of the refined edges. Among the most efficient methods (e.g., runtime and memory), the proposed method achieved the best generalization on the Tanks & Temples benchmarks. Additionally, the performance of the Miper-MVS was highly competitive in DTU benchmark. Our code is available at https://github.com/zhz120/Miper-MVS.

Review of multi-view stereo reconstruction methods based on deep learning

Review of multi-view stereo reconstruction methods based on deep learning

Heterogeneous Feature Fusion Module Based on CNN and Transformer for Multiview Stereo Reconstruction

For decades, a vital area of computer vision research has been multiview stereo (MVS), which creates 3D models of a scene using photographs. This study presents an effective MVS network for 3D reconstruction utilizing multiview pictures. Alternative learning-based reconstruction techniques work well, because CNNs (convolutional neural network) can extract only the image’s local features; however, they contain many artifacts. Herein, a transformer and CNN are used to extract the global and local features of the image, respectively. Additionally, hierarchical aggregation and heterogeneous interaction modules were used to improve these features. They are based on the transformer and can extract dense features with 3D consistency and global context that are necessary to provide accurate matching for MVS.

Application of Rotationally Symmetrical Triangulation Stereo Vision Sensor in National Dance Movement Detection and Recognition

Information acquisition is an important branch of information science. It is the product of the development and cross-integration of traditional sensing technology and other multidisciplines. It is characterized by the high precision, high speed, integration, and intelligence of information acquisition. An important part of information acquisition, it studies how to acquire the geometric structure and scale information of objects in three-dimensional space. With the continuous development of the level, the acquisition of 3D information has become more and more important in scientific research and industrial production. This paper takes 3D information acquisition technology as the main line. From the perspectives of fusion, effectiveness, physical limit, and measurement efficiency, high-resolution 3D information acquisition techniques are studied. In this paper, the upper and lower limb movements of each behavioral segment are segmented according to the difference. An omnidirectional stereo vision sensor composed of a single camera and two secondary conical mirrors is designed, which solves the problems of large size, narrow field of view, and asynchronous image acquisition of traditional binocular vision sensors, effectively ensuring image perspective. The invariance of projection avoids image distortion caused by curved mirror imaging and reduces the difficulty of subsequent work. The three-dimensional Delaunay triangulation algorithm is used to replace the traditional Poisson reconstruction algorithm to generate the triangular mesh model. The triangular mesh model obtained by the multiview stereo reconstruction algorithm is relatively rough. This article uses Zbrush’s ZRemesher and geometric-divide functions to smooth and simplify the model. The results show that the accuracy of the algorithm proposed in this paper is as high as 89.25% in the motion recognition of body elements. The dynamic mapping method is used to map the texture of the triangular mesh model, so that the realism of the model reaches 91.23%.

Analysis of the Style Characteristics of National Dance Based on 3D Reconstruction.

National dance is an important symbol of national spiritual culture, as it embodies each nation's unique history, living habits, ideology, and culture. Many Chinese ethnic groups have developed their own dance forms and styles, each with their own set of charms. Strengthening our understanding of its style and characteristics is critical to our understanding of the national dance. To begin with, this paper proposes a patch-based multiview stereo reconstruction scheme that can accurately reconstruct 3D character models. The movement capture system then collects national dance movement data and uses them to drive a three-dimensional character model that generates dance visual animation. In the human skeleton model, a spatial-temporal HMM (hidden Markov model) is proposed that not only studies the temporal information of human movements but also the spatial information of adjacent connection points. The recognition rate of this algorithm increases to 95.68 percent when the number of setting points is increased to ten and to 98.28 percent when the number of setting points is increased to twenty, according to the research. The experimental results show that the 3D reconstruction model of ethnic dance developed in this paper is capable of meeting the goal of ethnic dance digitalization while being simple to use, being low in cost, and with a good display effect.

Cassava root crown phenotyping using three-dimension (3D) multi-view stereo reconstruction

Phenotypic analysis of cassava root crowns (CRCs) so far has been limited to visual inspection and very few measurements due to its laborious process in the field. Here, we developed a platform for acquiring 3D CRC models using close-range photogrammetry for phenotypic analysis. The state of the art is a low cost and easy to set up 3D acquisition requiring only a background sheet, a reference object and a camera, compatible with field experiments in remote areas. We tested different software with CRC samples, and Agisoft and Blender were the most suitable software for generating high-quality 3D models and data analysis, respectively. We optimized the workflow by testing different numbers of images for 3D reconstruction and found that a minimum of 25 images per CRC can provide high quality 3D models. Up to ten traits, including 3D crown volumes, 3D crown surface, root density, surface-to-volume ratio, root numbers, root angle, crown diameter, cylinder soil volume, CRC compactness and root length can be extracted providing novel parameters for studying cassava storage roots. We applied this platform to partial-inbred cassava populations and demonstrated that our platform provides reliable 3D CRC modelling for phenotypic analysis, analysis of genetic variances and supporting breeding selection.

INVESTIGATING SPHERICAL EPIPOLAR RECTIFICATION FOR MULTI-VIEW STEREO 3D RECONSTRUCTION

Abstract. Multi-view stereo (MVS) reconstruction is essential for creating 3D models. The approach involves applying epipolar rectification followed by dense matching for disparity estimation. However, existing approaches face challenges in applying dense matching for images with different viewpoints primarily due to large differences in object scale. In this paper, we propose a spherical model for epipolar rectification to minimize distortions caused by differences in principal rays. We evaluate the proposed approach using two aerial-based datasets consisting of multi-camera head systems. We show through qualitative and quantitative evaluation that the proposed approach performs better than frame-based epipolar correction by enhancing the completeness of point clouds by up to 4.05% while improving the accuracy by up to 10.23% using LiDAR data as ground truth.

Assessing the relative accuracy of coral heights reconstructed from drones and structure from motion photogrammetry on coral reefs

Low-altitude high-resolution aerial photographs allow for the reconstruction of structural properties of shallow coral reefs and the quantification of their topographic complexity. This study shows the scope and limitations of two-media (air/water) Structure from Motion—Multi-View Stereo reconstruction method using drone aerial photographs to reconstruct coral height. We apply this method in nine different sites covering a total area of about 7000 m2, and we examine the suitability of the method to obtain topographic complexity estimates (i.e., seafloor rugosity). A simple refraction correction and survey design allowed reaching a root mean square error of 0.1 m for the generated digital models of the seafloor (without the refraction correction the root mean square error was 0.2 m). We find that the complexity of the seafloor extracted from the drone digital models is slightly underestimated compared to the one measured with a traditional in situ survey method.

View Path Planning via Online Multiview Stereo for 3-D Modeling of Large-Scale Structures

This study addresses a view-path-planning problem during 3-D scanning of a large-scale structure based on multiview stereo (MVS) for unmanned aerial platforms. Recently, most studies have adopted an explore-then-exploit strategy for 3-D scanning. The strategy first generates a coarse model from a simple overhead scanning and then plans an inspection path to cover the entire surface of the coarse model. However, even though the inspection path may be optimal, it is difficult to guarantee a complete and accurate reconstruction result due to defective factors of MVS such as occlusions, textureless surfaces, and insufficient parallaxes. Furthermore, the entire procedure of this strategy is inefficiently slow because of path redundancies and long MVS processing time. Therefore, we propose a novel view-path-planning method for 3-D scanning based on an online MVS reconstruction algorithm. The suggested method incrementally reconstructs the target model online and iteratively plans view paths by analyzing the current partial reconstructions. The method continuously analyzes the quality of the model and detects inaccurately reconstructed surfaces. It then plans an inspection path that provides a complete coverage of the detected surfaces while maximizing the performance of MVS. This method can construct a complete 3-D model in a single scanning trial without the need for rescanning. Extensive experiments show that our method outperforms the other state-of-the-art methods, especially in terms of the model completeness of complex structures.

Radar-Based Shape and Reflectivity Reconstruction Using Active Surfaces and the Level Set Method.

We investigate a multiview shape reconstruction problem based on an active surface model whose geometric evolution is driven by radar measurements acquired at sparse locations. Building on our previous work in the context of variational methods for the reconstruction of a scene conceptualized as the graph of a function, we generalize this inversion approach for a general geometry, now described by an active surface, strongly motivated by prior variational computer vision approaches to multiview stereo reconstruction from camera images. While conceptually similar, use of radar echoes within a variational scheme to drive the active surface evolution requires significant changes in regularization strategies compared to prior image based methodologies for the active surface evolution to work effectively. We describe all of these aspects and how we addressed them. While our long term objective is to develop a framework capable of fusing radar as well as other image based information, in which the active surface becomes an explicit shared reference for data fusion. In this paper, we explore the reconstruction using radar as a single modality, demonstrating that the presented approach can provide reconstructions of quality comparable to those from image based methods showing great potential for further development toward data fusion.