Multi-view Stereo Research Articles

Adaptive Feature Enhanced Multi-View Stereo With Epipolar Line Information Aggregation

Adaptive Feature Enhanced Multi-View Stereo With Epipolar Line Information Aggregation

DSM Reconstruction from Uncalibrated Multi-View Satellite Stereo Images by RPC Estimation and Integration

In this paper, we propose a 3D Digital Surface Model (DSM) reconstruction method from uncalibrated Multi-view Satellite Stereo (MVSS) images, where Rational Polynomial Coefficient (RPC) sensor parameters are not available. While recent investigations have introduced several techniques to reconstruct high-precision and high-density DSMs from MVSS images, they inherently depend on the use of geo-corrected RPC sensor parameters. However, RPC parameters from satellite sensors are subject to being erroneous due to inaccurate sensor data. In addition, due to the increasing data availability from the internet, uncalibrated satellite images can be easily obtained without RPC parameters. This study proposes a novel method to reconstruct a 3D DSM from uncalibrated MVSS images by estimating and integrating RPC parameters. To do this, we first employ a structure from motion (SfM) and 3D homography-based geo-referencing method to reconstruct an initial DSM. Second, we sample 3D points from the initial DSM as references and reproject them to the 2D image space to determine 3D–2D correspondences. Using the correspondences, we directly calculate all RPC parameters. To overcome the memory shortage problem while running the large size of satellite images, we also propose an RPC integration method. Image space is partitioned to multiple tiles, and RPC estimation is performed independently in each tile. Then, all tiles’ RPCs are integrated into the final RPC to represent the geometry of the whole image space. Finally, the integrated RPC is used to run a true MVSS pipeline to obtain the 3D DSM. The experimental results show that the proposed method can achieve 1.455 m Mean Absolute Error (MAE) in the height map reconstruction from multi-view satellite benchmark datasets. We also show that the proposed method can be used to reconstruct a geo-referenced 3D DSM from uncalibrated and freely available Google Earth imagery.

3D reconstruction and depth estimation method for local anomalies of rail surface based on multi-view stereo matching

Abstract Detecting rail surface anomalies has become crucial for ensuring the safety of train operations. However, manual detection methods are time-consuming and labor-intensive. Although traditional detection models based on one-dimensional signals or two-dimensional images can effectively identify the existence of defects, they are difficult to use to obtain local depth characteristic information, leading to difficulties in accurately assessing the extent of damage in abnormal regions. Rail track maintenance strategies are typically developed based on the detected different depth ranges of defects. To address this issue, this paper proposes a local depth estimation method based on the point cloud of the target rail surface reconstructed by PatchmatchNet. Firstly, according to the acquisition method of multi-view images and the sampling environment at the site, a practical data collection protocol is established for generating a multi-view dataset of rail surface. Next, dense point cloud of the target rail surface were reconstructed by PatchmatchNet. Subsequently, a method for estimating the local anomaly depth is developed based on the point cloud. Experimental results demonstrate that the proposed method achieves a minimum error of approximately 5.5% within a maximum depth range of 0.35mm, when compared to depth measurements obtained using a structured light camera. Finally, this paper presents a more comprehensive three-dimensional representation of the local abnormal physical structure, surpassing traditional one-dimensional and two-dimensional forms. This enhanced representation offers richer information for the effective determination of whether the anomalies constitute defects and aids in distinguishing true defects from other surface irregularities. Such advancement significantly improves the precision of defect assessment and supports the development of highly precise repair strategies.

A Scaled Monocular 3D Reconstruction Based on Structure from Motion and Multi-View Stereo

Three-dimensional digital modeling at actual scales is essential for digitally preserving cultural relics. While 3D reconstruction using a monocular camera offers a cost-effective solution, the lack of scale information in the resulting models limits their suitability for geometric measurements. Objects with monotonous textures, such as batteries, pose additional challenges due to insufficient feature points, increasing positional uncertainty. This article proposes a method incorporating point and line features to address the scale ambiguity in multi-view 3D reconstruction using monocular cameras. By pre-measuring the lengths of multiple sets of real line segments, building a lookup table, and associating the line features in different images, the table was input into the improved reconstruction algorithm to further optimize the scale information. Experimental results on real datasets showed that the proposed method outperformed the COLMAP method by 70.82% in reconstruction accuracy, with a scale recovery reaching millimeter-level accuracy. This method is highly generalizable, cost-effective, and supports lightweight computation, making it suitable for real-time operation on a CPU.

FaSS-MVS: Fast Multi-View Stereo with Surface-Aware Semi-Global Matching from UAV-Borne Monocular Imagery.

With FaSS-MVS, we present a fast, surface-aware semi-global optimization approach for multi-view stereo that allows for rapid depth and normal map estimation from monocular aerial video data captured by unmanned aerial vehicles (UAVs). The data estimated by FaSS-MVS, in turn, facilitate online 3D mapping, meaning that a 3D map of the scene is immediately and incrementally generated as the image data are acquired or being received. FaSS-MVS is composed of a hierarchical processing scheme in which depth and normal data, as well as corresponding confidence scores, are estimated in a coarse-to-fine manner, allowing efficient processing of large scene depths, such as those inherent in oblique images acquired by UAVs flying at low altitudes. The actual depth estimation uses a plane-sweep algorithm for dense multi-image matching to produce depth hypotheses from which the actual depth map is extracted by means of a surface-aware semi-global optimization, reducing the fronto-parallel bias of Semi-Global Matching (SGM). Given the estimated depth map, the pixel-wise surface normal information is then computed by reprojecting the depth map into a point cloud and computing the normal vectors within a confined local neighborhood. In a thorough quantitative and ablative study, we show that the accuracy of the 3D information computed by FaSS-MVS is close to that of state-of-the-art offline multi-view stereo approaches, with the error not even an order of magnitude higher than that of COLMAP. At the same time, however, the average runtime of FaSS-MVS for estimating a single depth and normal map is less than 14% of that of COLMAP, allowing us to perform online and incremental processing of full HD images at 1-2 Hz.

SA-SatMVS: Slope Feature-Aware and Across-Scale Information Integration for Large-Scale Earth Terrain Multi-View Stereo

Satellite multi-view stereo (MVS) is a fundamental task in large-scale Earth surface reconstruction. Recently, learning-based multi-view stereo methods have shown promising results in this field. However, these methods are mainly developed by transferring the general learning-based MVS framework to satellite imagery, which lacks consideration of the specific terrain features of the Earth’s surface and results in inadequate accuracy. In addition, mainstream learning-based methods mainly use equal height interval partition, which insufficiently utilizes the height hypothesis surface, resulting in inaccurate height estimation. To address these challenges, we propose an end-to-end terrain feature-aware height estimation network named SA-SatMVS for large-scale Earth surface multi-view stereo, which integrates information across different scales. Firstly, we transform the Sobel operator into slope feature-aware kernels to extract terrain features, and a dual encoder–decoder architecture with residual blocks is applied to incorporate slope information and geometric structural characteristics to guide the reconstruction process. Secondly, we introduce a pixel-wise unequal interval partition method using a Laplacian distribution based on the probability volume obtained from other scales, resulting in more accurate height hypotheses for height estimation. Thirdly, we apply an adaptive spatial feature extraction network to search for the optimal fusion method for feature maps at different scales. Extensive experiments on the WHU-TLC dataset also demonstrate that our proposed model achieves the best MAE metric of 1.875 and an RMSE metric of 3.785, which constitutes a state-of-the-art performance.

Self-Supervised 3D Semantic Occupancy Prediction from Multi-View 2D Surround Images

An accurate 3D representation of the geometry and semantics of an environment builds the basis for a large variety of downstream tasks and is essential for autonomous driving related tasks such as path planning and obstacle avoidance. The focus of this work is put on 3D semantic occupancy prediction, i.e., the reconstruction of a scene as a voxel grid where each voxel is assigned both an occupancy and a semantic label. We present a Convolutional Neural Network-based method that utilizes multiple color images from a surround-view setup with minimal overlap, together with the associated interior and exterior camera parameters as input, to reconstruct the observed environment as a 3D semantic occupancy map. To account for the ill-posed nature of reconstructing a 3D representation from monocular 2D images, the image information is integrated over time: Under the assumption that the camera setup is moving, images from consecutive time steps are used to form a multi-view stereo setup. In exhaustive experiments, we investigate the challenges presented by dynamic objects and the possibilities of training the proposed method with either 3D or 2D reference data. Latter being motivated by the comparably higher costs of generating and annotating 3D ground truth data. Moreover, we present and investigate a novel self-supervised training scheme that does not require any geometric reference data, but only relies on sparse semantic ground truth. An evaluation on the Occ3D dataset, including a comparison against current state-of-the-art self-supervised methods from the literature, demonstrates the potential of our self-supervised variant.

A biomimetic chiral auxetic vertebral meta-shell

Abstract The work presents a novel thin-walled biomimetic auxetic meta-shell for patient-specific vertebral orthopedic implants. The proposed design stemmed from the concept of an intrinsically multiple curved auxetic meta-structure, which is created by folding a 2D bio-inspired chiral geometry according to the morphology of human vertebral cortical bones. Through a multi-view stereo Digital Image Correlation (DIC) system, we investigated the mechanical response of a bio-grade titanium (Ti6Al4V ELI) additively manufactured prototype of the meta-structure under compressive loadings. In addition, we analyzed the morphology of the prototype using a scanning electron microscopy (SEM) and an optical image dimension measurement system both before and after compressive tests. An accurate Finite Element model, which exactly reproduced the geometry of the 3D printed meta-shell, was implemented and calibrated against experimental results, obtaining a precise prediction tool of its mechanical response. The findings of this work demonstrate that the designed meta-shell shows a peculiar auxetic behavior, a targeted stiffness matching to that of human vertebral bone tissues and a higher global elastic strain capability compared to those of monolithic traditional vertebral body replacements.


Modelling Water Availability in Livestock Ponds by Remote Sensing: Enhancing Management in Iberian Agrosilvopastoral Systems

Extensive livestock farming plays a crucial role in the economy of agrosilvopastoral systems of the southwestern Iberian Peninsula (known as dehesas and montados in Spanish and Portuguese, respectively) as well as providing essential ecosystem services. The existence of livestock in these areas heavily relies on the effective management of natural resources (annual pastures and water stored in ponds built ad hoc). The present work aims to assess the water availability in these ponds by developing equations to estimate the water volume based on the surface area, which can be quantified by means of remote sensing techniques. For this purpose, field surveys were carried out in September 2021, 2022 and 2023 at ponds located in representative farms, using unmanned aerial vehicles (UAVs) equipped with RGB sensors and survey-grade global navigation satellite systems and inertial measurement units (GNSS-IMU). These datasets were used to produce high-resolution 3D models by means of Structure-from-Motion and Multi-View Stereo photogrammetry, facilitating the estimation of the stored water volume within a Geographic Information System (GIS). The Volume–Area–Height relationships were calibrated to allow conversions between these parameters. Regression analyses were performed using the maximum volume and area data to derive mathematical models (power and quadratic functions) that resulted in significant statistical relationships (r2 > 0.90, p < 0.0001). The root mean square error (RMSE) varied from 1.59 to 17.06 m3 and 0.16 to 3.93 m3 for the power and quadratic function, respectively. Both obtained equations (i.e., power and quadratic general functions) were applied to the estimated water storage in similar water bodies using available aerial or satellite imagery for the period from 1984 to 2021.

NeRF-based Polarimetric Multi-view Stereo

In this paper, we introduce NeRF-based Polarimetric Multi-view Stereo (NPMVS), a novel 3D reconstruction method that combines the advantages of neural radiance field (NeRF) and shape-from-polarization (SfP) to address the challenge posed by textureless areas while preserving the fine-scale geometric details. Our method first leverages neural rendering to yield depth priors for each input view, subsequently estimates more accurate depths and normals using polarimetric refinement. We further introduce a pixel-wise depth rectification process to address the scaling problem inherent to the polarimetric refinement procedure. In addition, we contribute a new realistic pBRDF-based multi-view synthetic dataset, comprised of RGB and polarization images rendered under real-world lighting conditions, which will serve as a valuable resource for future research in this field. Experimental evaluations on both synthetic and real-world datasets validate the superiority of NPMVS, demonstrating its advantage over other state-of-the-art multi-view stereo and shape-from-polarization methods.

The Potential of Hybrid Data Integration Approaches Based UAV-SfM Photogrammetry in Identifying New Archaeological Features in The Archaeological UR Site

Structure-from-Motion (SfM) Photogrammetry-based unmanned aerial vehicles (UAV) are considered one of the most effective non-destructive techniques to detect features in archaeology. However, relying on only standalone-based photogrammetric approaches is not always valid in revealing new archaeological features and monuments. This study aims to verify the efficiency of the integration-based methodologies of SfM photogrammetry products to improve the detection of new archaeological features in the non-excavated regions of the archaeological site of Ur, south of Iraq. The methodology proposes an automatic workflow to extract 3D topographic models such as Digital Terrain Model (DTM) by Agisoft Metashape (MS) software and orthomosaic images as standard products for UAV-photogrammetry. These have delivered the following Structure-from-Motion and Multi-View Stereo (SfM-MVS) algorithms using an optimized flight planning, data capturing, and data processing pipeline. Stand-alone raster images derived from photogrammetry, such as (slope images, hill shade rasters, etc.), have limitations in archaeology compared to hybrid raster images in detecting covered and unrevealed features. The methodology focuses on fusing multiple rasters extracted from DTM using the QGIS software package analysis tools to detect features that may never be detected from single standalone image rasters. This is because combining multiple rasters of different parameters may highlight new potential features such as paths, temples, etc. The results findings revealed that the fusion methodology of raster images obtained from UAV-SfM Photogrammetry through its standard products and using high precision settings could successfully help to identify many new archaeological features in the undiscovered site in Ur, such as walls and houses, which may date back to the late periods of Ur. The results concluded that this enhances the role of photogrammetry and remote sensing techniques as non-destructive excavation tools to provide a rich database for archaeologists for further analysis and interpretation of undiscovered archaeological sites without destruction. Many new archaeological features were also revealed, and a general digital map of the site was prepared. It also prompts a review of the settlement theories in previous studies in the ancient city of UR and the documentation of excavated archaeological ruins.

Learning deformable hypothesis sampling for patchmatch multi-view stereo in the wild

The learnable PatchMatch formulations have recently made progress in Multi-View Stereo (MVS). However, their performance often degrades under complex wild scenes. In this study, we observe that the degradation is mainly caused by the noisy depth hypothesis sampling during iterations of PatchMatch MVS: (i) Within a single iteration, the method mixes all information of a fixed-shape region in the spatial dimension, which introduces sampling noise, especially in regions with abrupt depth changes; (ii) Between iterations, the depth candidate hypotheses are sampled indiscriminately in the depth dimension, which fails to filter out noise in the sampling range and propagates errors to the next iteration. Accordingly, we propose a deformable sampling learning strategy (DefLearn) guided by matching uncertainty to determine the hypothesis set from two perspectives: (i) Encoding the matching uncertainty within the image allows the search region of sampling to fit the depth distribution in the spatial dimension; (i) Modeling the cross-image matching uncertainty more accurately based on the obtained valid hypotheses to capture scene structure information for fine-grained hypothesis searching in the depth dimension. With DefLearn, we develop a deep probabilistic PatchMatch MVS network (PPMNet) for accurate depth estimation with a high awareness of diverse scene structures. Extensive experiments conducted on three challenging datasets demonstrate that our PPMNet significantly outperforms state-of-the-art methods in terms of accuracy and generalization ability, even in particularly challenging wild scenes. Code is available at https://github.com/Geo-Tell/PPMNet.

PSDF: Prior-Driven Neural Implicit Surface Learning for Multi-view Reconstruction.

Surface reconstruction has traditionally relied on the Multi-View Stereo (MVS)-based pipeline, which often suffers from noisy and incomplete geometry. This is due to that although MVS has been proven to be an effective way to recover the geometry of the scenes, especially for locally detailed areas with rich textures, it struggles to deal with areas with low texture and large variations of illumination where the photometric consistency is unreliable. Recently, Neural Implicit Surface Reconstruction (NISR) combines surface rendering and volume rendering techniques and bypasses the MVS as an intermediate step, which has emerged as a promising alternative to overcome the limitations of traditional pipelines. While NISR has shown impressive results on simple scenes, it remains challenging to recover delicate geometry from uncontrolled real-world scenes which is caused by its underconstrained optimization. To this end, the framework PSDF is proposed which resorts to external geometric priors from a pretrained MVS network and internal geometric priors inherent in the NISR model to facilitate high-quality neural implicit surface learning. Specifically, the visibility-aware feature consistency loss and depth prior-assisted sampling based on external geometric priors are introduced. These proposals provide powerfully geometric consistency constraints and aid in locating surface intersection points, thereby significantly improving the accuracy and delicate reconstruction of NISR. Meanwhile, the internal prior-guided importance rendering is presented to enhance the fidelity of the reconstructed surface mesh by mitigating the biased rendering issue in NISR. Extensive experiments on Tanks and Temples datasets show that PSDF achieves state-of-the-art performance on complex uncontrolled scenes.

Flexible technique for global calibration of multi-view stereo vision with a non-overlapping FOV using linear structured light projector

A flexible global calibration method for multi-view stereo vision with non-overlapping field of view (FOV) using linear structured light projector is presented in this paper. At least three groups of linear structured light plane equations are utilized to calculate the rotation between any pair of cameras with non-overlapping FOV, and then the closed solution of the translation is determined based on the constraint that the normal vector is perpendicular to the lines on linear structured light plane. Once the rotation and translation between all pairs of cameras with non-overlapping FOV are obtained, the rotation and translation of each camera relative to the reference position is eventually computed through global optimization. The experimental data reflects that the calibration accuracy is improved by 35.2% after global optimization, and the global calibration results of the proposed method deviates less from that of Zhang’s method.

Tidal connectivity modelling in wetlands associated with coastal lagoon of Punta Rasa natural reserve, Argentina

Tidal flows play a crucial role in connecting coastal lagoons to the sea through channels, in this way determining the hydrological dynamics of these environments. The Punta Rasa Natural Reserve (Ramsar site) is located in a transitional zone between the outer limit of the estuary of the Río de la Plata and the open sea. In this work, we study the northernmost lagoon at the end of the spit, which is connected to the sea by a narrow tidal channel. Wetlands are developing at the edges of the lagoon and in lower lying sectors. The aim of this study is to develop a conceptual model that explains the tidal connectivity of the coastal lagoon and to assess how it affects the marsh environments. Detailed topographic surveys were performed using Structure from Motion Multi-View Stereo from a UAV, while sea level data was analyzed using recordings at 20-min intervals over a five-year period. This dataset facilitated an analysis of the frequency of exceedances for threshold values of 0.8, 1.0, 1.2 and 1.4 m. The frequency analysis reveals that the highest exceedances of these thresholds occur during the summer months. When the water level reaches 1 m, the lagoon is fully connected to the sea via the tidal channel, resulting in complete inundation of the low marshland and the margins of the high marshland. At a tidal level of 1.4 m, the high marsh connected to the tidal channel and the coastal lagoon experience flooding. The results obtained in this study provides insights into modelling hydrological connectivity between marshes and the sea, as well as determining the frequency of tidal inundation in each wetland.

Geometry-Enhanced Attentive Multi-View Stereo for Challenging Matching Scenarios

Geometry-Enhanced Attentive Multi-View Stereo for Challenging Matching Scenarios

A Super-Resolution and 3D Reconstruction Method Based on OmDF Endoscopic Images.

In the field of endoscopic imaging, challenges such as low resolution, complex textures, and blurred edges often degrade the quality of 3D reconstructed models. To address these issues, this study introduces an innovative endoscopic image super-resolution and 3D reconstruction technique named Omni-Directional Focus and Scale Resolution (OmDF-SR). This method integrates an Omnidirectional Self-Attention (OSA) mechanism, an Omnidirectional Scale Aggregation Group (OSAG), a Dual-stream Adaptive Focus Mechanism (DAFM), and a Dynamic Edge Adjustment Framework (DEAF) to enhance the accuracy and efficiency of super-resolution processing. Additionally, it employs Structure from Motion (SfM) and Multi-View Stereo (MVS) technologies to achieve high-precision medical 3D models. Experimental results indicate significant improvements in image processing with a PSNR of 38.2902 dB and an SSIM of 0.9746 at a magnification factor of ×2, and a PSNR of 32.1723 dB and an SSIM of 0.9489 at ×4. Furthermore, the method excels in reconstructing detailed 3D models, enhancing point cloud density, mesh quality, and texture mapping richness, thus providing substantial support for clinical diagnosis and surgical planning.

A vision measurement method for ship hull plates based on multi-view stereo and image segmentation

The background denoising of point clouds is challenging for the measurement of ship hull plates. To address this issue, a vision measurement method is proposed by combining an identity cost volume network (ICV-Net) and a segment anything model (SAM). By applying the ship hull plate masks segmented by SAM to the corresponding depth maps inferred by ICV-Net, the background noise can be directly removed at the depth map level. Additionally, an edge extraction optimization algorithm complementary to SAM is proposed to eliminate the thickness-related noise. Moreover, an efficient depth map refinement algorithm is proposed by imposing the smoothness prior of ship hull plate surfaces and multi-view photometric consistency. Sub-pixel accurate depth maps of ship hull plates can be obtained by the depth map refinement. Experimental results suggest that the proposed method can automatically, accurately and efficiently remove the background noise of ship hull plates, which is crucial toward practical application.

The IGCP 714 Project “3GEO – Geoclimbing & Geotrekking in Geoparks” – Selection of Geodiversity Sites Equipped for Climbing for Combining Outdoor and Multimedia Activities

The IGCP 714 project “3GEO – Geoclimbing & Geotrekking in Geoparks” is financed by the International Geoscience Programme (IGCP) and supported by the International Union of Geological Sciences (IUGS). In this paper, we report on the results of the first phase of the project focused on the criteria to be adopted to select geodiversity sites equipped for climbing or trekking. The selection of geoclimbing sites and geotrekking routes is aimed at combining multimedia tools and outdoor activities for Geosciences promotion and conservation in UNESCO Global Geoparks, aspiring geoparks or geoparks project, and also in protected areas featuring geoheritage sites. Indeed, both outdoor activities and multimedia tools favour the pursuing of many of the United Nations Sustainable Development Goals (e.g., 3, 4, 8, 11). An international consortium of geoscientists from 12 different countries selected, through the proposed procedure, 22 geoclimbing sites, and then they also detected 30 geotrekking routes mirroring Earth geodiversity. At some test sites (geoclimbing and geotrekking) multimedia tools and digital outcrop models have been developed through different methodological approaches (e.g., Structure from Motion and Multi-View Stereo photogrammetry), to open the way to the second part of the project still in progress. These sites and the relative virtual models are herein also shown. The final aim of the IGCP 714 project is indeed to create an open data repository (digital outcrop models, videos, virtual tours, photos, scientific information, and interpretations) to upload data of the selected sites to mirror Earth geodiversity for different users including tourists and school groups.

Bidirectional Hybrid LSTM Based Recurrent Neural Network for Multi-View Stereo.

Recently, deep learning based multi-view stereo (MVS) networks have demonstrated their excellent performance on various benchmarks. In this paper, we present an effective and efficient recurrent neural network (RNN) for accurate and complete dense point cloud reconstruction. Instead of regularizing the cost volume via conventional 3D CNN or unidirectional RNN like previous attempts, we adopt a bidirectional hybrid Long Short-Term Memory (LSTM) based structure for cost volume regularization. The proposed bidirectional recurrent regularization is able to perceive full-space context information comparable to 3D CNNs while saving runtime memory. For post-processing, we introduce a visibility based approach for depth map refinement to obtain more accurate dense point clouds. Extensive experiments on DTU, Tanks and Temples and ETH3D datasets demonstrate that our method outperforms previous state-of-the-art MVS methods and exhibits high memory efficiency at runtime.