SfM Algorithm Research Articles

Enhancing Building Point Cloud Reconstruction from RGB UAV Data with Machine-Learning-Based Image Translation.

The performance of three-dimensional (3D) point cloud reconstruction is affected by dynamic features such as vegetation. Vegetation can be detected by near-infrared (NIR)-based indices; however, the sensors providing multispectral data are resource intensive. To address this issue, this study proposes a two-stage framework to firstly improve the performance of the 3D point cloud generation of buildings with a two-view SfM algorithm, and secondly, reduce noise caused by vegetation. The proposed framework can also overcome the lack of near-infrared data when identifying vegetation areas for reducing interferences in the SfM process. The first stage includes cross-sensor training, model selection and the evaluation of image-to-image RGB to color infrared (CIR) translation with Generative Adversarial Networks (GANs). The second stage includes feature detection with multiple feature detector operators, feature removal with respect to the NDVI-based vegetation classification, masking, matching, pose estimation and triangulation to generate sparse 3D point clouds. The materials utilized in both stages are a publicly available RGB-NIR dataset, and satellite and UAV imagery. The experimental results indicate that the cross-sensor and category-wise validation achieves an accuracy of 0.9466 and 0.9024, with a kappa coefficient of 0.8932 and 0.9110, respectively. The histogram-based evaluation demonstrates that the predicted NIR band is consistent with the original NIR data of the satellite test dataset. Finally, the test on the UAV RGB and artificially generated NIR with a segmentation-driven two-view SfM proves that the proposed framework can effectively translate RGB to CIR for NDVI calculation. Further, the artificially generated NDVI is able to segment and classify vegetation. As a result, the generated point cloud is less noisy, and the 3D model is enhanced.

Research on the Innovative Development of Digital Media Enabled by Computer Visual Arts

Abstract This paper explores the integration of computer visual art with digital media, highlighting the innovative potential of computer technology in creating dynamic digital media art. Initially, we review relevant theories of computer-based visual art and examine their applications in digital media art. We then detail an immersive scene design using Web3D technology, where images are processed via a computer vision camera imaging model, transformed, and analyzed with the SIFT algorithm to extract feature points. These are integrated with the incremental SFM algorithm for three-dimensional image reconstruction in immersive environments. We validate these methods with real datasets and controlled experiments to assess user experience improvements in digital media applications. Results indicate that the SIFT algorithm’s image-matching accuracy ranges from 50% to 78%, and the incremental SFM algorithm shows an average rotation error of only 0.183°. User sentiment scores improved significantly, from 11.583 to 23.452, post-experience, demonstrating enhanced realism and a new direction for digital media art.

A method for obtaining maize phenotypic parameters based on improved QuickShift algorithm

Phenotype is an important bridge for studying the mechanism of “genotype-phenotype-environment”, and studying precise measurement of crop individual phenotypic information is of great significance for accelerating breeding processes and assisting in precision agriculture monitoring. The purpose of this study is to develop a clustering algorithm for corn population point clouds to accurately extract the three-dimensional morphology of individual crops. Firstly, this study used drones to obtain field corn data, reconstructed corn plant models based on the SfM algorithm, simplified and rotated point clouds, and extracted crop point clouds using methods based on color and distance thresholds. For density segmentation, this study improved the QuickShift method to segment individual corn crops. Color weights were added to the clustering density calculation process, and the original point cloud was projected onto a horizontal plane to calculate the density clustering core. Then, the remaining points were subjected to QuickShift clustering segmentation in 3D to obtain the point cloud data of individual corn plants. The use of cylindrical fitting and conditional Euclidean clustering algorithms to automatically segment point clouds of stems and leaves has achieved estimation of plant height, stem thickness, minimum bounding box volume per plant, and number of leaves. Finally, the true and measured values of plant height and stem thickness were compared. The results showed that the accuracy of the improved QuickShift method for segmentation was 93.10%, with an average measured plant height of 14.21 cm and an average stem diameter of 0.72 cm. The determination coefficient and root mean square error of the automatic and manual measurement results of plant height are 0.9186 and 0.5785 cm, respectively. The determination coefficients and root mean square errors of the automatic and manual measurement results of stem thickness are 0.8227 and 0.0574 cm, respectively. This study uses an improved QuickShift to cluster and segment corn point cloud data, achieving the segmentation of individual corn plants in the field, providing an automated, efficient, and low-cost solution for accurately measuring crop phenotypic information.

Method for determining of shallow water depths based on data recorded by UAV/USV vehicles and processed using the SVR algorithm

Bathymetric measurements in waters shallower than 1 m are necessary to monitor seafloor relief changes in the coastal zone. This is especially important for ensuring the safety of navigation, navigation efficiency, as well as during the design and monitoring of hydrotechnical structures. Therefore, the aim of this article is to present a method for determining of shallow water depths based on data recorded by Unmanned Aerial Vehicle (UAV) and Unmanned Surface Vehicle (USV), as well as processed using the Support Vector Regression (SVR) algorithm. Validation studies of this method was carried out on two selected waterbodies (inland and sea) using the DJI Phantom 4 Real Time Kinematic (RTK) UAV and the AutoDron USV. Then, the geospatial data recorded by a drone was applied to determine depths of shallow waterbodies using the SfM algorithm. Subsequently, the determined depths were compared with the depths measured by a Global Navigation Satellite System (GNSS) RTK receiver or a Single Beam Echo Sounder (SBES), in order to assess the accuracy of the proposed method. The research showed that it is able to determine shallow waterbody depths with the accuracy requirements specified for the International Hydrographic Organization (IHO) S-44 special order (depth error ≤ 0.25 m (p = 0.95)), but only to depths not exceeding 1 m. In the inland waterbody, the depth accuracy was 0.22–0.24 m (p = 0.95), while in the sea waterbody the measurement accuracy was higher, amounting to 0.16 m (p = 0.95). However, for the depth range up to 2 m, the results are radically different, because the vertical position error with a 95% confidence level exceeds the value of 0.3 m in the analysed waterbodies.

Using UAV Photogrammetry and Automated Sensors to Assess Aquifer Recharge from a Coastal Wetland

Novel data-acquisition technologies have revolutionized the study of natural systems, allowing the massive collection of information in situ and remotely. Merging these technologies improves the understanding of complex hydrological interactions, such as those of wetland–aquifer systems, and facilitates their conservation and management. This paper presents the combination of UAV technology with water level dataloggers for the study of a coastal temporary wetland linked to an underlying sandy aquifer and influenced by the tidal regime. Wetland morphology was defined using UAV imagery and SfM algorithms during the dry period. The DTM (6.9 cm resolution) was used to generate a flood model, which was subsequently validated with an orthophoto from a wet period. This information was combined with water stage records at 10-min intervals from a network of dataloggers to infer the water balance of the wetland and the transfers to the aquifer. Inflows into the pond were around 6200 m3 (40% direct precipitation over the pond, 60% surface runoff). Outputs equalled the inputs (41% direct evaporation from water surface, 59% transfers into the aquifer). The proposed methodology has demonstrated its suitability to unravel complex wetland–aquifer interactions and to provide reliable estimations of the elements of the water balance.

Landscape Fusion Method Based on Augmented Reality and Multiview Reconstruction.

This paper proposes a fused landscape augmented reality method based on 3D model multiview reconstruction. Based on the principles related to augmented reality technology, the proposed method uses natural features of images for training and extraction, which solves the problems of convenience and aesthetics caused by artificial signs. By extracting and training natural features at different scales of the acquired images, Harris and FREAK algorithms are used to extract features and create binary descriptors for real-time acquired images. Feature matching is performed on the above two features to estimate the location where the reconstructed model will appear. At the same time, for the difficulties of 3D model reconstruction requiring relevant expertise and the defects of poor reconstruction effect, the SFM algorithm is used for multiview reconstruction of landscape models to realize the augmented reality fusion method of natural scenes and landscape models. After the experiments, the fusion achieved by this method works well, which proves that the method is feasible and has potential.

Research on Multi-View 3D Reconstruction Technology Based on SFM.

Multi-view 3D reconstruction technology is used to restore a 3D model of practical value or required objects from a group of images. This paper designs and implements a set of multi-view 3D reconstruction technology, adopts the fusion method of SIFT and SURF feature-point extraction results, increases the number of feature points, adds proportional constraints to improve the robustness of feature-point matching, and uses RANSAC to eliminate false matching. In the sparse reconstruction stage, the traditional incremental SFM algorithm takes a long time, but the accuracy is high; the traditional global SFM algorithm is fast, but its accuracy is low; aiming at the disadvantages of traditional SFM algorithm, this paper proposes a hybrid SFM algorithm, which avoids the problem of the long time consumption of incremental SFM and the problem of the low precision and poor robustness of global SFM; finally, the MVS algorithm of depth-map fusion is used to complete the dense reconstruction of objects, and the related algorithms are used to complete the surface reconstruction, which makes the reconstruction model more realistic.

SFM-BASED 3D RECONSTRUCTION OF HERITAGE ASSETS USING UAV THERMAL IMAGES

Abstract. In the last few years, notable progress has been made in the field of non-invasive diagnostic for the monitoring of heritage assets. In particular, multispectral imagery (more specifically thermal images will be addressed in this manuscript) allows investigations in the non-visible range of the electro-magnetic spectrum to be effectively carried out. Many researchers are currently exploring the possibilities related to the use of this kind of images in photogrammetric SfM-based processes to produce 2D and 3D value-added metric products, characterised by high level of detail and spatial resolution, including the information connected to the non-visible data. A data fusion-based strategy enables co-registering visible and thermal images in order to exploit the higher spatial resolution of the traditional true colour images. However, there are still many shortcomings to be addressed to properly and efficiently orient TIR (Thermal Infrared) images, connected (among other factors) to their low spatial resolution, or to the low contrast between adjacent materials characterised by similar emissivity. This paper proposes two different workflows to process thermal images using SfM algorithms, applied to three different case studies, each characterised by different characteristics and features (size, morphology, emissivity of the materials, etc.). The different pipelines are described and the obtained results are critically evaluated considering the metric accuracy, 3D geometric reconstruction and noise, completeness of the data and overall quality of the generated dense point cloud. Additionally, the effectiveness of the adopted strategies in connection with the peculiar features of the analysed case studies is also considered.

Segmentation of Individual Leaves of Field Grown Sugar Beet Plant Based on 3D Point Cloud

Accurate segmentation of individual leaves of sugar beet plants is of great significance for obtaining the leaf-related phenotypic data. This paper developed a method to segment the point clouds of sugar beet plants to obtain high-quality segmentation results of individual leaves. Firstly, we used the SFM algorithm to reconstruct the 3D point clouds from multi-view 2D images and obtained the sugar beet plant point clouds after preprocessing. We then segmented them using the multiscale tensor voting method (MSTVM)-based region-growing algorithm, resulting in independent leaves and overlapping leaves. Finally, we used the surface boundary filter (SBF) method to segment overlapping leaves and obtained all leaves of the whole plant. Segmentation results of plants with different complexities of leaf arrangement were evaluated using the manually segmented leaf point clouds as benchmarks. Our results suggested that the proposed method can effectively segment the 3D point cloud of individual leaves for field grown sugar beet plants. The leaf length and leaf area of the segmented leaf point clouds were calculated and compared with observations. The calculated leaf length and leaf area were highly correlated with the observations with R2 (0.80–0.82). It was concluded that the MSTVM-based region-growing algorithm combined with SBF can be used as a basic segmentation step for high-throughput plant phenotypic data extraction of field sugar beet plants.

Research on 3D Modeling of Wupaolong based on Sparse Point Cloud Reconstruction of SFM Algorithm

As a classic project for the inheritance of Chinese folk culture, Wupaolong has the characteristics of unique material selection and complex production technology. However, as the finished products of folk handicrafts may not comply with production standards, a common problem in the production of handicrafts. Focusing on the problems of poor visual effects such as uneven surface and jagged edges in the craft paper binding model constructed by point cloud, and the need to enhance the sense of reality, this paper proposes an optimization method for 3D model and triangulation based on 3D line features. This method first preprocesses the feature points and performs sparse matching and dense matching on the processed feature points to generate a dense point cloud. The dense point cloud is then processed by an adaptive weighted median filtering method. This is followed by application of surface modeling method based on Delaunay algorithm to realize the reconstruction of object triangulation. Based on the triangular patch, the reference plane of the craft paper binding surface is fitted, and the model surface is corrected and optimized based on the reference plane. With the aid of three-dimensional line segments, the triangulation at the edge of the craft paper binding is corrected and optimized. Finally, an optimized three-dimensional model of craft paper binding is obtained. Experimental results show that the proposed algorithm can improve the unevenness of the 3D model plane and edges, and maintain the characteristics of the 3D model plane and edges. Compared with the existing surface reconstruction algorithms based on point cloud features, the optimized triangulation and model have higher accuracy and quality, which can greatly improve the visual effect of the model and improve the accuracy of the three-dimensional model. The research results of this paper can realize 3D modeling development for the production process of Wupaolong, effectively improve the elevation accuracy of the model, realize the production standard of digital Wupaolong model, and provide reference for folk production process.

Using canopy height model derived from UAV imagery as an auxiliary for spectral data to estimate the canopy cover of mixed broadleaf forests.

Canopy cover is an important structural trait that is frequently used in forest inventories to assess sustainability as well as many other important aspects of forest stands. Remote sensing data is more effective and suitable for canopy cover estimating than traditional field measurements such as sample plots, especially at broad scales. Measurement and mapping this attribute in fine-scale is a difficult task. Aerial imagery using unmanned aerial vehicle (UAV) has been recognized as an excellent tool to estimate canopy attributes. In this study, we compared the potential of using digital hemispherical photography (DHP), digital cover photography (DCP), UAV RGB data, and canopy height model (CHM) for estimation of canopy cover of mix broad-leaved forest on seven different stands. The canopy cover was measured from two digital canopy photographic methods, including DHP (as the reference method) and DCP. The stand orthophotos were segmented using a multi-resolution image segmentation method. Afterward, the classification in two classes of the canopy cover and the non-canopy cover was conducted using minimum distance classification to estimate canopy cover. The CHM layer was generated based on the SfM algorithm and utilized in the canopy cover estimation in each stand as auxiliary data. The results showed a slight improvement when we used the CHM as auxiliary data. Overall, the results showed that the efficiency of the ground digital canopy photographic methods (zenith view) in multi-storied and dense forests is the lowest. In return, our method for digital aerial canopy photography (object-based canopy segmentation and classification) is simple, quick, efficient, and cost-effective.

Extraction of the discontinuity orientation from a digital surface model

The traditional field contact measurement for obtaining parameters of the rock mass discontinuity is of low efficiency and big workload, and the accuracy of the results are affected by human factors. In this paper a method is presented to automatically recognize the discontinuity based on the 3D digital surface model of rock mass obtained with the digital photogrammetry and SFM algorithm. The steps of rock mass DSM reconstruction include collecting rock mass images, matching image features based on the SIFT algorithm, reconstructing sparse point cloud, encrypting point cloud, and reconstructing the rock mass surface model. The main flow of the discontinuity recognition method include smoothing the DSM of rock mass, changing the searching radius and the angle threshold to split model plane, searching the discontinuity based on the regional growth principle, and fitting the discontinuity based on random sampling consistency to get the orientation. The method is applied to the underground experimental roadway in the Beishan area of Gansu, and the reconstruction of 3D digital surface model of roadway and the orientation acquisition of discontinuities are realized. The discontinuities are also mapped on the roadway model by groups. A comparison the results with those of the manual field measurement method and the existing discontinuity recognition software shows that the method proposed in this paper is of good accuracy and can provide a certain reference for engineering applications.

Dense Robust 3D Reconstruction and Measurement for 3D Printing Process Based on Vision

The 3D printing process lacks real-time inspection, which is still an open-loop manufacturing process, and the molding accuracy is low. Based on the 3D reconstruction theory of machine vision, in order to meet the applicability requirements of 3D printing process detection, a matching fusion method is proposed. The fast nearest neighbor (FNN) method is used to search matching point pairs. The matching point information of FFT-SIFT algorithm based on fast Fourier transform is superimposed with the matching point information of AKAZE algorithm, and then fused to obtain more dense feature point matching information and rich edge feature information. Combining incremental SFM algorithm with global SFM algorithm, an integrated SFM sparse point cloud reconstruction method is developed. The dense point cloud is reconstructed by PMVs algorithm, the point cloud model is meshed by Delaunay triangulation, and then the accurate 3D reconstruction model is obtained by texture mapping. The experimental results show that compared with the classical SIFT algorithm, the speed of feature extraction is increased by 25.0%, the number of feature matching is increased by 72%, and the relative error of 3D reconstruction results is about 0.014%, which is close to the theoretical error.

High-accuracy three-dimensional measurement based on multi-directional cooperative target with weighted SfM algorithm

Large-scale three-dimensional (3D) precision measurement is limited by the high-accuracy single-view measurement range. Therefore, it is necessary to perform multi-station coordinate transformation through high-precision 3D cooperative target to complete the measurement for large-scale objects. However, high-precision 3D cooperative target is difficult to be manufactured. The physical features on the cooperative target are also easy to occlude each other in complex measurement environments. To solve these problems and reduce the number of coordinate transformations in the process of global coordinate unification, a Multi-directional Cooperative (MDC) target consisting of designed coded planar feature target and polyhedron skeletons is designed. The designed MDC target is arbitrarily placed in the Field of View (FOV) and images of the target are obtained from multiple views. Based on the captured target images, a new Weighted SfM algorithm is proposed to realize the high-accuracy 3D measurement for the MDC target. The experimental results show that the proposed algorithm has good robustness and high accuracy for the designed MDC target.

A MATCH-MOVING METHOD COMBINING AI AND SFM ALGORITHMS IN HISTORICAL FILM FOOTAGE

Abstract. Searching for suitable material for photogrammetry is a key part in the documentation of Cultural Heritage. Photogrammetry can be used to produce a metrically certified 3D model. Material contained in historical film footage archives is especially useful for documentation when the heritage has been lost. In this research an innovative match-moving method is proposed that aims to exploit Artificial Intelligence and SfM algorithms to identify the frames extracted from a film footage in which the lost monument appears and that are suitable to be processed with photogrammetry for its 3D reconstruction. First of all the identification and tracking of the heritage in the videos was performed training an object detection Neural Network. Then the frames detected were automatically extracted with the coordinates of the bounding boxes that contain the monument. The camera motions were identified by selecting only the shots taken from multiple points of view of the same scene and analysing the evolution of the bounding boxes position over time. A further check of the material was necessary to select only sequences and to eliminate single frames and images from different historic periods. After this process, only the correct frames were automatically selected and processed with photogrammetry and the quality of the obtained 3D model was assessed. The method experimented in this research represents a powerful tool in the field of Cultural Heritage because it makes the selection of suitable material for photogrammetry automatic. Moreover it offers important insights that could be extended to other sectors.

LOW-COST TECHNOLOGICAL IMPLEMENTATIONS RELATED TO INTEGRATED APPLICATION EXPERIMENTS

Abstract. Thanks to the development of new Cuda technologies applied to graphics cards and to a more accessible and growing use by professionals in various commercial, artistic and research sectors, the latest generation digital instruments have implemented new methods of photogrammetric surveying and remote sensing, progressively reducing the cost of the instruments and that related to the application development of the different SfM algorithms, now widely used also in numerous open source software. The research presented in the article compares two of the most recently used free programs with the most satisfactory results in the field of 3D photogrammetric survey and photomodelling: Meshroom, developed by AliceVision and Regard3D, an open source software, compiled in 2015 by the Swiss IT engineer freelance Roman Hiestand. The test case study will concern the photo-modelling, through the free tools previously discussed, of medium size complex decorative details, one of the monumental gates of the ancient Roman city of Sepino, an archaeological site in Molise (IT) located at the foot of the Matese and extending over the Tammaro valley, so as to be able to fully test the capture characteristics and the trend and structure of the pipeline of the two software examined.

IMPROVING PERFORMANCE OF FEATURE EXTRACTION IN SFM ALGORITHMS FOR 3D SPARSE POINT CLOUD

Abstract. The use of Structure-from-Motion algorithms is a common practice to obtain a rapid photogrammetric reconstruction. However, the performance of these algorithms is limited by the fact that in some conditions the resulting point clouds present low density. This is the case when processing materials from historical archives, such as photographs and videos, which generates only sparse point clouds due to the lack of necessary information in the photogrammetric reconstruction. This paper explores ways to improve the performance of open source SfM algorithms in order to guarantee the presence of strategic feature points in the resulting point cloud, even if sparse. To reach this objective, a photogrammetric workflow is proposed to process historical images. The first part of the workflow presents a method that allows the manual selection of feature points during the photogrammetric process. The second part evaluates the metric quality of the reconstruction on the basis of a comparison with a point cloud that has a different density from the sparse point cloud. The workflow was applied to two different case studies. Transformations of wall paintings of the Karanlık church in Cappadocia were analysed thanks to the comparison of 3D model resulting from archive photographs and a recent survey. Then a comparison was performed between the state of the Komise building in Japan, before and after restoration. The findings show that the method applied allows the metric scale and evaluation of the model also in bad condition and when only low-density point clouds are available. Moreover, this tool should be of great use for both art and architecture historians and geomatics experts, to study the evolution of Cultural Heritage.

A submap joining algorithm for 3D reconstruction using an RGB-D camera based on point and plane features

In standard point-based methods, the depth measurements of the point features suffer from noise, which will lead to incorrect global structure of the environment. This paper presents a submap joining based SLAM with an RGB-D camera by introducing planes as well as points as features.This work is consisted of two steps: submap building and submap joining. Several adjacent keyframes, with the corresponding small patches, visual feature points, and planes observed from these keyframes, are used to build a submap. We fuse the submaps into a global map in a sequential fashion, such that, the global structure is recovered gradually through plane feature associations and optimization. We also show that the proposed algorithm can handle plane association problem incrementally in submap level, as the plane covariance can be obtained in each submap. The use of submap significantly reduces the computational cost during the optimization process, while keeping all information about planes. The proposed method is validated using both publicly available RGB-D benchmarks and datasets collected by authors. The algorithm can produce accurate trajectories and high quality 3D models on these challenging datasets, which are difficult for existing RGB-D SLAM or SFM algorithms.

Comparative Analysis of Structure-From-Motion Software’s –An Example of Letychiv (Ukraine) Castle and Convent Buildings

Abstract Letychiv (pl. Latyczów) is a town located in central Ukraine in the Khmelnytskyi Oblast. It has a unique and complicated history. Second World War left it in ruin, destroying buildings, infrastructure and decimating its once large population. Perhaps the most prominent part of the town currently is the building Dominican convent with adjoin Letychiv Assumption Church. This object is surrounded by what is left of the previously impressive Letychiv Castle, founded by Jan Potocki in 1598. Past 30 years have been dedicated by this small Catholic parish towards rebuilding monastery-castle-church complex. Since this is an ongoing project, it was decided to perform a photographic inventory of the current state of the construction and to create a 3D digital model of the castle, facade of the church and monastery, and the altar. This task have proven to be difficult due to complicated structure of the object. Facades and inner parts of the church are almost white with limited number of distinctive elements, painted in pail gold. Elements other than white are almost identical to each other. It leads to various errors in the processing of Structure-from-motion. This article describes how various versions of SfM algorithm work thru mention difficulties, compares results in terms of accuracy, level of detail and overall look. It also describes how SfM can help to document various stages of restoration of important historical objects.

Rapid mapping of landslide disaster using UAV- photogrammetry

Unmanned Aerial Vehicle (UAV) systems offered many advantages in several mapping applications such as slope mapping, geohazard studies, etc. This study utilizes UAV system for landslide disaster occurred in Jombang Regency, East Java. This study concentrates on type of rotor-wing UAV, that is because rotor wing units are stable and able to capture images easily. Aerial photograph were acquired in the form of strips which followed the procedure of acquiring aerial photograph where taken 60 photos. Secondary data of ground control points using GPS Geodetic and check points established using Total Station technique was used. The digital camera was calibrated using close range photogrammetric software and the recovered camera calibration parameters were then used in the processing of digital images. All the aerial photographs were processed using digital photogrammetric software and the output in the form of orthophoto was produced. The final result shows a 1: 1500 scale orthophoto map from the data processing with SfM algorithm with GSD accuracy of 3.45 cm. And the calculated volume of contour line delineation of 10527.03 m3. The result is significantly different from the result of terrestrial methode equal to 964.67 m3 or 8.4% of the difference of both.