Incremental Structure From Motion Research Articles

3D reconstruction of spherical images based on incremental structure from motion

ABSTRACT 3D reconstruction plays an increasingly important role in modern photogrammetric systems. Conventional satellite or aerial-based remote sensing (RS) platforms can provide the necessary data sources for the 3D reconstruction of large-scale landforms and cities. Even with low-altitude UAVs (Unmanned Aerial Vehicles), 3D reconstruction in complicated situations, such as urban canyons and indoor scenes, is challenging due to the frequent tracking failures of camera frames and high data collection costs. Recently, spherical images have been extensively exploited due to the capability of recording surrounding environments using one camera exposure. Classical 3D reconstruction pipelines, however, cannot be used for spherical images. Besides, there exist few software packages for the 3D reconstruction of spherical images. Based on the imaging geometry of spherical cameras, this study investigates the algorithms for relative orientation using spherical correspondences, absolute orientation using 3D correspondences between the scene and spherical points, as well as cost functions for BA (bundle adjustment) optimization. Besides, an incremental SfM (Structure from Motion) workflow has been proposed for spherical images by using the above-mentioned algorithms. The proposed solution is finally verified by using three spherical datasets captured by both consumer-grade and professional spherical cameras. The results demonstrate that the proposed SfM workflow can achieve the successful 3D reconstruction of complex scenes and provide useful clues for the implementation in open-source software packages. The source code would be publicly available.

Physical Structure Expression for Dense Point Clouds of Magnetic Levitation Image Data.

The research and development of an intelligent magnetic levitation transportation system has become an important research branch of the current intelligent transportation system (ITS), which can provide technical support for state-of-the-art fields such as intelligent magnetic levitation digital twin. First, we applied unmanned aerial vehicle oblique photography technology to acquire the magnetic levitation track image data and preprocessed them. Then, we extracted the image features and matched them based on the incremental structure from motion (SFM) algorithm, recovered the camera pose parameters of the image data and the 3D scene structure information of key points, and optimized the bundle adjustment to output 3D magnetic levitation sparse point clouds. Then, we applied multiview stereo (MVS) vision technology to estimate the depth map and normal map information. Finally, we extracted the output of the dense point clouds that can precisely express the physical structure of the magnetic levitation track, such as turnout, turning, linear structures, etc. By comparing the dense point clouds model with the traditional building information model, experiments verified that the magnetic levitation image 3D reconstruction system based on the incremental SFM and MVS algorithm has strong robustness and accuracy and can express a variety of physical structures of magnetic levitation track with high accuracy.

MCSfM: Multi-Camera-Based Incremental Structure-From-Motion.

Fully perceiving the surrounding world is a vital capability for autonomous robots. To achieve this goal, a multi-camera system is usually equipped on the data collecting platform and the structure from motion (SfM) technology is used for scene reconstruction. However, although incremental SfM achieves high-precision modeling, it is inefficient and prone to scene drift in large-scale reconstruction tasks. In this paper, we propose a tailored incremental SfM framework for multi-camera systems, where the internal relative poses between cameras can not only be calibrated automatically but also serve as an additional constraint to improve the system robustness. Previous multi-camera based modeling work has mainly focused on stereo setups or multi-camera systems with known calibration information, but we allow arbitrary configurations and only require images as input. First, one camera is selected as the reference camera, and the other cameras in the multi-camera system are denoted as non-reference cameras. Based on the pose relationship between the reference and non-reference camera, the non-reference camera pose can be derived from the reference camera pose and internal relative poses. Then, a two-stage multi-camera based camera registration module is proposed, where the internal relative poses are computed first by local motion averaging, and then the rigid units are registered incrementally. Finally, a multi-camera based bundle adjustment is put forth to iteratively refine the reference camera and the internal relative poses. Experiments demonstrate that our system achieves higher accuracy and robustness on benchmark data compared to the state-of-the-art SfM and SLAM (simultaneous localization and mapping) methods.

Leveraging vocabulary tree for simultaneous match pair selection and guided feature matching of UAV images

Structure from Motion (SfM) has been a golden standard technique for UAV image orientation. However, the high combinational complexity of match pairs and the high outlier ratio of initial matches become two major issues in SfM-based image orientation. This paper presents an integrated workflow to achieve simultaneously match pair selection and guided feature matching for image orientation. The core idea of the proposed algorithm is to explore the index structure of both inverted and direct indexes in the context of vocabulary tree-based image retrieval. First, the similarity scores between one query image and database images are calculated by using the word-image index structure stored in the inverted index, and match pairs are selected with a distance or ratio threshold-independent strategy. Second, by using a large vocabulary tree for descriptor quantification, guided feature matching is achieved by using the image-word index structure stored in the direct index, which restricts the searching space of nearest descriptors and achieves the direct assignment of candidate matches, instead of the nearest neighbor searching (NNS) method. Finally, integrated with an incremental SfM workflow, the performance of the proposed algorithm has been comprehensively verified through the analysis and comparison by using four UAV datasets that cover varying test sites. The experimental results demonstrate that the proposed method achieves match pair selection with linear time complexity and provides refined matches with speedup ratios ranging from 156 to 228 (2 to 3) compared with the CPU (GPU)-based NNS matching method. In addition, it can achieve competitive precision in both relative bundle adjustment (BA) without ground control points (GCPs) and absolute BA with GCPs.

On Incremental Structure from Motion Using Lines

Humans tend to build environments with structure, which consists of mainly planar surfaces. From the intersection of planar surfaces arise straight lines. Lines have more degrees of freedom than points. Thus, line-based structure-from-motion (SfM) provides more information about the environment. In this article, we present solutions for SfM using lines, namely, incremental SfM. These approaches consist of designing state observers for a camera’s dynamical visual system looking at a 3-D line. We start by presenting a model that uses spherical coordinates for representing the line’s moment vector. We show that this parameterization has singularities, and, therefore, we introduce a more suitable model that considers the line’s moment and shortest viewing ray. Concerning the observers, we present two different methodologies. The first uses a memory-less state-of-the-art framework for dynamic visual systems. Since the previous states of the robotic agent are accessible—while performing the 3-D mapping of the environment—the second approach aims at exploiting the use of memory to improve the estimation accuracy and convergence speed. The two models and the two observers are evaluated in simulation and real data, where mobile and manipulator robots are used.

Parallel Structure From Motion for UAV Images via Weighted Connected Dominating Set

Incremental Structure from Motion (ISfM) has been widely used for UAV image orientation. Its efficiency, however, decreases dramatically due to the sequential constraint. Although the divide-and-conquer strategy has been utilized for efficiency improvement, cluster merging becomes difficult or depends on seriously designed overlap structures. This paper proposes an algorithm to extract the global model for cluster merging and designs a parallel SfM solution to achieve efficient and accurate UAV image orientation. First, based on vocabulary tree retrieval, match pairs are selected to construct an undirected weighted match graph, whose edge weights are calculated by considering both the number and distribution of feature matches. Second, an algorithm, termed weighted connected dominating set (WCDS), is designed to achieve the simplification of the match graph and build the global model, which incorporates the edge weight in the graph node selection and enables the successful reconstruction of the global model. Third, the match graph is simultaneously divided into compact and non-overlapped clusters. After the parallel reconstruction, cluster merging is conducted with the aid of common 3D points between the global and cluster models. Finally, by using three UAV datasets that are captured by classical oblique and recent optimized views photogrammetry, the validation of the proposed solution is verified through comprehensive analysis and comparison. The experimental results demonstrate that the proposed parallel SfM can achieve 17.4 times efficiency improvement and comparative orientation accuracy. In absolute BA, the geo-referencing accuracy is approximately 2.0 and 3.0 times the GSD (Ground Sampling Distance) value in the horizontal and vertical directions, respectively. For parallel SfM, the proposed solution is a more reliable alternative.

Camera Self-Calibration with GNSS Constrained Bundle Adjustment for Weakly Structured Long Corridor UAV Images

Camera self-calibration determines the precision and robustness of AT (aerial triangulation) for UAV (unmanned aerial vehicle) images. The UAV images collected from long transmission line corridors are critical configurations, which may lead to the “bowl effect” with camera self-calibration. To solve such problems, traditional methods rely on more than three GCPs (ground control points), while this study designs a new self-calibration method with only one GCP. First, existing camera distortion models are grouped into two categories, i.e., physical and mathematical models, and their mathematical formulas are exploited in detail. Second, within an incremental SfM (Structure from Motion) framework, a camera self-calibration method is designed, which combines the strategies for initializing camera distortion parameters and fusing high-precision GNSS (Global Navigation Satellite System) observations. The former is achieved by using an iterative optimization algorithm that progressively optimizes camera parameters; the latter is implemented through inequality constrained BA (bundle adjustment). Finally, by using four UAV datasets collected from two sites with two data acquisition modes, the proposed algorithm is comprehensively analyzed and verified, and the experimental results demonstrate that the proposed method can dramatically alleviate the “bowl effect” of self-calibration for weakly structured long corridor UAV images, and the horizontal and vertical accuracy can reach 0.04 m and 0.05 m, respectively, when using one GCP. In addition, compared with open-source and commercial software, the proposed method achieves competitive or better performance.

Robust hierarchical structure from motion for large-scale unstructured image sets

Structure from Motion (SfM) is key to mixed computer vision and photogrammetry applications. However, the fast-growing needs for large-scale SfM bring challenges to current SfM solutions. Unlike traditional global and incremental SfM solutions, hierarchical SfM approaches demonstrate promising potential in effectively reconstructing large-scale image sets by dividing the image set into multiple image clusters, reconstructing each cluster separately, and gradually merging partial models into a complete model. However, current hierarchical SfM approaches still suffer from the following problems: accurate image clustering without ancillary information; automatic quality evaluation of each reconstruction unit and unreliable partial reconstruction removal; effective and efficient reconstruction of each image cluster; robust and accurate cluster merging considering the merging order and the handling of images taken with the same camera but divided into different clusters. These unstable factors limit the robustness and accuracy of hierarchical SfM approaches on different unstructured image sets.To systematically improve the performance of hierarchical SfM, we propose a novel robust hierarchical structure from motion (RHSfM) method for large-scale image sets, which does not rely on any additional information, such as Global Positioning System (GPS) and Inertial Navigation System (INS). (1) We develop an automatic image clustering method based on image correlation and present a dynamic adjustment strategy, obtaining reliable image clustering results. (2) We remove the poor reconstructions by introducing multiple quality evaluation standards. (3) We put forward a fast incremental SfM algorithm that optimizes the image adding mode with an image pre-screening strategy and gets rid of the dependence by the proposed dynamic adjustment strategy. (4) We achieve accurate cluster merging by creating an optimal merging list and employing a stepwise global optimization strategy that merges structures first and then cameras. Significantly, the entire process is fully automated with only a few input parameters, and the final result is not sensitive to these parameters.We verify our method on various real image sets that cover different image conditions, different scenes, and different image scales, especially two large-scale image sets with 121,506 and 153,396 images, respectively. The experimental results reveal that our approach outperforms the state-of-the-art SfM systems Colmap, 3DF Samantha, and Metashape in terms of robustness, accuracy, and efficiency. In particular, only our method successfully reconstructed all the seven challenging datasets. For the five datasets that the other systems can also reconstruct, our method obtains the highest accuracy, which is 25 percent better than the best result of the comparable methods on average; for the remaining two datasets, the accuracy of our method is higher than 0.75 pixels. Moreover, the efficiency of our method is about 18, 4.85, and 0.25 times faster than Colmap, 3DF Samantha, and Metashape averagely on the experimental image sets, respectively. After all, our contribution provides a comprehensive and practical solution for large-scale SfM.

Progressive Structure from Motion by Iteratively Prioritizing and Refining Match Pairs

Structure from motion (SfM) has been treated as a mature technique to carry out the task of image orientation and 3D reconstruction. However, it is an ongoing challenge to obtain correct reconstruction results from image sets consisting of problematic match pairs. This paper investigated two types of problematic match pairs, stemming from repetitive structures and very short baselines. We built a weighted view-graph based on all potential match pairs and propose a progressive SfM method (PRMP-PSfM) that iteratively prioritizes and refines its match pairs (or edges). The method has two main steps: initialization and expansion. Initialization is developed for reliable seed reconstruction. Specifically, we prioritize a subset of match pairs by the union of multiple independent minimum spanning trees and refine them by the idea of cycle consistency inference (CCI), which aims to infer incorrect edges by analyzing the geometric consistency over cycles of the view-graph. The seed reconstruction is progressively expanded by iteratively adding new minimum spanning trees and refining the corresponding match pairs, and the expansion terminates when a certain completeness of the block is achieved. Results from evaluations on several public datasets demonstrate that PRMP-PSfM can successfully accomplish the image orientation task for datasets with repetitive structures and very short baselines and can obtain better or similar accuracy of reconstruction results compared to several state-of-the-art incremental and hierarchical SfM methods.

Incremental Rotation Averaging

In this paper, we present a simple yet effective rotation averaging pipeline, termed Incremental Rotation Averaging (IRA), which is inspired by the well-developed incremental Structure from Motion (SfM) techniques. Unlike the traditional rotation averaging methods which estimate all the absolute rotations simultaneously and focus on designing either robust loss function or outlier filtering strategy, here the absolute rotations are estimated in an incremental way. Similar to the incremental SfM, our IRA is robust to relative rotation outliers and could achieve accurate rotation averaging results. In addition, we propose several key techniques, such as initial triplet and Next-Best-View selection, Weighted Local/Global Optimization, and Re-Rotation Averaging, to push the rotation averaging results one step further. Ablation studies and comparison experiments on the 1DSfM, Campus, and San Francisco datasets demonstrate the effectiveness of our IRA and its advantages over the state-of-the-art rotation averaging methods in accuracy and robustness.

Efficient structure from motion for large-scale UAV images: A review and a comparison of SfM tools

Unmanned aerial vehicle (UAV) images have gained extensive attention in varying fields, and the Structure from Motion (SfM) technique has become the gold standard for aerial triangulation of UAV images. With increasing data volume caused by the use of multi-view and high-resolution imaging systems and the enhancement of UAV platform’s endurance, the capability for orientation of large-scale UAV images is becoming a prominent and necessary feature for SfM-based solutions. A classical SfM pipeline consists of three major steps, i.e., (i) feature extraction for an individual image, (ii) feature matching for each image pair, and (iii) parameter solving based on iterative bundle adjustment. Most of the time costs are consumed in the second and third steps. This can be explained from three main aspects. First, for feature matching the large number of images and high overlapping degrees cause high combinational complexity of match pairs. Second, the efficiency of commonly utilized techniques for outlier removal would be seriously degenerated because of high outlier ratios of initial matches. Third, for parameter solving of camera poses and scene structures, the iterative execution of bundle adjustment (BA) leads to high computational costs in the incremental SfM workflow. Thus, this paper gives a systematic survey of the state-of-the-art for match pair selection from both ordered and unordered datasets, for outlier removal of initial matches dominated by outliers, and for efficiency improvement of BA, and conducts an experimental evaluation for six well-known SfM-based software packages on UAV image orientation.

A Method of Hierarchical Image Retrieval for Real-Time Photogrammetry Based on Multiple Features

In near real-time photogrammetry, the first step in processing each new added image is determining the most relevant image in pre-sequence unordered images quickly and exactly, which is pivotal for accurate image matching and 3D reconstruction. This paper presents a hierarchical image retrieval algorithm based on multiple features and details the choice for representation of multiple features which is critical to the improvement of accuracy of this algorithm. First, we represent global features using AlexNet-FC7(fully connected layers) or ResNet101-Pool5(pooling layers) and local features using SIFT (scale-invariant feature transform) in two parallel threads with support of GPU (Graphics Processing Unit). Next, we obtain candidates based on cosine similarities between global features of each pre-sequence image and new added image. Finally, we determine the most relevant image from those candidates according to feature matching results for each candidate and new added image. The experimental results confirm that the second step is rather fast and the third step is necessary to tackle the problem that global features cannot distinguish objects from the same class. The total time our algorithm takes is about 83.6ms for determining the most relevant image in 5063 pre-sequence unordered images of size $1024\times 768$ , which outperforms exhaustive pairwise matching, Bag of Words and multi-vocabulary trees. Accuracy of our algorithm also perform better than the state-of-the-art methods on three benchmark datasets. SIFT matching results obtained in the third step after eliminating mismatches with RANSAC (Random Sample Consensus) can also be used for high-precision incremental SFM (Structure from Motion) reconstruction.

Efficient and robust large-scale structure-from-motion via track selection and camera prioritization

Incremental Structure-from-Motion (SfM) techniques have exhibited superior practicability in many recent studies; however, efficiency and robustness remain key challenges for these techniques. In this work, we propose a new incremental SfM method that overcomes these problems in a united framework that contains two iteration loops. The inner loop is a track selection loop, where a well-conditioned subset of the feature tracks is iteratively selected to accelerate the time-consuming bundle adjustment. The outer loop is a camera registration loop, where the a priori camera rotations are estimated via rotation averaging on multiple orthogonal maximum spanning trees (OMSTs) of the view-graph and used as weak supervision for the registration. The calibrated camera poses that agree with the a priori camera rotations are preferentially registered, and after all the consistent cameras have been calibrated, the remaining cameras are incrementally registered. The results of extensive experiments demonstrate that our system can reconstruct both general and ambiguous image datasets, and our system outperforms many state-of-the-art SfM systems in terms of efficiency and robustness.

Improved SfM-Based Indoor Localization with Occlusion Removal

This article describes a novel 3D image-based indoor localization system integrated with an improved SfM (structure from motion) approach and an obstacle removal component. In contrast with existing state-of-the-art localization techniques focusing on static outdoor or indoor environments, the adverse effects, generated by moving obstacles in busy indoor spaces, are considered in this work. In particular, the problem of occlusion removal is converted into a separation problem of moving foreground and static background. A low-rank and sparse matrix decomposition approach is used to solve this problem efficiently. Moreover, a SfM with RT (re-triangulation) is adopted in order to handle the drifting problem of incremental SfM method in indoor scene reconstruction. To evaluate the performance of the system, three data sets and the corresponding query sets are established to simulate different states of the indoor environment. Quantitative experimental results demonstrate that both query registration rate and localization accuracy increase significantly after integrating the authors' improvements.

Brunovsky's Linear Form of Incremental Structure From Motion

Conventional active incremental structure from motion (ISfM) schemes require a precise knowledge of the linear and angular velocities of the vision system to compute the three-dimensional structure of the observed scene. Furthermore, they are generally coupled and nonlinear, which makes the reconstruction inaccurate. In this paper, we present a novel active ISfM scheme to overcome these difficulties. It relies on two appropriate transformations: the first one allows efficient decoupling of the nonlinear model, whereas the second one transforms the decoupled model into the Brunovsky's linear form. We then show that a Luenberger's observer can be designed to estimate the unavailable states, making our scheme globally asymptotically convergent and robust to noise on measurements. We present simulated and experimental results to illustrate the performance of the described methodology.

Towards an efficient 3D model estimation methodology for aerial and ground images

In this paper we propose an efficient approach for automatic generation of 3D models from images based on structure from motion (SfM) and multi-view stereo reconstruction techniques. Current imaging devices are capable of producing high-definition images and are an ubiquitous payload of unmanned aerial vehicles. However, the time required to obtain models quickly becomes prohibitive as the number of images increases. In our approach, which is image-based only, we use meta-data information such as GPS, keypoint filtering and multiple local bundle adjustment refinement instead of global optimization in a novel scheme to speed up the incremental SfM process. The results from real data show that our approach outperforms the time performance of current strategies while maintaining the quality of the resulting model. Experiments with an unorganized set of images were also conducted, and the results show that our method is able to efficiently estimate 3D models from collections of images with reduced re-projection error.

Learning a generic 3D face model from 2D image databases using incremental Structure-from-Motion

Over the last decade 3D face models have been extensively used in many applications such as face recognition, facial animation and facial expression analysis. 3D Morphable Models (MMs) have become a popular tool to build and fit 3D face models to images. Critical to the success of MMs is the ability to build a generic 3D face model. Major limitations in the MMs building process are: (1) collecting 3D data usually involves the use of expensive laser scans and complex capture setups, (2) the number of available 3D databases is limited, and typically there is a lack of expression variability and (3) finding correspondences and registering the 3D model is a labor intensive and error prone process. This paper proposes an incremental Structure-from-Motion (SfM) approach to learn a generic 3D face model from large collections of existing 2D hand-labeled images containing many subjects under different expressions and poses. The two major contributions of the paper are: (1) learning a generic 3D deformable face model from 2D databases and (2) incorporating a prior subspace into the incremental SfM formulation to provide robustness to noise, missing data and degenerate shape configurations. Experimental results on the CMU-PIE database show improvements in the generalization of the 3D face model across expression and identity.