Structure From Motion Pipeline Research Articles

On-the-Fly SfM: What you capture is What you get

Abstract. Over the last decades, ample achievements have been made on Structure from Motion (SfM). However, the vast majority of them basically work in an offline manner, i.e., images are firstly captured and then fed together into a SfM pipeline for obtaining poses and sparse point cloud. In this work, on the contrary, we present an on-the-fly SfM: running online SfM while image capturing, the newly taken On-the-Fly image is online estimated with the corresponding pose and points, i.e., what you capture is what you get. Specifically, our approach firstly employs a vocabulary tree that is unsupervised trained using learning-based global features for fast image retrieval of newly fly-in image. Then, a robust feature matching mechanism with least squares (LSM) is presented to improve image registration performance. Finally, via investigating the influence of newly fly-in image’s connected neighboring images, an efficient hierarchical weighted local bundle adjustment (BA) is used for optimization. Extensive experimental results demonstrate that on-the-fly SfM can meet the goal of robustly registering the images while capturing in an online way.

Differential SfM and image correction for a rolling shutter stereo rig

• A simple and flexible generalized RS stereo differential SfM algorithm over two consecutive frames. • Two effective and effcient RS-stereo non-linear optimization techniques based on the maximum likelihood criterion. • An RS-stereo image correction method to remove the inaccuracies induced by the RS effect. • The proposed model and method are universal and tractable. Most modern consumer-grade cameras are equipped with an electronic rolling shutter (RS), leading to image distortions when the camera moves during image acquisition. We explore the first structure and motion estimation problem of a dynamic generalized RS stereo camera. Such a general configuration is commonplace in robots and autonomous driving applications. We propose a tractable RS stereo differential structure from motion (SfM) algorithm, taking into account the RS effect during consecutive imaging, which effectively compensates for the RS-stereo image distortion by a linear scaling operation on each optical flow. We further propose embedding the cheirality into RANSAC and develop a robust RS-stereo-aware full-motion estimation framework. We demonstrate that the RS stereo motion and depth map refined by our non-linear optimization schemes within the maximum likelihood criterion can be used for image correction to recover high-quality global shutter (GS) stereo images. Moreover, using the proposed generalized RS stereo differential SfM pipeline, the corrected images produce an accurate 3D scene structure as the ground-truth structure. Extensive experiments on both synthetic and real RS stereo data demonstrate the effectiveness of our model and method in various configurations.

Multi-Camera Extrinsic Calibration for Real-Time Tracking in Large Outdoor Environments

Calibrating intrinsic and extrinsic camera parameters is a fundamental problem that is a preliminary task for a wide variety of applications, from robotics to computer vision to surveillance and industrial tasks. With the advent of Internet of Things (IoT) technology and edge computing capabilities, the ability to track motion activities in large outdoor areas has become feasible. The proposed work presents a network of IoT camera nodes and a dissertation on two possible approaches for automatically estimating their poses. One approach follows the Structure from Motion (SfM) pipeline, while the other is marker-based. Both methods exploit the correspondence of features detected by cameras on synchronized frames. A preliminary indoor experiment was conducted to assess the performance of the two methods compared to ground truth measurements, employing a commercial tracking system of millimetric precision. Outdoor experiments directly compared the two approaches on a larger setup. The results show that the proposed SfM pipeline more accurately estimates the pose of the cameras. In addition, in the indoor setup, the same methods were used for a tracking application to show a practical use case.

A Quadrifocal Tensor SFM Photogrammetry Positioning and Calibration Technique for HOFS Aerial Sensors

Nowadays, the integration between photogrammetry and structure from motion (SFM) has become much closer, and many attempts have been made to combine the two approaches to realize the positioning, calibration, and 3D reconstruction of a large number of images. For the positioning and calibration of high oblique frame sweep (HOFS) aerial cameras, a quadrifocal tensor SFM photogrammetry technique is proposed to resolve the positioning and calibration task of such cameras. It adopts the quadrifocal tensor idea into the OpenMVG SFM pipeline to solve the complexity problem caused by the small single-viewing imaging area and the high image overlapping ratio. It also integrates the photogrammetry iteration idea into the OpenMVG SFM pipeline to enhance the positioning and calibration accuracy, which includes a coarse to fine three-stage Bundle Adjustment (BA) processing approach. In this paper, the overall workflow of the proposed technique was first introduced in detail, from feature extraction and image matching, relative rotation and translation estimation, global rotation and translation estimation, and the quadrifocal tensor model construction to the three-stage BA process and calibration. Then, experiments were carried out in the Zhengzhou area, implementing four types of adjustment methods. The results suggest that the proposed quadrifocal tensor SFM photogrammetry is suitable for large tilt frame sweep camera positioning and calibration without prior information on detailed camera intrinsic parameters and structure. The modifications made to the OpenMVG SFM pipeline enhanced the precision of image positioning and calibration and provided the precision level of professional photogrammetry software.

Structure-From-Motion With Varying Principal Point

We consider the problem of structure-from-motion (SfM) for images with fixed calibration but varying principal point. This scenario occurs for archival imagery taken using historic glass plate and film cameras without fiducial markers, when images have been inconsistently cropped or when image plates are broken into multiple fragments.We derive initialisation and pose estimation methods and regularisation penalties tuned specifically for this scenario leading to a complete archival imagery SfM pipeline. We illustrate the performance of our methods on challenging real world examples from image archives. Specifically, we use archival images of the East coast of Greenland from the British Arctic Air Route Expedition (BAARE). This is of particular glaciological interest for measuring historic ice loss. We use a modern digital elevation model (ArcticDEM), masked to stable regions, as ground truth to evaluate our method.

View-graph construction framework for robust and efficient structure-from-motion

A view-graph is vital for both the accuracy and robustness of structure-from-motion (SfM). Conventional matrix decomposition techniques treat all edges of view-graph equally; hence, many edge outliers are produced in matching pairs with fewer feature matches. To address this problem, we propose an incremental framework for view-graph construction, where the robustness of matched pairs that have a larger number of feature matches is propagated to their connected images. Given pairwise feature matches, a verified maximum spanning tree (VMST) is first constructed; for each edge in the VMST, we perform a local reconstruction and register its visible cameras. Based on the local reconstruction, pairwise relative geometries are computed and some new epipolar edges are produced. In this way, these newly computed edges inherit the robustness and accuracy of VMST, and by embedding them into VMST, our view-graph is constructed. We feed our view-graph into a standard SfM pipeline and compare this newly formed system with many of state-of-the-art SfM methods. The experimental results demonstrate that our view-graph provides a better foundation for conventional SfM systems, and enables them to reconstruct both general and ambiguous images.

Efficient structure from motion for large-scale UAV images: A review and a comparison of SfM tools

Unmanned aerial vehicle (UAV) images have gained extensive attention in varying fields, and the Structure from Motion (SfM) technique has become the gold standard for aerial triangulation of UAV images. With increasing data volume caused by the use of multi-view and high-resolution imaging systems and the enhancement of UAV platform’s endurance, the capability for orientation of large-scale UAV images is becoming a prominent and necessary feature for SfM-based solutions. A classical SfM pipeline consists of three major steps, i.e., (i) feature extraction for an individual image, (ii) feature matching for each image pair, and (iii) parameter solving based on iterative bundle adjustment. Most of the time costs are consumed in the second and third steps. This can be explained from three main aspects. First, for feature matching the large number of images and high overlapping degrees cause high combinational complexity of match pairs. Second, the efficiency of commonly utilized techniques for outlier removal would be seriously degenerated because of high outlier ratios of initial matches. Third, for parameter solving of camera poses and scene structures, the iterative execution of bundle adjustment (BA) leads to high computational costs in the incremental SfM workflow. Thus, this paper gives a systematic survey of the state-of-the-art for match pair selection from both ordered and unordered datasets, for outlier removal of initial matches dominated by outliers, and for efficiency improvement of BA, and conducts an experimental evaluation for six well-known SfM-based software packages on UAV image orientation.

PHOTOGRAMMETRIC SOLUTIONS FOR 3D MODELING OF CULTURAL HERITAGE SITES IN REMOTE AREAS

Abstract. The realistic possibility of using non-metric digital cameras to achieve reliable 3D models has eased the application of photogrammetry in different domains. Documentation, conservation and dissemination of the Cultural Heritage (CH) can be obtained and implemented through virtual copies and replicas. Structure-from-Motion (SfM) photogrammetry has widely proven its impressive potential for image-based 3D reconstruction resulting in great 3D point clouds’ acquisitions but at minimal cost. Images from Unmanned Aerial Vehicles (UAVs) can be also processed within SfM pipeline to obtain point cloud of Cultural Heritage sites in remote regions. Both aerial and terrestrial images can be integrated to obtain a more complete 3D. In this paper, the application of SfM photogrammetry for surveying of the Ziggurat Chogha Zanbil in Iran is presented. Here point clouds have been derived from oblique and nadir photos captured from UAV as well as terrestrial photos. The obtained four point clouds have been compared on the basis of different techniques to highlight differences among them.

SEMANTIC ROAD SCENE KNOWLEDGE FOR ROBUST SELF-CALIBRATION OF ENVIRONMENT-OBSERVING VEHICLE CAMERAS

Abstract. Environment-observing vehicle camera self-calibration using a structure from motion (SfM) algorithm allows calibration over vehicle lifetime without the need of special calibration objects being present in the calibration images. Scene-specific problems with feature-based correspondence search and reconstruction during the SfM pipeline might be caused by critical objects like moving objects, poor-texture objects or reflecting objects and might have negative influence on camera calibration. In this contribution, a method to use semantic road scene knowledge by means of semantic masks for a semantic-guided SfM algorithm is proposed to make the calibration more robust. Semantic masks are used to exclude image parts showing critical objects from feature extraction, whereby semantic knowledge is obtained by semantic segmentation of the road scene images. The proposed method is tested with an image sequence recorded in a suburban road scene. It has been shown that semantic guidance leads to smaller deviations of the estimated interior orientation and distortion parameters from reference values obtained by test field calibration compared to a standard SfM algorithm.

Mapping Cultural Heritage in Coastal Areas with UAS: The Case Study of Lesvos Island

Dynamic processes in coastal zones and human activities in the coastal environment produce pressure on cultural heritage, especially in touristic places. Unmanned aerial systems (UAS) are used as an additional tool for monitoring cultural heritage sites in sensitive coastal areas. UASs provide low-cost accurate spatial data and high-resolution imagery products in various spatial and temporal scales. The use of UAS for mapping cultural heritage sites in the coastal zone is of increasing interest among scientists and archaeologists in terms of monitoring, documentation, mapping, and restoration. This study outlines the integration of UAS data acquisition and structure from motion (SfM) pipeline for the visualization of selected cultural heritage areas (ancient harbors) in the coastal zone. The UAS-SfM methodology produces very detailed orthophoto maps for mapping and detecting cultural heritage sites. Additionally, a metadata cataloging system has been developed in order to facilitate online searching operations for all products of the data acquisition, SfM pipeline, and cartographic processes. For this reason, a specific metadata profile was implemented, based on the European INSPIRE framework. As a result, datasets reusability and catalogs interoperability are promoted.

Extreme Feature Regions Detection and Accurate Quality Assessment for Point-Cloud 3D Reconstruction

The 3-D reconstruction methods based on structure from motion (SfM) pipeline mainly use the traditional scale invariant feature detecting methods for feature matching. This type of methods suffers from the accuracy with affine matching in the image-based modeling system. In this paper, we propose an affine-invariant feature detection and matching method which is accurate and fast based on the three types of critical points in Morse theory called precise extreme feature region (PEFR). We also exploit the evaluation method for the reconstruction results without the ground truth. We propose the average descending rate to quantitatively evaluate the accuracy of the reconstruction model. Using PEFR in an SfM pipeline to build 3-D point-cloud, our method presents better results and less runtime compared to the state-of-the-art approaches. In addition, our quality assessment method presents a practical comparison of the reconstruction result based on the geometric accuracy of the reconstruction models. The experimental results prove that our PEFR method is faster and more accurate and works well on the reconstruction pipeline. The assessment method is fit for the quality assessment on 3-D reconstruction.

Structure from Motion of Underwater Scenes Considering Image Degradation and Refraction

Abstract Structure from Motion (SfM) can reconstruct three-dimensional (3D) structures using only image sequences. However, SfM applied to the underwater environment is different from the air environment because image formation in underwater environments suffers from two major problems: image degradation and refraction. Consequently, images captured in underwater environments are loss of contrast, hazy and distorted geometrically. To achieve an accurate 3D reconstruction in such underwater environments, image degradation and refraction problems should be solved simultaneously. In this work, we propose a systematic SfM pipeline containing image enhancement and refraction reduced SfM. The image enhancement part improves the image quality for the following steps in the reconstruction part. We confirm the effectiveness of the proposed pipeline using images captured by a submerged robot in an extreme underwater environment, the disaster area of Unit 3 Primary Containment Vessel (PCV) at Fukushima Daiichi Nuclear Power Station. Experimental results show that our proposed method can successfully reconstruct the structures in that extreme underwater environment.

Evaluating the Performance of Structure from Motion Pipelines

Structure from Motion (SfM) is a pipeline that allows three-dimensional reconstruction starting from a collection of images. A typical SfM pipeline comprises different processing steps each of which tackles a different problem in the reconstruction pipeline. Each step can exploit different algorithms to solve the problem at hand and thus many different SfM pipelines can be built. How to choose the SfM pipeline best suited for a given task is an important question. In this paper we report a comparison of different state-of-the-art SfM pipelines in terms of their ability to reconstruct different scenes. We also propose an evaluation procedure that stresses the SfM pipelines using real dataset acquired with high-end devices as well as realistic synthetic dataset. To this end, we created a plug-in module for the Blender software to support the creation of synthetic datasets and the evaluation of the SfM pipeline. The use of synthetic data allows us to easily have arbitrarily large and diverse datasets with, in theory, infinitely precise ground truth. Our evaluation procedure considers both the reconstruction errors as well as the estimation errors of the camera poses used in the reconstruction.

CROWD-SOURCED VISUAL DATA COLLECTION FOR MONITORING INDOOR CONSTRUCTION IN 3D

Complete and accurate 3D monitoring of indoor construction progress using visual data is challenging. It requires (a) capturing a large number of overlapping images, which is time-consuming and labor-intensive to collect, and (b) processing using Structure from Motion (SfM) algorithms, which can be computationally expensive. To address these inefficiencies, this paper proposes a hybrid SfM-SLAM 3D reconstruction algorithm along with a decentralized data collection workflow to map indoor construction work locations in 3D and any desired frequency. The hybrid 3D reconstruction method is composed of a pipeline of Structure from Motion (SfM) coupled with Multi-View Stereo (MVS) to generate 3D point clouds and a SLAM (Simultaneous Localization and Mapping) algorithm to register the separately formed models together. Our SfM and SLAM pipelines are built on binary Oriented FAST and Rotated BRIEF (ORB) descriptors to tightly couple these two separate reconstruction workflows and enable fast computation. To elaborate the data capture workflow and validate the proposed method, a case study was conducted on a real-world construction site. Compared to state-of-the-art methods, our preliminary results show a decrease in both registration error and processing time, demonstrating the potential of using daily images captured by different trades coupled with weekly walkthrough videos captured by a field engineer for complete 3D visual monitoring of indoor construction operations.

Structure from motion using dense CNN features with keypoint relocalization

Structure from motion (SfM) using imagery that involves extreme appearance changes is yet a challenging task due to a loss of feature repeatability. Using feature correspondences obtained by matching densely extracted convolutional neural network (CNN) features significantly improves the SfM reconstruction capability. However, the reconstruction accuracy is limited by the spatial resolution of the extracted CNN features which is not even pixel-level accuracy in the existing approach. Providing dense feature matches with precise keypoint positions is not trivial because of memory limitation and computational burden of dense features. To achieve accurate SfM reconstruction with highly repeatable dense features, we propose an SfM pipeline that uses dense CNN features with relocalization of keypoint position that can efficiently and accurately provide pixel-level feature correspondences. Then, we demonstrate on the Aachen Day-Night dataset that the proposed SfM using dense CNN features with the keypoint relocalization outperforms a state-of-the-art SfM (COLMAP using RootSIFT) by a large margin.

THERMAL TEXTURE GENERATION AND 3D MODEL RECONSTRUCTION USING SFM AND GAN

Abstract. Realistic 3D models with textures representing thermal emission of the object are widely used in such fields as dynamic scene analysis, autonomous driving, and video surveillance. Structure from Motion (SfM) methods provide a robust approach for the generation of textured 3D models in the visible range. Still, automatic generation of 3D models from the infrared imagery is challenging due to an absence of the feature points and low sensor resolution. Recent advances in Generative Adversarial Networks (GAN) have proved that they can perform complex image-to-image transformations such as a transformation of day to night and generation of imagery in a different spectral range. In this paper, we propose a novel method for generation of realistic 3D models with thermal textures using the SfM pipeline and GAN. The proposed method uses visible range images as an input. The images are processed in two ways. Firstly, they are used for point matching and dense point cloud generation. Secondly, the images are fed into a GAN that performs the transformation from the visible range to the thermal range. We evaluate the proposed method using real infrared imagery captured with a FLIR ONE PRO camera. We generated a dataset with 2000 pairs of real images captured in thermal and visible range. The dataset is used to train the GAN network and to generate 3D models using SfM. The evaluation of the generated 3D models and infrared textures proved that they are similar to the ground truth model in both thermal emissivity and geometrical shape.

Structure from motion with efficient homography-based line matching.

A novel structure from motion (SfM) pipeline that combines point features and line features together is presented in this paper. Line segments usually are more difficult to handle than points in SfM due to the instability of their end points and the property of being unconstrained by epipolar geometry. Our proposed method first implements a very robust and efficient line segment matching by applying homography and optical flow tracking. Global homography and multiple homography schemes are combined to get a better effect. By virtue of sampling lines, line matching is converted to point matching. A unified global SfM pipeline is then available for both lines and points. In the end, the proposed method is evaluated through various datasets. The result shows that our line-based SfM can generate accurate 3D models with lines and give meaningful representation for a man-made environment.

Tracks selection for robust, efficient and scalable large-scale structure from motion

Currently global structure-from-motion (SfM) pipeline consists of four steps: estimating camera rotations first, then computing camera positions, triangulating tracks, and finally doing bundle adjustment. However, for large-scale SfM problems, the tracks are usually too noisy and redundant for the bundle adjustment. Thus in this work, we propose a novel fast tracks selection method to improve both efficiency and robustness of the bundle adjustment. Firstly, three selection criteria: Compactness, Accurateness, and Connectedness, are introduced, where the first two are to calculate a selection priority for each track and the third is to guarantee the completeness of scene structure. Then, to satisfy these criteria, a more informative subset of tracks is selected by covering multiple spanning trees of epipolar geometry graph. Since tracks selection acts only an intermediate step in the whole SfM pipeline, it can be in principle embedded into any global SfM pipelines. To validate the effectiveness of our tracks selection module, we insert it into a state-of-the-art global SfM system and compare it with three other selection methods. Extensive experiments show that by embedding our tracks selection module, the new SfM system performs similarly or better than the original one in terms of reconstruction completeness and accuracy, but is much more efficient and scalable for large-scale scene reconstructions. Finally, our tracks selection module is further embedded into two other global SfM systems to demonstrated its versatility.

Coastline Zones Identification and 3D Coastal Mapping Using UAV Spatial Data

Spatial data acquisition is a critical process for the identification of the coastline and coastal zones for scientists involved in the study of coastal morphology. The availability of very high-resolution digital surface models (DSMs) and orthophoto maps is of increasing interest to all scientists, especially those monitoring small variations in the earth’s surface, such as coastline morphology. In this article, we present a methodology to acquire and process high resolution data for coastal zones acquired by a vertical take off and landing (VTOL) unmanned aerial vehicle (UAV) attached to a small commercial camera. The proposed methodology integrated computer vision algorithms for 3D representation with image processing techniques for analysis. The computer vision algorithms used the structure from motion (SfM) approach while the image processing techniques used the geographic object-based image analysis (GEOBIA) with fuzzy classification. The SfM pipeline was used to construct the DSMs and orthophotos with a measurement precision in the order of centimeters. Consequently, GEOBIA was used to create objects by grouping pixels that had the same spectral characteristics together and extracting statistical features from them. The objects produced were classified by fuzzy classification using the statistical features as input. The classification output classes included beach composition (sand, rubble, and rocks) and sub-surface classes (seagrass, sand, algae, and rocks). The methodology was applied to two case studies of coastal areas with different compositions: a sandy beach with a large face and a rubble beach with a small face. Both are threatened by beach erosion and have been degraded by the action of sea storms. Results show that the coastline, which is the low limit of the swash zone, was detected successfully by both the 3D representations and the image classifications. Furthermore, several traces representing previous sea states were successfully recognized in the case of the sandy beach, while the erosion and beach crests were detected in the case of the rubble beach. The achieved level of detail of the 3D representations revealed new beach characteristics, including erosion crests, berm zones, and sand dunes. In conclusion, the UAV SfM workflow provides information in a spatial resolution that permits the study of coastal changes with confidence and provides accurate 3D visualizations of the beach zones, even for areas with complex topography. The overall results show that the presented methodology is a robust tool for the classification, 3D visualization, and mapping of coastal morphology.

Multi-view structure-from-motion for hybrid camera scenarios

We describe a pipeline for structure-from-motion (SfM) with mixed camera types, namely omnidirectional and perspective cameras. For the steps of this pipeline, we propose new approaches or adapt the existing perspective camera methods to make the pipeline effective and automatic. We model our cameras of different types with the sphere camera model. To match feature points, we describe a preprocessing algorithm which significantly increases scale invariant feature transform (SIFT) matching performance for hybrid image pairs. With this approach, automatic point matching between omnidirectional and perspective images is achieved. We robustly estimate the hybrid fundamental matrix with the obtained point correspondences. We introduce the normalization matrices for lifted coordinates so that normalization and denormalization can be performed linearly for omnidirectional images. We evaluate the alternatives of estimating camera poses in hybrid pairs. A weighting strategy is proposed for iterative linear triangulation which improves the structure estimation accuracy. Following the addition of multiple perspective and omnidirectional images to the structure, we perform sparse bundle adjustment on the estimated structure by adapting it to use the sphere camera model. Demonstrations of the end-to-end multi-view SfM pipeline with the real images of mixed camera types are presented.