SFM Methods Research Articles

View‐graph key‐subset extraction for efficient and robust structure from motion

AbstractStructure from motion (SfM) is used to recover camera poses and the sparse structure of real scenes from multiview images. SfM methods construct a view‐graph from the matching relationships of images. Redundancy and incorrect edges are usually observed in it. Redundancy inhibits the efficiency and incorrect edges result in the misalignment of structures. In addition, the uneven distribution of vertices usually affects the global accuracy. To address these problems, we propose a coarse‐to‐fine approach in which the poses of an extracted key‐subset of images are first computed and then all remaining images are oriented. The core of this approach is view‐graph key‐subset extraction, which not only prunes redundant data and incorrect edges but also obtains properly distributed key‐subset vertices. The extraction approach is based on a replaceability score and an iteration‐update strategy. In this way, only vertices with high SfM importance are preserved in the key‐subset. Different public datasets are used to evaluate our approach. Due to the absence of ground‐truth camera poses in large‐scale datasets, we present new datasets with accurate camera poses and point clouds. The results demonstrate that our approach greatly increases the efficiency of SfM. Furthermore, the robustness and accuracy can be improved.

Elliptical specularity detection in endoscopy with application to normal reconstruction.

To detect specularities as elliptical blobs in endoscopy. The rationale is that in the endoscopic setting, specularities are generally small and that knowing the ellipse coefficients allows one to reconstruct the surface normal. In contrast, previous works detect specular masks as free-form shapes and consider the specular pixels as nuisance. A pipeline combining deep learning with handcrafted steps for specularity detection. This pipeline is general and accurate in the context of endoscopic applications involving multiple organs and moist tissues. A fully convolutional network produces an initial mask which specifically finds specular pixels, being mainly composed of sparsely distributed blobs. Standard ellipse fitting follows for local segmentation refinement in order to only keep the blobs fulfilling the conditions for successful normal reconstruction. Convincing results in detection and reconstruction on synthetic and real images, showing that the elliptical shape prior improves the detection itself in both colonoscopy and kidney laparoscopy. The pipeline achieved a mean Dice of 84% and 87% respectively in test data for these two use cases, and allows one to exploit the specularities as useful information for inferring sparse surface geometry. The reconstructed normals are in good quantitative agreement with external learning-based depth reconstruction methods manifested, as shown by an average angular discrepancy of [Formula: see text] in colonoscopy. First fully automatic method to exploit specularities in endoscopic 3D reconstruction. Because the design of current reconstruction methods can vary considerably for different applications, our elliptical specularity detection could be of potential interest in clinical practice thanks to its simplicity and generalisability. In particular, the obtained results are promising towards future integration with learning-based depth inference and SfM methods.

Possible approaches for processing of spherical images using SfM

Spherical cameras are being used more frequently in surveying because of their low cost and possibility to process spherical images with conventional SfM methods. Fish-eye lenses are now frequently found on modern 360° cameras, allowing for the capture of the entire scene and subsequent model reconstitution by a non-photogrammetrist specialist. Gyrospherics are a feature of cameras that ensure image stabilization to reduce blur when the camera is moving. This feature allows it to capture moving scenes in time-lapse mode. However, two main factors - hardware parameters and software algorithms - affect the quality of the images that are captured. The 360° camera design allows for a variety of data processing methods by SfM. These images were created using multiple image sensors and lenses from each 360° camera. These images can be processed individually or by applying rules to define relationships between images. Also, images from cameras can be stitched and processed with a spherical camera model. In this paper is proposed processing methods of data from 360° cameras and estimated accuracy of each method.

Photogrammetry-Based Volume Measurement Framework for the Particle Density Estimation of LECA.

This paper presents a photogrammetry-based volume measurement framework for the particle density estimation of Lightweight expanded clay aggregate (LECA). The results are compared with computed tomography (CT) and Archimedes’ method measurements. All of the steps required in order to apply the proposed approach are explained. Next, we discuss how the interpretation of open pores affects the results of volume measurements. We propose to process the shapes obtained from different methods by applying an Ambient Occlusion algorithm with the same threshold, t = 0.175. The difference between the CT and SfM methods is less than 0.006 g/cm3, proving that the photogrammetry-based approach is accurate enough. The Archimedes’ method significantly overestimates the density of the particles. Nevertheless, its accuracy is acceptable for most engineering purposes. Additionally, we evaluate the accuracy of shape reconstruction (in terms of the Hausdorff distance). For 95% of the grain’s surface, the maximum error is between 0.073 mm and 0.129 mm (depending on the grain shape). The presented approach is helpful for measuring the particle density of porous aggregates. The proposed methodology can be utilized in order to estimate intergranular porosity, which is valuable information for the calibration of DEM models.

Integration of terrestrial laser scanning and structure from motion for the assessment of industrial chimney geometry

The idea of this study is to attempt to combine the TLS and SfM methods for the measurement of a 120 m industrial chimney. The integrated model should have full point cloud coverage, maintaining an accuracy comparable to laser scanning. TLS was used to construct a coherent model of the lower parts of the chimney, to which the point cloud from the SfM method was then attached. For full point coverage, internal consistency of a few millimeters was achieved for each method and the external consistency was around 10 mm. As elevation increased causing uneven loss of scanner range, the consistency of the TLS deteriorated. However, integration of both measurement methods was possible even in areas of partial point cloud coverage. Sufficient coverage occurred on the length of about 55 m and allowed to integrate clouds with an average error of 13 mm.

A Deep Learning-Based Hyperspectral Keypoint Representation Method and Its Application for 3D Reconstruction

In previous studies, the keypoints and their feature descriptors of a hyperspectral image have not received enough attention. Compared with RGB images, the rich feature information of a hyperspectral image can not only constrain the spatial positions of the keypoints, but also provide rich spectral features for their descriptors. In this paper, we propose a deep neural network method called Hyperspectral SuperPoint Network (HSSPN) to detect the keypoints from a hyperspectral image and generate the corresponding feature descriptors. The HSSPN method contains a Shared Encoder (which is responsible for fully extracting the features of the spatial and spectral dimensions) and two decoders: the Keypoint Decoder accurately detects the keypoints according to the feature map outputted by the Shared Encoder, while the Descriptor Decoder is responsible for generating stable feature descriptors. We collected some hyperspectral images with an electronically controlled turntable, and then annotated them with a Hyperspectral Homographic Adaptation (HHA) network. The trained HSSPN gave an excellent keypoint matching performance on the test set. Hyperspectral 3D reconstruction can obtain information in a 3D space and a spectral space at the same time. Most of the previous reconstruction methods are separated for the treatments of 3D reconstruction and hyperspectral imaging, which require complicated registrations. In this study, we propose a SfM-based hyperspectral 3D reconstruction method, in which we first collect hyperspectral images from multiple viewing angles, and then apply the HSSPN method to detect the keypoints and calculate their feature descriptors. These keypoints and feature descriptors are then used to reconstruct each spectral image of the target with the SfM method. The 3D reconstruction models at these wavelengths are simply superimposed to obtain a hyperspectral 3D reconstruction model. Compared with conventional SIFT-based SfM methods, our method not only contains richer spectral information, but also achieves better results in the reconstruction accuracy.

Quick and Dirty (and Accurate) 3D Paleoseismic Trench Models Using Coded Scale Bars

Abstract Structure-from-motion (SfM) modeling has dramatically increased the speed of generating geometrically accurate orthophoto mosaics of paleoseismic trenches, but some aspects of this technique remain time and labor intensive. Model accuracy relies on control points to establish scale, reduce distortion, and orient 3D models. Traditional SfM methods use total station or Global Navigation Satellite System (GNSS) surveys to constrain models, but collecting control points along a vertical trench wall is often inhibited by poor line of sight to the survey sensor or limited sky view and requires many hours in the field and office. We used physical scale bars printed with coded targets to constrain SfM models of a dusty, 46-m-long trench excavation across the Teton fault (Wyoming, U.S.A.). We present a workflow for generating quick and accurate 3D SfM models and orthophoto mosaics and compare the effectiveness of using scale bar, GNSS, and total-station control in the models. Our results show that the scale bar model deviates from total station survey points by an average of 3.1 cm (maximum of 5.3 cm). In addition, the scale-bar model only deviates an average of 1.7 cm (maximum 3.5 cm) when compared to the best model alternative, the SfM model controlled by the total station survey. Scale bars eliminate several hours needed to collect and incorporate control points from total station or GNSS surveys and significantly simplify the workflow, at the cost of slightly increased 3D model and orthophoto mosaic error. Our results further suggest that trench models can be constrained with at least four physical scale bars, but using five to six physical scale bars provides redundant control for field deployment and model optimization. The scale bar method for paleoseismic trenches proves to be portable and fast, minimizes the need for specialized survey equipment, and maintains model accuracy needed for mapping trench walls.

A hybrid global structure from motion method for synchronously estimating global rotations and global translations

Over the last few decades, the methods of global image orientation, which is also called global SfM, have attracted a lot of attention from researchers, mainly thanks to its advantage of time efficiency. Based on the input of relative orientation results, most conventional global SfM methods employ a two-step strategy consisting of global rotation estimation and global translation estimation. This paper, on the contrary, introduces a hybrid global approach that intends to solve global rotations and translations synchronously, but hierarchically. To improve the robustness and time efficiency, we first propose a novel efficient method that is much faster than the previous approaches for extracting an optimal minimum cover of a connected image triplet set (OMCTS). The OMCTS makes all the available images contained in a minimum number of connected image triplets, as well as all of those selected triplets, satisfy the constraint that the three corresponding relative orientations are as compatible as possible to each other. In order to solve non-collinear triplets in the OMCTS, some fundamental characterizations of essential matrices in the multiple-image setting are used, and image pose parameters are then estimated via averaging the constrained essential matrices. For the collinear triplets, the above approach is invalid and the image pose parameters are then alternatively determined from the relative orientations using the depth of tie points from each individual local spatial intersection. Finally, all image orientations are moved to a common coordinate system by traversing the solved connected triplets using similarity transformations. Compared to the state-of-the-art global SfM methods, the performance and capability of the proposed hybrid approach are thoroughly demonstrated on various public datasets (mainly including ordered and unordered internet images, oblique aerial images, hard and complex datasets, etc.).

A HYBRID GLOBAL IMAGE ORIENTATION METHOD FOR SIMULTANEOUSLY ESTIMATING GLOBAL ROTATIONS AND GLOBAL TRANSLATIONS

Abstract. In recent years, the determination of global image orientation, i.e. global SfM, has gained a lot of attentions from researchers, mainly due to its time efficiency. Most of the global methods take relative rotations and translations as input for a two-step strategy comprised of global rotation averaging and global translation averaging. This paper by contrast presents a hybrid approach that aims to solve global rotations and translations simultaneously, but hierarchically. We first extract an optimal minimum cover connected image triplet set (OMCTS) which includes all available images with a minimum number of triplets, all of them with the three related relative orientations being compatible to each other. For non-collinear triplets in the OMCTS, we introduce some basic characterizations of the corresponding essential matrices and solve for the image pose parameters by averaging the constrained essential matrices. For the collinear triplets, on the other hand, the image pose parameters are estimated by relative orientation using the depth of object points from individual local spatial intersection. Finally, all image orientations are estimated in a common coordinate frame by traversing every solved triplet using a similarity transformation. We show results of our method on different benchmarks and demonstrate the performance and capability of the proposed approach by comparing with other global SfM methods.

Graph-based parallel large scale structure from motion

While Structure from Motion achieves great success in 3D reconstruction, it still meets challenges on large scale scenes. Incremental SfM approaches are robust to outliers, but are limited by low efficiency and easy suffer from drift problem. Though Global SfM methods are more efficient than incremental approaches, they are sensitive to outliers, and would also meet memory limitation and time bottleneck. In this work, large scale SfM is deemed as a graph problem, where graph are respectively constructed in image clustering step and local reconstructions merging step. By leveraging the graph structure, we are able to handle large scale dataset in divide-and-conquer manner. Firstly, images are modelled as graph nodes, with edges are retrieved from geometric information after feature matching. Then images are divided into independent clusters by a image clustering algorithm, and followed by a subgraph expansion step, the connection and completeness of scenes are enhanced by walking along a maximum spanning tree, which is utilized to construct overlapping images between clusters. Secondly, Image clusters are distributed into servers to execute SfM in parallel mode. Thirdly, after local reconstructions complete, we construct a minimum spanning tree to find accurate similarity transformations. Then the minimum spanning tree is transformed into a Minimum Height Tree to find a proper anchor node, and is further utilized to prevent error accumulation. We evaluate our approach on various kinds of datasets and our approach shows superiority over the state-of-the-art in accuracy and efficiency. Our algorithm is open-sourced in https://github.com/AIBluefisher/GraphSfM.

Improving progress monitoring by fusing point clouds, semantic data and computer vision

Automated construction-progress monitoring enables the required transparency for improved process control, and is thus being increasingly adopted by the construction industry. Many recent approaches use Scan-to/vs-BIM methods for capturing the as-built status of large construction sites. However, they often lack accuracy or are incomplete due to occluded elements and reconstruction inaccuracies. To overcome these limitations and exploit the rich project knowledge from the design phase, the authors propose taking advantage of the extensive geometric-semantic information provided by Building Information Models. In particular, valuable knowledge on the construction processes is inferred from BIM objects and their precedence relationships. SfM methods enable 3D building elements to be located and projected into the picture's 2D coordinate system. On this basis, the paper presents a machine-learning-based object-detection approach that supports progress monitoring by verifying element categories compared to the expected data from the digital model. The results show that, depending on the type of construction and the type of occlusions, the detection of built elements can rise by up to 50% compared to an SfM-based, purely geometric as-planned vs. as-built comparison.

A FULLY AUTOMATED INCREMENTAL PHOTOGRAMMETRIC PROCESSING DEDICATED FOR COLLABORATIVE REMOTE-COMPUTING WORKFLOW

Abstract. Image based-modeling practices in the field of Cultural Heritage studies are nowadays no longer seen as one-shot applications but as various and complex multimodal scenarios. Current use of SFM and photogrammetric methods implies their extensions to facilitate the management of complex multimodal data sets carried-out by different experts around a single heritage asset. In order to fully benefit of collaborative semantic enrichment of spatially oriented resources, a versatile and robust solution have been developed to enable incremental registration of image-sets within the web-based platform AIOLI. For this purpose, this paper will present an on-going development of a Totally Automated Co-registration and Orientations (TACO) work-flow.

Constrained planar motion analysis by decomposition

General SFM methods give poor results for images captured by constrained motions such as planar motion. In this paper, we propose new SFM algorithms for images captured under a common but constrained planar motion: the image plane is perpendicular to the motion plane. We show that a 2D image captured under such constrained planar motion can be decoupled into two 1D images: one 1D projective and one 1D affine. We then introduce the 1D affine camera model for completing 1D camera models. Next, we describe new subspace reconstruction methods, and apply these methods to the images captured by concentric mosaics, which undergo a special case of constrained planar motion. Finally, we demonstrate both in theory and experiments the advantage of the decomposition method over the general SFM methods by incorporating the constrained motion into the earliest stage of motion analysis.

Comparison of daptomycin MIC results by DIN, NCCLS, SFM, and SRGA methods for 297 Gram-positive organisms

Daptomycin is a novel lipopeptide antibiotic with potent in vitro antibacterial activity against Gram-positive pathogens. For daptomycin minimal inhibitory concentration (MIC) testing, National Committee for Clinical Laboratory Standards (NCCLS) recommends the use of broth containing physiological levels of calcium (50 μg/ml). The daptomycin susceptibility of 297 organisms was determined by NCCLS (Mueller–Hinton (MH) broth), Deutsches Institut für Normung (DIN; isotonic broth), Société Française de Microbiologie (SFM; three batches MH agar), and Swedish Reference Group for Antibiotics (SRGA; PDM agar). All media were supplemented to 50 μg/ml Ca 2+. There was good correlation between DIN and SFM methods (for staphylococci) with NCCLS results. Enterococci MICs using SFM methods were one to three dilutions lower and pneumococci results were one dilution higher than NCCLS. SRGA results were higher than NCCLS by one to four dilutions. Use of isotonic agar is an accepted alternative to isosensitest agar for the DIN method.

A perspective factorization method for Euclidean reconstruction with uncalibrated cameras

AbstractStructure from motion (SFM), which is recovering camera motion and scene structure from image sequences, has various applications, such as scene modelling, robot navigation, object recognition and virtual reality. Most of previous research on SFM requires the use of intrinsically calibrated cameras. In this paper we describe a factorization‐based method to recover Euclidean structure from multiple perspective views with uncalibrated cameras. The method first performs a projective reconstruction using a bilinear factorization algorithm, and then converts the projective solution to a Euclidean one by enforcing metric constraints. The process of updating a projective solution to a full metric one is referred as normalization in most factorization‐based SFM methods. We present three normalization algorithms which enforce Euclidean constraints on camera calibration parameters to recover the scene structure and the camera calibration simultaneously, assuming zero skew cameras. The first two algorithms are linear, one for dealing with the case that only the focal lengths are unknown, and another for the case that the focal lengths and the constant principal point are unknown. The third algorithm is bilinear, dealing with the case that the focal lengths, the principal points and the aspect ratios are all unknown. The results of experiments are presented. Copyright © 2002 John Wiley & Sons, Ltd.

Scanning Force Microscope Measurement of the Parameters of the Profiles of Submillimeter VLSI Components

A geometric model of a scanning force microscope (atomic force microscope) (SFM) cantilever with micro- and nanostructures is considered. A linear gauge is proposed for SFM calibration and for definition of the SFM probe parameters. SEM and SFM methods of measuring submicrometer dimensions of relief structures are compared.

Force measurements between layers of amphiphiles, polyelectrolytes and biomolecules

Force measurement techniques are providing increasing insight into the organization and properties of layers of organic molecules and biomolecules. Structural forces and forces of specific intermolecular interaction or recognition are being quantified for the first time using SFA and SFM methods. These data are essential for the rational design of biomolecular surfaces and bulk materials.