Structure From Motion Problem Research Articles

Parallel Structure from Motion for Sparse Point Cloud Generation in Large-Scale Scenes.

Scene reconstruction uses images or videos as input to reconstruct a 3D model of a real scene and has important applications in smart cities, surveying and mapping, military, and other fields. Structure from motion (SFM) is a key step in scene reconstruction, which recovers sparse point clouds from image sequences. However, large-scale scenes cannot be reconstructed using a single compute node. Image matching and geometric filtering take up a lot of time in the traditional SFM problem. In this paper, we propose a novel divide-and-conquer framework to solve the distributed SFM problem. First, we use the global navigation satellite system (GNSS) information from images to calculate the GNSS neighborhood. The number of images matched is greatly reduced by matching each image to only valid GNSS neighbors. This way, a robust matching relationship can be obtained. Second, the calculated matching relationship is used as the initial camera graph, which is divided into multiple subgraphs by the clustering algorithm. The local SFM is executed on several computing nodes to register the local cameras. Finally, all of the local camera poses are integrated and optimized to complete the global camera registration. Experiments show that our system can accurately and efficiently solve the structure from motion problem in large-scale scenes.

Direct Optical-Flow-Aware Computational Framework for 3D Reconstruction

In this paper, a direct computational method is presented which combines optical flow and structure from motion (SfM) by putting the SfM problem in the framework of optical flow estimation. In other word, the optical flow is reparametrized in term of the camera’s motion and scene’s depth, resulting in a similar variation optimization as in optical flow estimation. Meanwhile, three techniques are proposed to improve the accuracy and robustness of the direct approach, including the fast guided interpolation (FGI), the left-right consistency constraint and the soft segment constraint. Experimental results on the Middlebury dataset and KITTI2012 dataset show that the proposed approach can achieve highly-accurate 3D reconstruction with the dense and smooth surface which results in a state-of-the-art performance in optical flow.

A survey of structure from motion.

The structure from motion (SfM) problem in computer vision is to recover the three-dimensional (3D) structure of a stationary scene from a set of projective measurements, represented as a collection of two-dimensional (2D) images, via estimation of motion of the cameras corresponding to these images. In essence, SfM involves the three main stages of (i) extracting features in images (e.g. points of interest, lines,etc.) and matching these features between images, (ii) camera motion estimation (e.g. using relative pairwise camera positions estimated from the extracted features), and (iii) recovery of the 3D structure using the estimated motion and features (e.g. by minimizing the so-calledreprojection error). This survey mainly focuses on relatively recent developments in the literature pertaining to stages (ii) and (iii). More specifically, after touching upon the early factorization-based techniques for motion and structure estimation, we provide a detailed account of some of the recent cameralocationestimation methods in the literature, followed by discussion of notable techniques for 3D structure recovery. We also cover the basics of thesimultaneous localization and mapping(SLAM) problem, which can be viewed as a specific case of the SfM problem. Further, our survey includes a review of the fundamentals of feature extraction and matching (i.e. stage (i) above), various recent methods for handling ambiguities in 3D scenes, SfM techniques involving relatively uncommon camera models and image features, and popular sources of data and SfM software.

Projective Multiview Structure and Motion from Element-Wise Factorization

The Sturm-Triggs type iteration is a classic approach for solving the projective structure-from-motion (SfM) factorization problem, which iteratively solves the projective depths, scene structure, and camera motions in an alternated fashion. Like many other iterative algorithms, the Sturm-Triggs iteration suffers from common drawbacks, such as requiring a good initialization, the iteration may not converge or may only converge to a local minimum, and so on. In this paper, we formulate the projective SfM problem as a novel and original element-wise factorization (i.e., Hadamard factorization) problem, as opposed to the conventional matrix factorization. Thanks to this formulation, we are able to solve the projective depths, structure, and camera motions simultaneously by convex optimization. To address the scalability issue, we adopt a continuation-based algorithm. Our method is a global method, in the sense that it is guaranteed to obtain a globally optimal solution up to relaxation gap. Another advantage is that our method can handle challenging real-world situations such as missing data and outliers quite easily, and all in a natural and unified manner. Extensive experiments on both synthetic and real images show comparable results compared with the state-of-the-art methods.

Pose-Free Structure From Motion Using Depth From Motion Constraints

Structure from motion (SFM) is the problem of recovering the geometry of a scene from a stream of images taken from unknown viewpoints. One popular approach to estimate the geometry of a scene is to track scene features on several images and reconstruct their position in 3-D. During this process, the unknown camera pose must also be recovered. Unfortunately, recovering the pose can be an ill-conditioned problem which, in turn, can make the SFM problem difficult to solve accurately. We propose an alternative formulation of the SFM problem with fixed internal camera parameters known a priori. In this formulation, obtained by algebraic variable elimination, the external camera pose parameters do not appear. As a result, the problem is better conditioned in addition to involving much fewer variables. Variable elimination is done in three steps. First, we take the standard SFM equations in projective coordinates and eliminate the camera orientations from the equations. We then further eliminate the camera center positions. Finally, we also eliminate all 3-D point positions coordinates, except for their depths with respect to the camera center, thus obtaining a set of simple polynomial equations of degree two and three. We show that, when there are merely a few points and pictures, these "depth-only equations" can be solved in a global fashion using homotopy methods. We also show that, in general, these same equations can be used to formulate a pose-free cost function to refine SFM solutions in a way that is more accurate than by minimizing the total reprojection error, as done when using the bundle adjustment method. The generalization of our approach to the case of varying internal camera parameters is briefly discussed.

Computing Smooth Time Trajectories for Camera and Deformable Shape in Structure from Motion with Occlusion

We address the classical computer vision problems of rigid and nonrigid structure from motion (SFM) with occlusion. We assume that the columns of the input observation matrix W describe smooth 2D point trajectories over time. We then derive a family of efficient methods that estimate the column space of W using compact parameterizations in the Discrete Cosine Transform (DCT) domain. Our methods tolerate high percentages of missing data and incorporate new models for the smooth time trajectories of 2D-points, affine and weak-perspective cameras, and 3D deformable shape. We solve a rigid SFM problem by estimating the smooth time trajectory of a single camera moving around the structure of interest. By considering a weak-perspective camera model from the outset, we directly compute euclidean 3D shape reconstructions without requiring postprocessing steps such as euclidean upgrade and bundle adjustment. Our results on real SFM data sets with high percentages of missing data compared positively to those in the literature. In nonrigid SFM, we propose a novel 3D shape trajectory approach that solves for the deformable structure as the smooth time trajectory of a single point in a linear shape space. A key result shows that, compared to state-of-the-art algorithms, our nonrigid SFM method can better model complex articulated deformation with higher frequency DCT components while still maintaining the low-rank factorization constraint. Finally, we also offer an approach for nonrigid SFM when W is presented with missing data.

An iterative multiresolution scheme for SFM with missing data: Single and multiple object scenes

Most of the techniques proposed for tackling the Structure from Motion problem (SFM) cannot deal with high percentages of missing data in the matrix of trajectories. Furthermore, an additional problem should be faced up when working with multiple object scenes: the rank of the matrix of trajectories should be estimated. This paper presents an iterative multiresolution scheme for SFM with missing data to be used in both the single and multiple object cases. The proposed scheme aims at recovering missing entries in the original input matrix. The objective is to improve the results by applying a factorization technique to the partially or totally filled in matrix instead of to the original input one. Experimental results obtained with synthetic and real data sequences, containing single and multiple objects, are presented to show the viability of the proposed approach.

Cyclic optimization algorithms for simultaneous structure and motion recovery in computer vision

With few exceptions, most previous approaches to the structure from motion (SFM) problem in computer vision have been based on a decoupling between motion and depth recovery, usually via the epipolar constraint. This article offers closed-form cyclic optimization algorithms for the simultaneous recovery of motion and depth in the discrete SFM problem. Cyclic coordinate descent (CCD) algorithms in which each stage admits closed-form solutions are developed for two widely used fitting criteria: the geometric error in one image, and the reprojection error criterion. As a by-product, analytic gradients that can be used in descent-based optimization methods are also obtained. The computational efficiency, statistical consistency, noise robustness, and accuracy of the algorithms are assessed via experiments with synthetic image data.

A unified probabilistic model of the perception of three-dimensional structure from optic flow

Human observers can perceive the three- dimensional (3-D) structure of their environment using various cues, an important one of which is optic flow. The motion of any point's projection on the retina depends both on the point's movement in space and on its distance from the eye. Therefore, retinal motion can be used to extract the 3-D structure of the environment and the shape of objects, in a process known as structure-from-motion (SFM). However, because many combinations of 3-D structure and motion can lead to the same optic flow, SFM is an ill-posed inverse problem. The rigidity hypothesis is a constraint supposed to formally solve the SFM problem and to account for human performance. Recently, however, a number of psychophysical results, with both moving and stationary human observers, have shown that the rigidity hypothesis alone cannot account for human performance in SFM tasks, but no model is known to account for the new results. Here, we construct a Bayesian model of SFM based mainly on one new hypothesis, that of stationarity, coupled with the rigidity hypothesis. The predictions of the model, calculated using a new and powerful methodology called Bayesian programming, account for a wide variety of experimental findings.

Efficient nearly-optimal motion and structure from images via analysis of dimensional reduction: application to safety checking system

In computing 3D motion and structure from image correspondences, often called the structure from motion (SFM) problem, dimensions of the used variable set are very large because it contains both motion and structure parameters. As a result, to solve the problem incurs much computational burden. However, in on-line applications of the SFM problem, computational efficiency needs to be stressed as some accuracy of the solutions is sacrificed. In this respect, various dimensional reduction methods are often introduced to improve computational efficiency and this usually leads to various reduced-form SFM problems. The so-obtained reduced-form SFM problems depend on fewer unknowns than the original SFM problem, thus allowing, in principle, for a less computationally intensive estimation, albeit potentially sacrificing accuracy of the results. It is thus interesting to study how much accuracy is lost. This is done by analytically proving results on equivalence or proximity of solutions for some example cases of the so-obtained reduced-form SFM problems. And then, based on the analysis, the author also proposes how to reduce the loss of accuracy in the reduced-form SFM problems (in the meaning of adjusting those reduced-form SFM problems to better approximate the original optimal SFM problem; that is, an optimal SFM problem that does not use any dimensional reduction methods). Experimental results are given to show the effect in practice. Finally, as an example application, a safety checking system using vision is considered.

Recovering the missing components in a large noisy low-rank matrix: application to SFM

In computer vision, it is common to require operations on matrices with "missing data," for example, because of occlusion or tracking failures in the Structure from Motion (SFM) problem. Such a problem can be tackled, allowing the recovery of the missing values, if the matrix should be of low rank (when noise free). The filling in of missing values is known as imputation. Imputation can also be applied in the various subspace techniques for face and shape classification, online "recommender" systems, and a wide variety of other applications. However, iterative imputation can lead to the "recovery" of data that is seriously in error. In this paper, we provide a method to recover the most reliable imputation, in terms of deciding when the inclusion of extra rows or columns, containing significant numbers of missing entries, is likely to lead to poor recovery of the missing parts. Although the proposed approach can be equally applied to a wide range of imputation methods, this paper addresses only the SFM problem. The performance of the proposed method is compared with Jacobs' and Shum's methods for SFM.

Structure from Motion Using Sequential Monte Carlo Methods

In this paper, the structure from motion (SfM) problem is addressed using sequential Monte Carlo methods. A new SfM algorithm based on random sampling is derived to estimate the posterior distributions of camera motion and scene structure for the perspective projection camera model. Experimental results show that challenging issues in solving the SfM problem, due to erroneous feature tracking, feature occlusion, motion/structure ambiguity, mixed-domain sequences, mismatched features, and independently moving objects, can be well modeled and effectively addressed using the proposed method.

A PRACTICAL APPROACH FOR 3D BUILDING MODELING FROM UNCALIBRATED VIDEO SEQUENCES

This paper presents an approach for reconstructing a realistic 3D model of a building from its uncalibrated video sequences taken by a hand-held camera. The novelty of this approach lies in the integration of some prior scene knowledge in the different stages of the Structure From Motion problem (SFM). First, the coplanarity of buildings is considered in the calculation of the fundamental matrices to deal with the critical configurations. Second, the line parallelism and plane orthogonality are transformed to the constraints on the absolute quadric during camera auto-calibration. This makes some critical cases solvable and the reconstruction more Euclidean. The approach is implemented and validated using simulated data and real image data. The experimental results at the end of the paper show the effectiveness of our approach.

Robust structure from motion estimation using inertial data

The utility of using inertial data for the structure-from-motion (SfM) problem is addressed. We show how inertial data can be used for improved noise resistance, reduction of inherent ambiguities, and handling of mixed-domain sequences. We also show that the number of feature points needed for accurate and robust SfM estimation can be significantly reduced when inertial data are employed. Cramér-Rao lower bounds are computed to quantify the improvements in estimating motion parameters. A robust extended-Kalman-filter-based SfM algorithm using inertial data is then developed to fully exploit the inertial information. This algorithm has been tested by using synthetic and real image sequences, and the results show the efficacy of using inertial data for the SfM problem.

3D structure from 2D motion

This article motivates the structure from motion (SfM) approaches by describing some current practical applications. This is followed by a brief discussion of the background of the field. Then, several techniques are outlined that show various important approaches and paradigms to the SfM problem. Critical issues, advantages and disadvantages are pointed out. Subsequently, we present our SfM approach for recursive estimation of motion, structure, and camera geometry in a nonlinear dynamic system framework. Results are given for synthetic and real images. These are used to assess the accuracy and stability of the technique. We then discuss some practical and real-time applications we have encountered and the reliability and flexibility of the approach in those settings. Finally, we conclude with results from an independent evaluation study conducted by industry where the proposed SfM algorithm compared favorably to alternative approaches.

Rigid motion and structure from several sets of parallel lines in a monocular image sequence

An image line is a very reliable feature in the feature-based structure-from-motion (SFM) problem. It has at least three advantages: (1) it is not easily affected by occlusion. (2) It can achieve subpixel accuracy. (3) The correspondence problem is easier and less ambiguous. Although some generalized SFM algorithms using image line features have been recently developed, they usually suffer from high error sensitivity and large matrix size. Besides, in order to make their estimations more robust, many line correspondences (much more than the least requirement in theory) are usually needed. These shortages are not beneficial for practical use. Because many sets of parallel lines are frequently encountered in an artificial 3D environment and can be easily identified, we try to utilize these identified parallel lines to simplify the generalized SFM problem and improve the performance of parameter estimation. A fast and reliable algorithm is proposed in our paper. Both computer-generated and real-world images are tested and several satisfactory results have been obtained. In addition, its error sensitivity and data degeneracy will be discussed.

Surface Interpolation in Three-Dimensional Structure-from-Motion Perception

Although it is appreciated that humans can use a number of visual cues to perceive the three-dimensional (3-D) shape of an object, for example, luminance, orientation, binocular disparity, and motion, the exact mechanisms employed are not known (De Yoe and Van Essen 1988). An important approach to understanding the computations performed by the visual system is to develop algorithms (Marr 1982) or neural network models (Lehky and Sejnowski 1988; Siegel 1987) that are capable of computing shape from specific cues in the visual image. In this study we investigated the ability of observers to see the 3-D shape of an object using motion cues, so called structure-from-motion (SFM). We measured human performance in a two-alternative forced choice task using novel dynamic random-dot stimuli with limited point lifetimes. We show that the human visual system integrates motion information spatially and temporally (across several point lifetimes) as part of the process for computing SFM. We conclude that SFM algorithms must include surface interpolation to account for human performance. Our experiments also provide evidence that local velocity information, and not position information derived from discrete views of the image (as proposed by some algorithms), is used to solve the SFM problem by the human visual system.