Planar Scene Research Articles

Effective energy-based multi-view piecewise planar stereo

For piecewise planar scene modeling, many challenging issues still persist, in particular, how to generate sufficient candidate planes and how to assign an optimal plane for each spatial patch. To address these issues, we present a novel multi-view piecewise planar stereo method for the complete reconstruction. In our method, reconstruction is formulated as an energy-based plane labeling problem, where photo-consistency and geometric constraints are incorporated to a unified superpixel-level MRF (Markov Random Field) framework. To enhance the efficacy of the plane inference and optimization, an effective multi-direction plane sweeping with much restricted search space is carried out to generate sufficient and reliable candidate planes. Experiments show that our method can effectively handle many challenging factors (e.g., slant surfaces, textureless regions) and achieve satisfactory results.

EVALUATION OF METHODS FOR COREGISTRATION AND FUSION OF RPAS-BASED 3D POINT CLOUDS AND THERMAL INFRARED IMAGES

This paper discusses the automatic coregistration and fusion of 3d point clouds generated from aerial image sequences and corresponding thermal infrared (TIR) images. Both RGB and TIR images have been taken from a RPAS platform with a predefined flight path where every RGB image has a corresponding TIR image taken from the same position and with the same orientation with respect to the accuracy of the RPAS system and the inertial measurement unit. To remove remaining differences in the exterior orientation, different strategies for coregistering RGB and TIR images are discussed: (i) coregistration based on 2D line segments for every single TIR image and the corresponding RGB image. This method implies a mainly planar scene to avoid mismatches; (ii) coregistration of both the dense 3D point clouds from RGB images and from TIR images by coregistering 2D image projections of both point clouds; (iii) coregistration based on 2D line segments in every single TIR image and 3D line segments extracted from intersections of planes fitted in the segmented dense 3D point cloud; (iv) coregistration of both the dense 3D point clouds from RGB images and from TIR images using both ICP and an adapted version based on corresponding segmented planes; (v) coregistration of both image sets based on point features. The quality is measured by comparing the differences of the back projection of homologous points in both corrected RGB and TIR images.

EVALUATION OF METHODS FOR COREGISTRATION AND FUSION OF RPAS-BASED 3D POINT CLOUDS AND THERMAL INFRARED IMAGES

Abstract. This paper discusses the automatic coregistration and fusion of 3d point clouds generated from aerial image sequences and corresponding thermal infrared (TIR) images. Both RGB and TIR images have been taken from a RPAS platform with a predefined flight path where every RGB image has a corresponding TIR image taken from the same position and with the same orientation with respect to the accuracy of the RPAS system and the inertial measurement unit. To remove remaining differences in the exterior orientation, different strategies for coregistering RGB and TIR images are discussed: (i) coregistration based on 2D line segments for every single TIR image and the corresponding RGB image. This method implies a mainly planar scene to avoid mismatches; (ii) coregistration of both the dense 3D point clouds from RGB images and from TIR images by coregistering 2D image projections of both point clouds; (iii) coregistration based on 2D line segments in every single TIR image and 3D line segments extracted from intersections of planes fitted in the segmented dense 3D point cloud; (iv) coregistration of both the dense 3D point clouds from RGB images and from TIR images using both ICP and an adapted version based on corresponding segmented planes; (v) coregistration of both image sets based on point features. The quality is measured by comparing the differences of the back projection of homologous points in both corrected RGB and TIR images.

IMAGE STITCHING WITH PERSPECTIVE-PRESERVING WARPING

Image stitching algorithms often adopt the global transform, such as homography, and work well for planar scenes or parallax free camera motions. However, these conditions are easily violated in practice. With casual camera motions, variable taken views, large depth change, or complex structures, it is a challenging task for stitching these images. The global transform model often provides dreadful stitching results, such as misalignments or projective distortions, especially perspective distortion. To this end, we suggest a perspective-preserving warping for image stitching, which spatially combines local projective transforms and similarity transform. By weighted combination scheme, our approach gradually extrapolates the local projective transforms of the overlapping regions into the non-overlapping regions, and thus the final warping can smoothly change from projective to similarity. The proposed method can provide satisfactory alignment accuracy as well as reduce the projective distortions and maintain the multi-perspective view. Experimental analysis on a variety of challenging images confirms the efficiency of the approach.

IMAGE STITCHING WITH PERSPECTIVE-PRESERVING WARPING

Abstract. Image stitching algorithms often adopt the global transform, such as homography, and work well for planar scenes or parallax free camera motions. However, these conditions are easily violated in practice. With casual camera motions, variable taken views, large depth change, or complex structures, it is a challenging task for stitching these images. The global transform model often provides dreadful stitching results, such as misalignments or projective distortions, especially perspective distortion. To this end, we suggest a perspective-preserving warping for image stitching, which spatially combines local projective transforms and similarity transform. By weighted combination scheme, our approach gradually extrapolates the local projective transforms of the overlapping regions into the non-overlapping regions, and thus the final warping can smoothly change from projective to similarity. The proposed method can provide satisfactory alignment accuracy as well as reduce the projective distortions and maintain the multi-perspective view. Experimental analysis on a variety of challenging images confirms the efficiency of the approach.

Registration of multitemporal aerial optical images using line features

Registration of multitemporal images is generally considered difficult because scene changes can occur between the times the images are obtained. Since the changes are mostly radiometric in nature, features are needed that are insensitive to radiometric differences between the images. Lines are geometric features that represent straight edges of rigid man-made structures. Because such structures rarely change over time, lines represent stable geometric features that can be used to register multitemporal remote sensing images. An algorithm to establish correspondence between lines in two images of a planar scene is introduced and formulas to relate the parameters of a homography transformation to the parameters of corresponding lines in images are derived. Results of the proposed image registration on various multitemporal images are presented and discussed.

Change detection in underwater imagery.

In this work, we deal with the problem of change detection in an underwater scenario given an unblurred-blurred image pair of a planar scene taken at different times. The blur is primarily due to the dynamic nature of the water surface and its nature is space-invariant in the presence of cyclic water flows. Exploiting the sparsity of the induced blur as well as the occlusions, we propose a distort-difference pipeline that employs an alternating minimization framework to perform change detection in the presence of geometric distortions (skew) as well as photometric degradations (blur and global illumination variations). The method can effectively yield both sharp and blurred occluder maps. Using synthetic as well as real data, we demonstrate how the proposed technique advances the state of the art.

Large Scale Image Mosaic Construction for Agricultural Applications

We present a novel technique for stitching images including those obtained from aerial vehicles flying at low altitudes. Existing image stitching/mosaicking methods rely on inter-image homography computation based on a planar scene assumption. This assumption holds when images are taken from high-altitudes (hence the depth variation is negligible). It is often violated when flying at low altitudes. Furthermore, to avoid scale and resolution changes, existing methods rely on primarily translational motion at fixed altitudes. Our method removes these limitations and performs well even when aerial images are taken from low altitudes by an aerial vehicle performing complex motions. It starts by extracting the ground geometry from a sparse reconstruction of the scene obtained from a small fraction of the input images. Next, it selects the best image (from the entire sequence) for each location on the ground using a novel camera selection criterion. This image is then independently rectified to obtain the corresponding portion of the mosaic. Therefore, the technique avoids performing costly joint-optimization over the entire sequence. It is validated using challenging input sequences motivated by agricultural applications.

A Self-Calibration Method of Zooming Camera

In this article we proposed a novel approach to self- calibrate a camera with variable focal length. We show that the estimation of camera’s intrinsic parameters is possible from only two points of an unknown planar scene. The projection of these points by using the projection matrices in two images only permit us to obtain a system of equations according to the camera’s intrinsic parameters . From this system we formulated a nonlinear cost function which its minimization allows us to estimate the camera’s intrinsic parameters in each view. The results on synthetic and real data justify the robustness of our method in term of reliability and convergence.

Time-of-Flight Sensor Calibration for a Color and Depth Camera Pair.

We present a calibration method of a time-of-flight (ToF) sensor and a color camera pair to align the 3D measurements with the color image correctly. We have designed a 2.5D pattern board with irregularly placed holes to be accurately detected from low resolution depth images of a ToF camera as well as from high resolution color images. In order to improve the accuracy of the 3D measurements of a ToF camera, we propose to perform ray correction and range bias correction. We reset the transformation of the ToF sensor which transforms the radial distance into the scene depth in Cartesian coordinate through ray correction. Then we capture a planar scene from different depths to correct the distance error that is shown to be dependent not only on the distance but also on the pixel location. The range error profiles along the calibrated distance are classified according to their wiggling shapes and each cluster of profiles with similar shape are separately estimated using a B-spline function. The standard deviation of the remaining random noise is recorded as an uncertainty information of distance measurements. We show the performance of our calibration method quantitatively and qualitatively on various datasets, and validate the impact of our method by demonstrating an RGB-D shape refinement application.

Nonlinear ego-motion estimation from optical flow for online control of a quadrotor UAV

For the control of unmanned aerial vehicles (UAVs) in GPS-denied environments, cameras have been widely exploited as the main sensory modality for addressing the UAV state estimation problem. However, the use of visual information for ego-motion estimation presents several theoretical and practical difficulties, such as data association, occlusions, and lack of direct metric information when exploiting monocular cameras. In this paper, we address these issues by considering a quadrotor UAV equipped with an onboard monocular camera and an inertial measurement unit (IMU). First, we propose a robust ego-motion estimation algorithm for recovering the UAV scaled linear velocity and angular velocity from optical flow by exploiting the so-called continuous homography constraint in the presence of planar scenes. Then, we address the problem of retrieving the (unknown) metric scale by fusing the visual information with measurements from the onboard IMU. To this end, two different estimation strategies are proposed and critically compared: a first exploiting the classical extended Kalman filter (EKF) formulation, and a second one based on a novel nonlinear estimation framework. The main advantage of the latter scheme lies in the possibility of imposing a desired transient response to the estimation error when the camera moves with a constant acceleration norm with respect to the observed plane. We indeed show that, when compared against the EKF on the same trajectory and sensory data, the nonlinear scheme yields considerably superior performance in terms of convergence rate and predictability of the estimation. The paper is then concluded by an extensive experimental validation, including an onboard closed-loop control of a real quadrotor UAV meant to demonstrate the robustness of our approach in real-world conditions.

Deskewing of underwater images.

We address the problem of restoring a static planar scene degraded by skewing effect when imaged through adynamic water surface. In particular, we investigate geometric distortions due to unidirectional cyclic waves and circular ripples,phenomena that are most prevalent in fluid flow. Although the camera and scene are stationary, light rays emanating from a scene undergo refraction at the fluid–air interface. This refraction effect is time varying for dynamic fluids and results in nonrigid distortions (skew) in the captured image. These distortions can be associated with motion blur depending on the exposure time of the camera. In the first part of this paper, we establish the condition under which the blur induced due to unidirectional cyclic waves can be treated as space invariant. We proceed to derive a mathematical model for blur formation and propose a restoration scheme using a single degraded observation. In the second part, we reveal how the blur induced by circular ripples(though space variant) can be modeled as uniform in the polar domain and develop a method for deskewing. The proposed methods are tested on synthetic as well as real examples.

Image Alignment by Piecewise Planar Region Matching

Robust image registration is a challenging problem, especially when dealing with severe changes in illumination and viewpoint. Previous methods assume a global geometric model (e.g., homography) and, hence, are only able to align images under predefined constraints (e.g., planar scenes and parallax-free camera motion). However, these constraints may not hold for natural scenes and uncontrolled imaging conditions. Therefore, this paper proposes a novel method which approximates image regions with planes by incorporating piecewise local geometric models. The approximated planar regions are obtained by exploiting a hierarchical figure-ground segmentation method. Each such planar region assumes an affine transformation. To achieve the alignment of the planar regions, an energy function is defined which employs intensity, a key-point descriptor, and geometric information under a global constraint. By re-segmenting and re-merging planar regions iteratively in an energy minimization framework, the method is able to align images even under significant changes in illumination and viewpoint. Experiments on two datasets show that the proposed method outperforms state-of-the-art, especially in the case of large appearance variations and it is, therefore, applicable to web-images (i.e., unconstrained setting) which are taken from the same scene with different viewpoints.

Robust method for camera self-calibration by an unkown planar scene

In this paper we present a self-calibration method for a CCD camera with varying intrinsic parameters based on an unknown planar scene. The advantage of our method is reducing the number of images (two images) needed to estimate the parameters of the camera used. Moreover, self-calibration equations are related to the number of points matched (very numerous and easy to detect) rather than to the number of images, since the use of a large number of images requires high computation time. On the other hand, we base on the points matched, which are numerous, when estimating the projection matrices and homographies between the images. The latter are used with the images of the absolute conic to formulate a system of non-linear equations (self-calibration equations depend on the number of matched pairs). Finally, the intrinsic parameters of the camera can be obtained by minimizing a non-linear cost function in a two-step procedure: initialization and optimization. Experiment results show the robustness of our algorithms in terms of stability and convergence.

Bayesian perspective-plane (BPP) with maximum likelihood searching for visual localization

The proposed "Perspective-Plane" in this paper is similar to the well-known "Perspective-n-Point (PnP)" or "Perspective-n-Line (PnL)" problems in computer vision. However, it has broader applications and potentials, because planar scenes are more widely available than control points or lines in daily life. We address this problem in the Bayesian framework and propose the "Bayesian Perspective-Plane (BPP)" algorithm, which can deal with more generalized constraints rather than type-specific ones. The BPP algorithm consists of three steps: 1) plane normal computation by maximum likelihood searching from Bayesian formulation; 2) plane distance computation; and 3) visual localization. In the first step, computation of the plane normal is formulated within the Bayesian framework, and is solved by using the proposed Maximum Likelihood Searching Model (MLS-M). Two searching modes of 2D and 1D are discussed. MLS-M can incorporate generalized planar and out-of-plane deterministic constraints. With the computed normal, the plane distance is recovered from a reference length or distance. The positions of the object or the camera can be determined afterwards. Extensions of the proposed BPP algorithm to deal with un-calibrated images and for camera calibration are discussed. The BPP algorithm has been tested with both simulation and real image data. In the experiments, the algorithm was applied to recover planar structure and localize objects by using different types of constraints. The 2D and 1D searching modes were illustrated for plane normal computation. The results demonstrate that the algorithm is accurate and generalized for object localization. Extensions of the proposed model for camera calibration were also illustrated in the experiment. The potential of the proposed algorithm was further demonstrated to solve the classic Perspective-Three-Point (P3P) problem and classify the solutions in the experiment. The proposed BPP algorithm suggests a practical and effective approach for visual localization.

Geometric 3D point cloud compression

The use of 3D data in mobile robotics applications provides valuable information about the robot’s environment but usually the huge amount of 3D information is unmanageable by the robot storage and computing capabilities. A data compression is necessary to store and manage this information but preserving as much information as possible. In this paper, we propose a 3D lossy compression system based on plane extraction which represent the points of each scene plane as a Delaunay triangulation and a set of points/area information. The compression system can be customized to achieve different data compression or accuracy ratios. It also supports a color segmentation stage to preserve original scene color information and provides a realistic scene reconstruction. The design of the method provides a fast scene reconstruction useful for further visualization or processing tasks.

Homography-based Visual Servoing for Autonomous Underwater Vehicles

A nonlinear visual servoing approach is proposed for the stabilisation of fully-actuated autonomous underwater vehicles (AUVs) by exploiting the homography matrix between the two images of a planar scene. In a cascade manner, an outer-loop control defines a reference setpoint based on the homography matrix, and an inner-loop control ensures the stabilisation of the setpoint by assigning thrust and torque controls. In contrast with conventional kinematic solution, the proposed controller deals with the high nonlinearity and coupling of the system dynamics and ensures almost-global asymptotical stability. In addition, the interactions of the AUV with the surrounding fluid (e.g., added mass and drag effects) are often difficult to model precisely whereas they may significantly perturb its motion. The proposed controller –augmented with an effective integral action– allows for the compensation of model uncertainties and for robust performance against such perturbations. Simulation results illustrating these properties on a realistic AUV model subject to sea current are reported.

A highly accurate dense approach for homography estimation using modified differential evolution

The homography corresponds to a 3×3 matrix which transfers image points between two images of a planar scene or two images captured by cameras under purely rotational motion. Homography estimation is crucial to many computer vision tasks involving multi-view geometry. Unlike most existing algorithms using sparse information extracted from images, this paper proposes a dense homography estimation method taking into account both the color and gradient information of the whole image. In particular, homography estimation is recast as a two-objective optimization problem, which is solved within a global optimization procedure guided by sparse control points (SCPs) based on modified differential evolution (DE). In order to improve the computational efficiency, two strategies namely pre-evaluation and coarse-to-fine (C2F) search are integrated into the proposed framework. The experimental results on synthetically rendered images and real images demonstrate that the new method can consistently improve the accuracy of homography estimation compared with two most successful feature based algorithms, and the incorporated acceleration strategies are able to speed up the minimization procedure by about 15 times. The applicability of our proposed method is further demonstrated in two real-world applications, namely occlusion removal and image mosaicing.

Non-uniform image deblurring using an optical computing system

Removing non-uniform blur caused by camera shaking is troublesome because of its high computational cost. We analyze the efficiency bottlenecks of a non-uniform deblurring algorithm and propose an efficient optical computation deblurring framework that implements the time-consuming and repeatedly required modules, i.e., non-uniform convolution and perspective warping, by light transportation. Specifically, the non-uniform convolution and perspective warping are optically computed by a hybrid system that is composed of an off-the-shelf projector and a camera mounted on a programmable motion platform. Benefitting from the high speed and parallelism of optical computation, our system has the potential to accelerate existing non-uniform motion deblurring algorithms significantly. To validate the effectiveness of the proposed approach, we also develop a prototype system that is incorporated into an iterative deblurring framework to effectively address the image blur of planar scenes that is caused by 3D camera rotation around the x-, y- and z-axes. The results show that the proposed approach has a high efficiency while obtaining a promising accuracy and has a high generalizability to more complex camera motions.

Evaluation of two-view geometry methods with automatic ground-truth generation

A large number of methods have been published that aim to evaluate various components of multi-view geometry systems. Most of these have focused on the feature extraction, description and matching stages (the visual front end), since geometry computation can be evaluated through simulation. Many data sets are constrained to small scale scenes or planar scenes that are not challenging to new algorithms, or require special equipment. This paper presents a method for automatically generating geometry ground truth and challenging test cases from high spatio-temporal resolution video. The objective of the system is to enable data collection at any physical scale, in any location and in various parts of the electromagnetic spectrum.The data generation process consists of collecting high resolution video, computing accurate sparse 3D reconstruction, video frame culling and down sampling, and test case selection. The evaluation process consists of applying a test 2-view geometry method to every test case and comparing the results to the ground truth. This system facilitates the evaluation of the whole geometry computation process or any part thereof against data compatible with a realistic application. A collection of example data sets and evaluations is included to demonstrate the range of applications of the proposed system.