Non-planar Scenes Research Articles

Motion-guided small MAV detection in complex and non-planar scenes

In recent years, there has been a growing interest in the visual detection of micro aerial vehicles (MAVs) due to its importance in numerous applications. However, the existing methods based on either appearance or motion features encounter difficulties when the background is complex or the MAV is too small. In this paper, we propose a novel motion-guided MAV detector that can accurately identify small MAVs in complex and non-planar scenes. This detector first exploits a motion feature enhancement module to capture the motion features of small MAVs. Then it uses multi-object tracking and trajectory filtering to eliminate false positives caused by motion parallax. Finally, an appearance-based classifier and an appearance-based detector that operates on the cropped regions are used to achieve precise detection results. Our proposed method can effectively and efficiently detect extremely small MAVs from dynamic and complex backgrounds because it aggregates pixel-level motion features and eliminates false positives based on the motion and appearance features of MAVs. Experiments on the ARD-MAV dataset demonstrate that the proposed method could achieve high performance in small MAV detection under challenging conditions and outperform other state-of-the-art methods across various metrics.

A Multilayer Perceptron-Based Spherical Visual Compass Using Global Features.

This paper presents a visual compass method utilizing global features, specifically spherical moments. One of the primary challenges faced by photometric methods employing global features is the variation in the image caused by the appearance and disappearance of regions within the camera's field of view as it moves. Additionally, modeling the impact of translational motion on the values of global features poses a significant challenge, as it is dependent on scene depths, particularly for non-planar scenes. To address these issues, this paper combines the utilization of image masks to mitigate abrupt changes in global feature values and the application of neural networks to tackle the modeling challenge posed by translational motion. By employing masks at various locations within the image, multiple estimations of rotation corresponding to the motion of each selected region can be obtained. Our contribution lies in offering a rapid method for implementing numerous masks on the image with real-time inference speed, rendering it suitable for embedded robot applications. Extensive experiments have been conducted on both real-world and synthetic datasets generated using Blender. The results obtained validate the accuracy, robustness, and real-time performance of the proposed method compared to a state-of-the-art method.

Image Matching by Bare Homography.

This paper presents Slime, a novel non-deep image matching framework which models the scene as rough local overlapping planes. This intermediate representation sits in-between the local affine approximation of the keypoint patches and the global matching based on both spatial and similarity constraints, providing a progressive pruning of the correspondences, as planes are easier to handle with respect to general scenes. Slime decomposes the images into overlapping regions at different scales and computes loose planar homographies. Planes are mutually extended by compatible matches and the images are split into fixed tiles, with only the best homographies retained for each pair of tiles. Stable matches are identified according to the consensus of the admissible stereo configurations provided by pairwise homographies. Within tiles, the rough planes are then merged according to their overlap in terms of matches and further consistent correspondences are extracted. The whole process only involves homography constraints. As a result, both the coverage and the stability of correct matches over the scene are amplified, together with the ability to spot matches in challenging scenes, allowing traditional hybrid matching pipelines to make up lost ground against recent end-to-end deep matching methods. In addition, the paper gives a thorough comparative analysis of recent state-of-the-art in image matching represented by end-to-end deep networks and hybrid pipelines. The evaluation considers both planar and non-planar scenes, taking into account critical and challenging scenarios including abrupt temporal image changes and strong variations in relative image rotations. According to this analysis, although the impressive progress done in this field, there is still a wide room for improvements to be investigated in future research.

SIFT Matching by Context Exposed.

This paper investigates how to step up local image descriptor matching by exploiting matching context information. Two main contexts are identified, originated respectively from the descriptor space and from the keypoint space. The former is generally used to design the actual matching strategy while the latter to filter matches according to the local spatial consistency. On this basis, a new matching strategy and a novel local spatial filter, named respectively blob matching and Delaunay Triangulation Matching (DTM) are devised. Blob matching provides a general matching framework by merging together several strategies, including rank-based pre-filtering as well as many-to-many and symmetric matching, enabling to achieve a global improvement upon each individual strategy. DTM alternates between Delaunay triangulation contractions and expansions to figure out and adjust keypoint neighborhood consistency. Experimental evaluation shows that DTM is comparable or better than the state-of-the-art in terms of matching accuracy and robustness. Evaluation is carried out according to a new benchmark devised for analyzing the matching pipeline in terms of correct correspondences on both planar and non-planar scenes, including several state-of-the-art methods as well as the common SIFT matching approach for reference. This evaluation can be of assistance for future research in this field.

Vehicle trajectory data extraction from the horizontal curves of mountainous roads

ABSTRACT Trajectory data is essential for understanding the driver-vehicle-road interaction which is crucial for safety and operational assessment. The video image-processing technique is useful to collect such data on the entire traffic stream. Existing techniques assume that the road is planar which limits the application of the video-based traffic data collection to the plain terrains. With the objective to collect trajectory data from mountainous terrain, the present study proposes an approach to convert the pixel coordinates into real-world coordinates. The hypothesis was that a non-planar scene could be converted into a sequence of piece-wise planar scenes with a separate projection matrix for each planar region. The analysis shows that the proposed method could effectively transform the pixel coordinates into the real-world coordinates. Comparison of the estimated and observed speed/path profiles indicate the adequacy of the proposed method in collecting video-based trajectory data from mountainous terrain.

Unsupervised learning framework for interest point detection and description via properties optimization

This paper presents an unsupervised interest point detection and description method named Properties Optimization Point (POP), which provides a unified objective to optimize different properties of interest point. First, the proposed objective formulates the interest point set as a latent variable, which is flexible to integrate different properties. With the latent variable, the probability formulations are designed for four traditional properties (sparsity, repeatability, invariability, discriminability). Second, a novel property termed as informativeness indicating the information complexity of a local area is designed to determine the areas containing high information, from which interest points are encouraged to be extracted. Third, an efficient approximate Expectation Maximization is proposed to optimize the non-differentiable objective which integrates the above five properties. Finally, POP is instantiated with fully convolutional networks. Experimental results demonstrate that POP outperforms state-of-the-art methods on a number of image matching benchmarks containing both planar and non-planar scenes.

InpaintFusion: Incremental RGB-D Inpainting for 3D Scenes.

State-of-the-art methods for diminished reality propagate pixel information from a keyframe to subsequent frames for real-time inpainting. However, these approaches produce artifacts, if the scene geometry is not sufficiently planar. In this article, we present InpaintFusion, a new real-time method that extends inpainting to non-planar scenes by considering both color and depth information in the inpainting process. We use an RGB-D sensor for simultaneous localization and mapping, in order to both track the camera and obtain a surfel map in addition to RGB images. We use the RGB-D information in a cost function for both the color and the geometric appearance to derive a global optimization for simultaneous inpainting of color and depth. The inpainted depth is merged in a global map by depth fusion. For the final rendering, we project the map model into image space, where we can use it for effects such as relighting and stereo rendering of otherwise hidden structures. We demonstrate the capabilities of our method by comparing it to inpainting results with methods using planar geometric proxies.

Is There Anything New to Say About SIFT Matching?

SIFT is a classical hand-crafted, histogram-based descriptor that has deeply influenced research on image matching for more than a decade. In this paper, a critical review of the aspects that affect SIFT matching performance is carried out, and novel descriptor design strategies are introduced and individually evaluated. These encompass quantization, binarization and hierarchical cascade filtering as means to reduce data storage and increase matching efficiency, with no significant loss of accuracy. An original contextual matching strategy based on a symmetrical variant of the usual nearest-neighbor ratio is discussed as well, that can increase the discriminative power of any descriptor. The paper then undertakes a comprehensive experimental evaluation of state-of-the-art hand-crafted and data-driven descriptors, also including the most recent deep descriptors. Comparisons are carried out according to several performance parameters, among which accuracy and space-time efficiency. Results are provided for both planar and non-planar scenes, the latter being evaluated with a new benchmark based on the concept of approximated patch overlap. Experimental evidence shows that, despite their age, SIFT and other hand-crafted descriptors, once enhanced through the proposed strategies, are ready to meet the future image matching challenges. We also believe that the lessons learned from this work will inspire the design of better hand-crafted and data-driven descriptors.

A spatio-temporal multi-scale binary descriptor.

Binary descriptors are widely used for multi-view matching and robotic navigation. However, their matching performance decreases considerably under severe scale and viewpoint changes in non-planar scenes. To overcome this problem, we propose to encode the varying appearance of selected 3D scene points tracked by a moving camera with compact spatio-temporal descriptors. To this end, we first track interest points and capture their temporal variations at multiple scales. Then, we validate feature tracks through 3D reconstruction and compress the temporal sequence of descriptors by encoding the most frequent and stable binary values. Finally, we determine multiscale correspondences across views with a matching strategy that handles severe scale differences. The proposed spatio-temporal multi-scale approach is generic and can be used with a variety of binary descriptors. We show the effectiveness of the joint multiscale extraction and temporal reduction through comparisons of different temporal reduction strategies and the application to several binary descriptors.

A visible-light and infrared video database for performance evaluation of video/image fusion methods

In general, the fusion of visible-light and infrared images produces a composite representation where both data are pictured in a single image. The successful development of image/video fusion algorithms relies on realistic infrared/visible-light datasets. To the best of our knowledge, there is a particular shortage of databases with registered and synchronized videos from the infrared and visible-light spectra suitable for image/video fusion research. To address this need we recorded an image/video fusion database using infrared and visible-light cameras under varying illumination conditions. Moreover, different scenarios have been defined to better challenge the fusion methods, with various contexts and contents providing a wide variety of meaningful data for fusion purposes, including non-planar scenes, where objects appear on different depth planes. However, there are several difficulties in creating datasets for research in infrared/visible-light image fusion. Camera calibration, registration, and synchronization can be listed as important steps of this task. In particular, image registration between imagery from sensors of different spectral bands imposes additional difficulties, as it is very challenging to solve the correspondence problem between such images. Motivated by these challenges, this work introduces a novel spatiotemporal video registration method capable of generating registered and temporally aligned infrared/visible-light video sequences. The proposed workflow improves the registration accuracy when compared to the state-of-the art. By applying the proposed methodology to the recorded database we have generated the visible-light and infrared video database for image fusion, a publicly available database to be used by the research community to test and benchmark fusion schemes.

Drivable Road Reconstruction for Intelligent Vehicles Based on Two-View Geometry

This paper presents a road reconstruction algorithm based on two-view geometry. A general geometric model for both planar and nonplanar scenes is proposed with respect to a reference plane. Then, points in the scene can be described geometrically by the projective parallax with respect to the reference plane. In practice, the grounding plane of the vehicle is selected as the reference plane. A row-wise image registration method is proposed to estimate the parallax information effectively. Based on the geometric model, points in the scene can be reconstructed as relative position and height information with respect to the reference plane. For road detection, the scene can be segmented where the height changes abruptly. Then, the road region is obtained by a diffusion strategy. The road surface can be reconstructed using the proposed model. Experiments are conducted to show the effectiveness and robustness of the proposed approach. The proposed approach is general in the sense that it does not rely on the priors of road appearance and structure, and it is not restricted to specific stereo camera configurations making it extendable to other multicamera systems.

Multi-Viewpoint Panorama Construction With Wide-Baseline Images.

We present a novel image stitching approach, which can produce visually plausible panoramic images with input taken from different viewpoints. Unlike previous methods, our approach allows wide baselines between images and non-planar scene structures. Instead of 3D reconstruction, we design a mesh-based framework to optimize alignment and regularity in 2D. By solving a global objective function consisting of alignment and a set of prior constraints, we construct panoramic images, which are locally as perspective as possible and yet nearly orthogonal in the global view. We improve composition and achieve good performance on misaligned areas. Experimental results on challenging data demonstrate the effectiveness of the proposed method.

Kinectrack: 3D Pose Estimation Using a Projected Dense Dot Pattern.

Kinectrack is a novel approach to six-DoF tracking that provides agile real-time pose estimation using only commodity hardware. The dot pattern emitter and IR camera components of the standard Kinect device are separated to allow the emitter to roam freely relative to a fixed camera. The six-DoF pose of the emitter component is recovered by matching the dense dot pattern observed by the camera to a pre-captured reference image. A novel matching technique is introduced to obtain the dense dot pattern correspondences efficiently in wide- and adaptive-baseline scenarios that requires only a small subset of the full dense dot pattern to fall within the field of view of the fixed camera. An auto-calibration process is proposed in order to obtain the intrinsic parameters of the fixed camera and the internal dot pattern reference image of the emitter. The system simultaneously recovers the six-DoF pose of the emitter device and the piecewise planar 3D scene structure. Kinectrack provides a low-cost method for tracking an object without any on-board computation, with small size and only simple electronics. This paper extends the original ISMAR 2012 submission, including a demonstration of robust pose tracking for AR and examples of matching in planar and non-planar scenes.

Keypoint descriptor matching with context-based orientation estimation

This paper presents a matching strategy to improve the discriminative power of histogram-based keypoint descriptors by constraining the range of allowable dominant orientations according to the context of the scene under observation. This can be done when the descriptor uses a circular grid and quantized orientation steps, by computing or providing a global reference orientation based on the feature matches.The proposed matching strategy is compared with the standard approaches used with the SIFT and GLOH descriptors and the recent rotation invariant MROGH and LIOP descriptors. A new evaluation protocol based on an approximated overlap error is presented to provide an effective analysis in the case of non-planar scenes, thus extending the current state-of-the-art results.

Wide-baseline stereo matching based on the line intersection context for real-time workspace modeling

Line matching in widely separated views is challenging because of large perspective distortion and violation of the planarity assumption in local regions. We introduce a novel method of wide-baseline image matching based on the coplanar line intersections for poorly textured and/or nonplanar structured scenes. The local areas of the coplanar line pairs are normalized into canonical frames by rectifying the coplanar line pairs to be orthogonal. Then, the 3D interpretation of the intersection context of the coplanar line pairs helps to match the nonplanar local regions. Furthermore, for calibrated stereo cameras, we propose a matching criterion based on 3D planar homography to improve the matching accuracy while reconstructing most likely physically existing planar patches. Experimental results demonstrate the effectiveness of the proposed method for real-world scenes.

AIR: An Agent for Robust Image Matching and Retrieval

This paper presents a novel scheme coined AIR (Agent for Image Recognition), acting as an agent, to oversee the image matching and retrieval processes. Firstly, neighboring keypoints within close spatial proximity are examined and used to hypothesize true keypoint matches. While this approach is robust to noise (e.g. a tree) since spatial relation is considered, missing (undetected) keypoints in one image can also be recovered resulting in more keypoint matches. Secondly, the agent is able to recognize instability of projective transformations in certain cases (e.g. non-planar scenes). The geometric approach is substituted with LIS (Longest Increasing Subsequence) approach which does not require any complex geometric transformations. The effectiveness of AIR is substantiated by an image retrieval experiment which demonstrates that it achieves a twofold increase in true matches and higher matching accuracy when compared to RANSAC homography approach.

Motion-Estimation-Based Visual Servoing of Nonholonomic Mobile Robots

A 2-1/2-D visual servoing strategy, which is based on a novel motion-estimation technique, is presented for the stabilization of a nonholonomic mobile robot (which is also called the “parking problem”). By taking into account the planar motion constraint of mobile robots, the proposed motion-estimation technique can be applied in both planar and nonplanar scenes. In addition, this approach requires no matrix estimation or decomposition, and it avoids ambiguity and degeneracy problems for the homography or fundamental matrix-based algorithms. Moreover, the field-of-view (FOV) constraint of the onboard camera is largely alleviated because the presented algorithm works well with few feature points. In order to incorporate the advantages of position-based visual servoing and image-based visual servoing, a composite error vector is defined that includes both image signals and the estimated rotational angle. Subsequently, a smooth time-varying feedback controller is adopted to cope with the nonholonomic constraints, which yields global exponential convergent rate for the closed-loop system. On the basis of the perturbed linear system theory, we show that practical exponential stability can be achieved, despite the lack of depth information, which is inherent for monocular camera systems. Both simulation and experiment results are collected to investigate the feasibility of the proposed approach.

Improving Harris corner selection strategy

This study describes a corner selection strategy based on the Harris approach. Corners are usually defined as interest points for which intensity variation in the principal directions is locally maximised, as response from a filter given by the linear combination of the determinant and the trace of the autocorrelation matrix. The Harris corner detector, in its original definition, is only rotationally invariant, but scale-invariant and affine-covariant extensions have been developed. As one of the main drawbacks, corner detector performances are influenced by two user-given parameters: the linear combination coefficient and the response filter threshold. The main idea of the authors' approach is to search only the corners near enhanced edges and, by a z-score normalisation, to avoid the introduction of the linear combination coefficient. Combining these strategies allows a fine and stable corner selection without tuning the method. The new detector has been compared with other state-of-the-art detectors on the standard Oxford data set, achieving good results showing the validity of the approach. Analogous results have been obtained using the local detector evaluation framework on non-planar scenes by Fraundorfer and Bischof.

Shape from Defocus via Diffusion

Defocus can be modeled as a diffusion process and represented mathematically using the heat equation, where image blur corresponds to the diffusion of heat. This analogy can be extended to non-planar scenes by allowing a space-varying diffusion coefficient. The inverse problem of reconstructing 3-D structure from blurred images corresponds to an "inverse diffusion" that is notoriously ill-posed. We show how to bypass this problem by using the notion of relative blur. Given two images, within each neighborhood, the amount of diffusion necessary to transform the sharper image into the blurrier one depends on the depth of the scene. This can be used to devise a global algorithm to estimate the depth profile of the scene without recovering the deblurred image, using only forward diffusion.

Feature-based image mosaicing

We propose an automatic image mosaicing method which can construct a panoramic image from a collection of digital still images. Our method is fast and robust enough to process a nonplanar scene with unrestricted camera motion. The method includes the following two techniques. First, we use a multiresolution patch-based optical flow estimation for making feature correspondences to obtain a homography automatically. This enabled us to transform images fast and accurately. Second, we have developed a technique to obtain a homography from only three points, instead of four in general, in order to divide a scene into triangles. Experiments using real images confirm the effectiveness of our method. © 2000 Scripta Technica, Syst Comp Jpn, 31(7): 1–9, 2000