Keypoint Matching Research Articles

Example-Based 3D Trajectory Extraction of Objects From 2D Videos

For semantic analysis of activities and events in videos, it is important to capture the spatio-temporal relation among objects in the 3D space. In this paper, we present a probabilistic method that extracts 3D trajectories of objects from 2D videos, captured from a monocular moving camera. Compared with existing methods that rely on restrictive assumptions, we propose a method that can extract 3D trajectories with much less restriction by adopting new example-based techniques, which compensate the lack of information. Here, we estimate the focal length of the camera based on similar candidates, and use it to compute depths of detected objects. Contrary to other 3D trajectory extraction methods, our method is able to process videos taken from a stable camera as well as a non-calibrated moving camera without restrictions. For this, we modify Reversible Jump Markov Chain Monte Carlo particle filtering to be more suitable for camera odometry without relying on geometrical feature points. Moreover, our method decreases time consumption by reducing the number of object detections with keypoint matching. Finally, we evaluate our method on known data sets by showing the robustness of our system and demonstrating its efficiency in dealing with different kind of videos.

Efficient Model-Based Object Pose Estimation Based on Multi-Template Tracking and PnP Algorithms

Three-Dimensional (3D) object pose estimation plays a crucial role in computer vision because it is an essential function in many practical applications. In this paper, we propose a real-time model-based object pose estimation algorithm, which integrates template matching and Perspective-n-Point (PnP) pose estimation methods to deal with this issue efficiently. The proposed method firstly extracts and matches keypoints of the scene image and the object reference image. Based on the matched keypoints, a two-dimensional (2D) planar transformation between the reference image and the detected object can be formulated by a homography matrix, which can initialize a template tracking algorithm efficiently. Based on the template tracking result, the correspondence between image features and control points of the Computer-Aided Design (CAD) model of the object can be determined efficiently, thus leading to a fast 3D pose tracking result. Finally, the 3D pose of the object with respect to the camera is estimated by a PnP solver based on the tracked 2D-3D correspondences, which improves the accuracy of the pose estimation. Experimental results show that the proposed method not only achieves real-time performance in tracking multiple objects, but also provides accurate pose estimation results. These advantages make the proposed method suitable for many practical applications, such as augmented reality.

Keypoint Transfer for Fast Whole-Body Segmentation.

We introduce an approach for image segmentation based on sparse correspondences between keypoints in testing and training images. Keypoints represent automatically identified distinctive image locations, where each keypoint correspondence suggests a transformation between images. We use these correspondences to transfer the label maps of entire organs from the training images to the test image. The keypoint transfer algorithm includes three steps: 1) keypoint matching; 2) voting-based keypoint labeling; and 3) keypoint-based probabilistic transfer of organ segmentations. We report segmentation results for abdominal organs in whole-body CT and MRI, as well as in contrast-enhanced CT and MRI. Our method offers a speed-up of about three orders of magnitude in comparison with common multi-atlas segmentation while achieving an accuracy that compares favorably. Moreover, keypoint transfer does not require the registration to an atlas or a training phase. Finally, the method allows for the segmentation of scans with a highly variable field-of-view.

An Improved Randomized Local Binary Features for Keypoints Recognition.

In this paper, we carry out researches on randomized local binary features. Randomized local binary features have been used in many methods like RandomForests, RandomFerns, BRIEF, ORB and AKAZE to matching keypoints. However, in those existing methods, the randomness of feature operators only reflects in sampling position. In this paper, we find the quality of the binary feature space can be greatly improved by increasing the randomness of the basic sampling operator. The key idea of our method is to use a Randomized Intensity Difference operator (we call it RID operator) as a basic sampling operator to observe image patches. The randomness of RID operators are reflected in five aspects: grids, position, aperture, weights and channels. Comparing with the traditional incompletely randomized binary features (we call them RIT features), a completely randomized sampling manner can generate higher quality binary feature space. The RID operator can be used on both gray and color images. We embed different kinds of RID operators into RandomFerns and RandomForests classifiers to test their recognition rate on both image and video datasets. The experiment results show the excellent quality of our feature method. We also propose the evaluation criteria for robustness and distinctiveness to observe the effects of randomization on binary feature space.

OS-SIFT: A Robust SIFT-Like Algorithm for High-Resolution Optical-to-SAR Image Registration in Suburban Areas

Although the scale-invariant feature transform (SIFT) algorithm has been successfully applied to both optical image registration and synthetic aperture radar (SAR) image registration, SIFT-like algorithms have failed to register high-resolution (HR) optical and SAR images due to large geometric differences and intensity differences. In this paper, to perform optical-to-SAR (OS) image registration, we proposed an advanced SIFT-like algorithm (OS-SIFT) that consists of three main modules: keypoint detection in two Harris scale spaces, orientation assignment and descriptor extraction, and keypoint matching. Considering the inherent properties of SAR images and optical images, the multiscale ratio of exponentially weighted averages and multiscale Sobel operators are used to calculate consistent gradients for the SAR images and optical images on the basis of which, as a result, two Harris scale spaces can be constructed. Keypoints are detected by finding the local maxima in the scale space followed by a localization refinement method based on the spatial relationship of the keypoints. Moreover, gradient location orientation histogram-like descriptors are extracted using multiple image patches to increase the distinctiveness. The experimental results on simulated images and several HR satellite images show that the proposed OS-SIFT algorithm gives a robust registration result for optical-to-SAR images and outperforms other state-of-the-art algorithms in terms of registration accuracy.

Object Detection Algorithm Based on Gaussian Mixture Model and Sift Keypoint Match

Object Detection Algorithm Based on Gaussian Mixture Model and Sift Keypoint Match

A Contrario Comparison of Local Descriptors for Change Detection in Very High Spatial Resolution Satellite Images of Urban Areas

Change detection is a key problem for many remote sensing applications. In this paper, we present a novel unsupervised method for change detection between two high-resolution remote sensing images possibly acquired by two different sensors. This method is based on keypoints matching, evaluation, and grouping, and does not require any image co-registration. It consists of two main steps. First, global and local mapping functions are estimated through keypoints extraction and matching. Second, based on these mappings, keypoint matchings are used to detect changes and then grouped to extract regions of changes. Both steps are defined through an a contrario framework, simplifying the parameter setting and providing a robust pipeline. The proposed approach is evaluated on synthetic and real data from different optic sensors with different resolutions, incidence angles, and illumination conditions.

Robust object tracking via superpixels and keypoints

Most of the part-based methods just use the initial appearance model and feature information of the object. When the object is affected by occlusion, deformation and illumination factors, these methods can not be stable to the tracking object. In this paper, a tracking method is proposed based on keypoint matching and superpixel matching. Our method not only uses the initial feature information of the object, but also uses the feature information between adjacent frames. We use the superpixel to over-segment the candidate region which can be obtained by voting between the globally matched feature points, and then construct superpixel descriptors. The similarity between superpixels is based on the distance of the superpixel feature descriptor. Eventually, the object is selected according to the superpixel vote. Furthermore, we use qualitative and quantitative evaluations to evaluate our method on 18 challenging image sequences. Experimental results show that the proposed method outperforms 6 state-of-the-art tracking algorithms.

Automatic Detection of Video Shot Boundary in Social Media Using a Hybrid Approach of HLFPN and Keypoint Matching

Shot boundary detection (SBD) is an important and fundamental step in video content analysis such as content-based video indexing, browsing, and retrieval. In this paper, a hybrid SBD method is presented by integrating a high-level fuzzy Petri net (HLFPN) model with keypoint matching. The HLFPN model with histogram difference is executed as a predetection. Next, the speeded-up robust features (SURF) algorithm that is reliably robust to image affine transformation and illumination variation is used to figure out all possible false shots and the gradual transition based on the assumption from the HLFPN model. The top-down design can effectively lower down the computational complexity of SURF algorithm. The proposed approach has increased the precision of SBD and can be applied in different types of videos.

New Keypoint Matching Method Using Local Convolutional Features for Power Transmission Line Icing Monitoring.

Power transmission line icing (PTLI) problems, which cause tremendous damage to the power grids, has drawn much attention. Existing three-dimensional measurement methods based on binocular stereo vision was recently introduced to measure the ice thickness in PTLI, but failed to meet requirements of practical applications due to inefficient keypoint matching in the complex PTLI scene. In this paper, a new keypoint matching method is proposed based on the local multi-layer convolutional neural network (CNN) features, termed Local Convolutional Features (LCFs). LCFs are deployed to extract more discriminative features than the conventional CNNs. Particularly in LCFs, a multi-layer features fusion scheme is exploited to boost the matching performance. Together with a location constraint method, the correspondence of neighboring keypoints is further refined. Our approach achieves 1.5%, 5.3%, 13.1%, 27.3% improvement in the average matching precision compared with SIFT, SURF, ORB and MatchNet on the public Middlebury dataset, and the measurement accuracy of ice thickness can reach 90.9% compared with manual measurement on the collected PTLI dataset.

Adaptive Correlation Model for Visual Tracking Using Keypoints Matching and Deep Convolutional Feature.

Although correlation filter (CF)-based visual tracking algorithms have achieved appealing results, there are still some problems to be solved. When the target object goes through long-term occlusions or scale variation, the correlation model used in existing CF-based algorithms will inevitably learn some non-target information or partial-target information. In order to avoid model contamination and enhance the adaptability of model updating, we introduce the keypoints matching strategy and adjust the model learning rate dynamically according to the matching score. Moreover, the proposed approach extracts convolutional features from a deep convolutional neural network (DCNN) to accurately estimate the position and scale of the target. Experimental results demonstrate that the proposed tracker has achieved satisfactory performance in a wide range of challenging tracking scenarios.

RAPID: Measuring Deformation of Biological Tissues from MR Images Through the Riemannian Pseudo Kernel

Due to the nonlinear deformation of nonrigid and nonuniform tissues, it is challenging to accurately measure the displacements of feature points distributed on the inner parts, boundaries, and separatrices of tissue layers. To address this challenge, we propose a feature point matching technique called RAPID to measure MR 2D slice deformation of nonuniform and nonrigid biological tissues. We propose to use the covariance of several neighboring point statistics computed around a keypoint, as the keypoint descriptor. Inspired by the kernel methods, we advocate adopting a Riemannian pseudo kernel to map SPD matrices to a high dimensional Hilbert space, where the Euclidean geometry applies. We compare our RAPID with two existing schemes (i.e., SIFT and SURF). Our experimental results show that our RAPID is superior to SIFT and SURF, because the benefits offered by RAPID are two-fold. First, our RAPID increases the number of matched data points. Second, RAPID substantially improves the key-point matching accuracy of SIFT and SURF.

Identifying Corresponding Patches in SAR and Optical Images With a Pseudo-Siamese CNN

In this letter, we propose a pseudo-siamese convolutional neural network (CNN) architecture that enables to solve the task of identifying corresponding patches in very-high-resolution (VHR) optical and synthetic aperture radar (SAR) remote sensing imagery. Using eight convolutional layers each in two parallel network streams, a fully connected layer for the fusion of the features learned in each stream, and a loss function based on binary cross-entropy, we achieve a one-hot indication if two patches correspond or not. The network is trained and tested on an automatically generated dataset that is based on a deterministic alignment of SAR and optical imagery via previously reconstructed and subsequently co-registered 3D point clouds. The satellite images, from which the patches comprising our dataset are extracted, show a complex urban scene containing many elevated objects (i.e. buildings), thus providing one of the most difficult experimental environments. The achieved results show that the network is able to predict corresponding patches with high accuracy, thus indicating great potential for further development towards a generalized multi-sensor key-point matching procedure. Index Terms-synthetic aperture radar (SAR), optical imagery, data fusion, deep learning, convolutional neural networks (CNN), image matching, deep matching

A Novel Correspondence Selection Technique for Affine Rigid Image Registration

This paper presents a novel technique called correspondence selection for rigid transformations in order to effectively refine keypoint matches in rigid image registration. The proposed technique mainly lies in the following two components. First, keypoint matches are ranked and selected by the distance ratio between the best match and the second best match. Second, keypoint matches are further selected by ranking the geometric similarity between corresponding keypoint triplets. These two components enhance the discriminative power of potential keypoint matches in a progressive way. The proposed technique is generally applicable to affine rigid image registration. Experiments have been conducted using a set of benchmark datasets in the field of image registration, indicating that the proposed technique is very effective and achieves the state-of-the-art performance in refining keypoint matches for affine rigid image registration.

An Evaluation on the Robustness of Five Popular Keypoint Descriptors to Image Modifications Specific to Laser Scanning Microscopy

Laser scanning microscopy (LSM) techniques are of paramount importance at this time for key domains such as biology, medicine, or materials science. Computer vision methods are instrumental for boosting the potential of LSM, providing reliable results for important tasks, such as image segmentation, registration, classification, or retrieval in a fraction of the time that a human expert would require (at similar or even higher accuracy levels). Image keypoint extraction and description represent essential building blocks of modern computer vision approaches, and the development of such techniques has gained massive interest over the past couple of decades. In this paper, we compare side-by-side five popular keypoint description techniques, scale invariant feature transform (SIFT), speeded-up robust features (SURF), binary robust invariant scalable keypoints (BRISK), fast retina keypoint (FREAK) and BLOCK, with respect to their capacity to represent in a reproducible manner image regions contained in LSM data sets acquired under different acquisition conditions. We evaluate this capacity in terms of descriptor matching performance, using data sets acquired in a principled manner and a thorough Precision-Recall analysis. We identify which of the five evaluated techniques is most robust to specific LSM image modifications associated to the laser beam power, photomultiplier gain, or pixel dwell, and show that certain pre-processing steps have the potential to enhance keypoint matching.

Effective Descriptors based Face Recognition Technique for Robotic Surveillance Systems

The aim is to propose SPHORB Face Recognition technique for Robotic Surveillance systems. Surveillance bots with efficient face recognition system becomes even more powerful and adds to the applications of the bots. In this paper we compare the results of face recognition using SPHORB (a new fast and robust binary feature detector and descriptor for spherical panoramic image) [1] algorithm with ORB (Oriented Fast and Rotated Brief) [4] algorithm for classifying 2D and 3D images Database. In the first part 2D face images are randomly selected from LFW [32] with good sample of images across gender and ethnicity. The number of keypoints that were identified and number of matching keypoints between the 2D images using SPHORB and ORB were compared. From the results we determine that the SPHORB algorithm has identified more keypoints for the same 2D image than ORB, it has processed more images but ORB gives better accuracy than SPHORB for 2D images. In the next part we investigated the image matching between a front face 2D image with a 3D image. The front face image and 3D images are randomly selected from the ThatsMyFace.com [2] with good sample of images across genders, ethnicity and age. The front face image is compared with a randomly selected 3D image from the sample class set using SPHORB and ORB algorithms. The number of keypoints that were identified and number of matching keypoints between images using SPHORB and ORB were compared. From the results we determine that the SPHORB algorithm has identified more keypoints for the same image than ORB, it has processed more images and has accuracy comparable to the ORB algorithm.

Scalable object instance recognition based on keygraph matching

We propose a generalisation of the local feature matching framework, where keypoints are replaced by k-keygraphs, i.e., isomorphic directed attributed graphs of cardinality k whose vertices are keypoints. Keygraphs have structural and topological properties which are discriminative and efficient to compute, based on graph edge length and orientation as well as vertex scale and orientation. Keypoint matching is performed based on descriptor similarity. Next, 2-keygraphs are calculated; as a result, the number of incorrect keypoint matches reduced in 75% (while the correct keypoint matches were preserved). Then, 3-keygraphs are calculated, followed by 4-keygraphs; this yielded a significant reduction of 99% in the number of remaining incorrect keypoint matches. The stage that finds 2-keygraphs has a computational cost equal to a small fraction of the cost of the keypoint matching stage, while the stages that find 3-keygraphs or 4-keygraphs have a negligible cost. In the final stage, RANSAC finds object poses represented as affine transformations mapping images. Our experiments concern large-scale object instance recognition subject to occlusion, background clutter and appearance changes. By using 4-keygraphs, RANSAC needed 1% of the iterations in comparison with 2-keygraphs or simple keypoints. As a result, using 4-keygraphs provided a better efficiency as well as allowed a larger number of initial keypoints matches to be established, which increased performance.

Image feature detection using an improved implementation of maximally stable extremal regions for augmented reality applications

ABSTRACTAugmented Reality for all practical purposes requires extensive computation, accurate view alignment and real-time performance. To address some of these limitations, an improved method of feature detection is proposed using Maximally Stable Extremal Regions. The approach used for feature detection extracts the regions of interest using a true flood fill approach for building and maintaining the component tree. This approach has true worst-case linear complexity (Linear-MSER). In the present work, Linear-MSER is implemented at multiple scales of an image in order to increase the affine invariance properties of the detector (MSLinear-MSER).The two detectors, Linear-MSER and MSLinear-MSER, are then combined separately with Scale Invariant Feature Transform and Speeded-Up Robust Feature descriptors for performance comparison. Performance evaluation is done under varying imaging conditions like changes in viewpoint, scale, blur, illumination and JPEG compression. Results show that, MSLinear-MSER+SIFT performs best in terms of time-complexity and number of keypoint matches when executed at six octaves and five levels. This observation is true for all image-sets taken into consideration, containing images that are affine transformed in one way or other. To exhibit the efficiency of MSLinear-MSER+SIFT, a prototype of an AR system is also developed and discussed in this article using this approach.

Fast shape-from-template using local features

Reconstructing the 3D surface of an object using only a single image is a challenging task, which has recently attracted attention. In this paper, a template-based approach is presented to reconstruct the surface of an isometric deformable object. The proposed approach brings a solution for a class of computer vision problems named shape-from-template (SfT). In SfT, the goal is to solve single-image reconstruction for an object given its 3D template model in some rest shape. To this end, corresponding keypoints between the template and the so-called deformed image are first established. Then, a very fast method is used to estimate the first-order differential flow around the extracted keypoint pairs as an affine transformation. This is done using the keypoint pairs’ surrounding texture patch. In our method, we estimate this affine transformation using the keypoint pairs’ closest neighbors. This is both faster and more stable. Finally, the depth of each keypoint in the deformed image is estimated from its associated affine transformation. The robustness of keypoint matching is essential to the process. Indeed, outliers defeat depth estimation dramatically. We propose two new approaches to detect and remove the possible outliers based on geometrical properties of the matched keypoints. These two geometrical outlier removal approaches are faster than existing ones and can be used with almost any image descriptor. Experimental results show that the proposed approaches are very effective and outperform existing ones.

Color Balance and Fusion for Underwater Image Enhancement.

We introduce an effective technique to enhance the images captured underwater and degraded due to the medium scattering and absorption. Our method is a single image approach that does not require specialized hardware or knowledge about the underwater conditions or scene structure. It builds on the blending of two images that are directly derived from a color-compensated and white-balanced version of the original degraded image. The two images to fusion, as well as their associated weight maps, are defined to promote the transfer of edges and color contrast to the output image. To avoid that the sharp weight map transitions create artifacts in the low frequency components of the reconstructed image, we also adapt a multiscale fusion strategy. Our extensive qualitative and quantitative evaluation reveals that our enhanced images and videos are characterized by better exposedness of the dark regions, improved global contrast, and edges sharpness. Our validation also proves that our algorithm is reasonably independent of the camera settings, and improves the accuracy of several image processing applications, such as image segmentation and keypoint matching.