Keypoint Matching Research Articles

Keypoint Transfer Segmentation.

We present an image segmentation method that transfers label maps of entire organs from the training images to the novel image to be segmented. The transfer is based on sparse correspondences between keypoints that represent automatically identified distinctive image locations. Our segmentation algorithm consists of three steps: (i) keypoint matching, (ii) voting-based keypoint labeling, and (iii) keypoint-based probabilistic transfer of organ label maps. We introduce generative models for the inference of keypoint labels and for image segmentation, where keypoint matches are treated as a latent random variable and are marginalized out as part of the algorithm. We report segmentation results for abdominal organs in whole-body CT and in contrast-enhanced CT images. The accuracy of our method compares favorably to common multi-atlas segmentation while offering a speed-up of about three orders of magnitude. Furthermore, keypoint transfer requires no training phase or registration to an atlas. The algorithm's robustness enables the segmentation of scans with highly variable field-of-view.

Log-Spiral Keypoint: A Robust Approach toward Image Patch Matching.

Matching of keypoints across image patches forms the basis of computer vision applications, such as object detection, recognition, and tracking in real-world images. Most of keypoint methods are mainly used to match the high-resolution images, which always utilize an image pyramid for multiscale keypoint detection. In this paper, we propose a novel keypoint method to improve the matching performance of image patches with the low-resolution and small size. The location, scale, and orientation of keypoints are directly estimated from an original image patch using a Log-Spiral sampling pattern for keypoint detection without consideration of image pyramid. A Log-Spiral sampling pattern for keypoint description and two bit-generated functions are designed for generating a binary descriptor. Extensive experiments show that the proposed method is more effective and robust than existing binary-based methods for image patch matching.

Personal identification based on multiple keypoint sets of dorsal hand vein images

This paper presents a biometric identification system based on near-infrared imaging of dorsal hand veins and matching of the keypoints that are extracted from the dorsal hand vein images by the scale-invariant feature transform. The whole system is covered in detail, which includes the imaging device used, image processing methods proposed for geometric correction, region-of-interest extraction, image enhancement and vein pattern segmentation, as well as image classification by extraction and matching of keypoints. In addition to several constraints introduced to minimise incorrectly matched keypoints, a particular focus is placed on the use of multiple training images of each hand class to improve the recognition performance for a large database with more than 200 hand classes. By organising multiple keypoint sets extracted from multiple training images of each hand class into three sets, namely, the union, the intersection and the exclusion, based on their inter-class and intra-class relationships, this study shows the contribution made by each set to the recognition performance and demonstrates the feasibility of achieving 100% correct recognition by combining the three sets, based on the experiments conducted using more than 2000 dorsal hand vein images.

Automating the Photogrammetric Bridging Based on MMS Image Sequence Processing

Abstract. The photogrammetric bridging or traverse is a special bundle block adjustment (BBA) for connecting a sequence of stereo-pairs and of determining the exterior orientation parameters (EOP). An object point must be imaged in more than one stereo-pair. In each stereo-pair the distance ratio between an object and its corresponding image point varies significantly. We propose to automate the photogrammetric bridging based on a fully automatic extraction of homologous points in stereo-pairs and on an arbitrary Cartesian datum to refer the EOP and tie points. The technique uses SIFT algorithm and the keypoint matching is given by similarity descriptors of each keypoint based on the smallest distance. All the matched points are used as tie points. The technique was applied initially to two pairs. The block formed by four images was treated by BBA. The process follows up to the end of the sequence and it is semiautomatic because each block is processed independently and the transition from one block to the next depends on the operator. Besides four image blocks (two pairs), we experimented other arrangements with block sizes of six, eight, and up to twenty images (respectively, three, four, five and up to ten bases). After the whole image pairs sequence had sequentially been adjusted in each experiment, a simultaneous BBA was run so to estimate the EOP set of each image. The results for classical ("normal case") pairs were analyzed based on standard statistics regularly applied to phototriangulation, and they show figures to validate the process.

An efficient registration algorithm based on spin image for LiDAR 3D point cloud models

Spin image is a good point feature descriptor of the 3D surface and has been used in model registration for many applications from medical image processing to cooperation of multiple robots. However, researches show that current Spin-Image based Registration (SIR) algorithms present disadvantages in computational efficiency and robustness. Thus in this paper, aiming at 3D model acquired from LiDAR sensor, a new SIR algorithm is proposed to solve these problems. The new algorithm is on the basis of a new-constructed three-dimensional feature space, which, composed of the curvature, the Tsallis entropy of spin image, and the reflection intensity of laser sensor, is combined with the concept of KD-tree to firstly realize the primary key point matching, i.e., to find the Corresponding Point Candidate Set (CPCS). After that, spin-image based corresponding point searching is conducted with respect to each CPCS to precisely obtain the final corresponding points. The most absorbing advantages of the proposed method are as the following two aspects: on one hand, due to the introduction of the extra features, the fault corresponding relation introduced by spin image based method can be effectively reduced and thus the registration precision and robustness can be improved greatly; on the other hand, the CPCS obtained using low-dimensional feature space and KD-tree reduces extraordinarily the computational burden due to spin-image based correspondence searching. This greatly improves the computational efficiency of the proposed algorithm. Finally, in order to verify the feasibility and validity of the proposed algorithm, experiments are conducted and the results are analyzed.

BIG-OH: BInarization of gradient orientation histograms

Extracting local keypoints and keypoint descriptions from images is a primary step for many computer vision and image retrieval applications. In the literature, many researchers have proposed methods for representing local texture around keypoints with varying levels of robustness to photometric and geometric transformations. Gradient-based descriptors such as the Scale Invariant Feature Transform (SIFT) are among the most consistent and robust descriptors. The SIFT descriptor, a 128-element vector consisting of multiple gradient histograms computed from local image patches around a keypoint, is widely considered as the gold standard keypoint descriptor. However, SIFT descriptors require at least 128bytes of storage per descriptor. Since images are typically described by thousands of keypoints, it may require more space to store the SIFT descriptors for an image than the original image itself. This may be prohibitive in extremely large-scale applications and applications on memory-constrained devices such as tablets and smartphones. In this paper, with the goal of reducing the memory requirements of keypoint descriptors such as SIFT, without affecting their performance, we propose BIG-OH, a simple yet extremely effective method for binary quantization of any descriptor based on gradient orientation histograms. BIG-OH's memory requirements are very small—when it uses SIFT's default parameters for the construction of the gradient orientation histograms, it only requires 16bytes per descriptor. BIG-OH quantizes gradient orientation histograms by computing a bit vector representing the relative magnitudes of local gradients associated with neighboring orientation bins. In a series of experiments on keypoint matching with different types of keypoint detectors under various photometric and geometric transformations, we find that the quantized descriptor has performance comparable to or better than other descriptors, including BRISK, CARD, BRIEF, D-BRIEF, SQ, and PCA-SIFT. Our experiments also show that BIG-OH is extremely effective for image retrieval, with modestly better performance than SIFT. BIG-OH's drastic reduction in memory requirements, obtained while preserving or improving the image matching and image retrieval performance of SIFT, makes it an excellent descriptor for large image databases and applications running on memory-constrained devices.

An automated 3D registration method for optical coherence tomography volumes.

Optical coherence tomography (OCT) is able to provide high resolution volumetric data for biological tissues. However, the field of view (FOV) of OCT is sometimes smaller than the field of interest, which limits the clinical application of OCT. One way to overcome the drawback is to stitch multiple 3D volumes. In this paper, we propose a novel method to register multiple overlapped volumetric OCT data into a single volume. The relative positions of overlapped volumes were estimated on en face plane and at depth. On en face plane, scale invariant feature transform (SIFT) was implemented to extract the keypoints in each volume. Based on the invariant features, volumes were paired through keypoint matching. Then, we formulated the relationship between paired offsets and absolute positions as a linear model and estimated the centroid of each volume using least square method. Moreover, we calibrated the depth displacement in each paired volume and aligned the z coordinates of volumes globally. The algorithm was validated through stitching multiple volumetric OCT datasets of human cervix tissue and of swine heart. The experimental results demonstrated that our method is capable of visualizing biological samples over a wider FOV, which enhances the investigation of tissue structure such as fiber orientation.

Key point detection by max pooling for tracking.

Inspired by the recent image feature learning work, we propose a novel key point detection approach for object tracking. Our approach can select mid-level interest key points by max pooling over the local descriptor responses from a set of filters. Linear filters are first learned from targets in first frames. Then max pooling is performed over data driven spatial supporting field to detect discriminant key points, and thus the detected key points bear higher level semantic meanings, which we apply in tracking by structured key point matching. We show that our tracking system is robust to occlusions and cluttered background. Testing on several challenging tracking sequences, we demonstrate that our proposed tracking system can achieve competitive or better performances than the state-of-the-art trackers.

Improved SIFT match for optical satellite images registration by size classification of blob-like structures

Due to high rate of false match and expensive computation cost, the existing scale-invariant feature transform (SIFT) operators are not efficient to register two optical satellite images that have a great spatial resolution difference. Some scale restriction schemes were proposed to reduce the false match of SIFT keypoints and computational cost. However, it is observed that many keypoints are still not correctly matched. This problem often leads to the failure of automatic registration in industrial applications. To solve this problem, the images being registered are normalized to adjust the scale of blob-like structures and to preclude useless blob-like structures. Then blob-like structures are classified according to their physical sizes, and keypoint matching is restricted to matching for the blob-like structures having the same physical sizes. The scale normalization and size classification significantly improve the correct match rate as well as computational cost. Experiments on two pairs of satellite images demonstrate the effectiveness of the proposed method.

SIFT와 다중측면히스토그램을 이용한 다중물체추적

In multiple object tracking, accurate detection for each of objects that appear sequentially and effective tracking in complicated cases that they are overlapped with each other are very important. In this paper, we propose a multiple object tracking system that has a concrete detection and tracking characteristics by using multi-lateral histogram and SIFT feature extraction algorithm. Especially, by limiting the matching area to object's inside and by utilizing the location informations in the keypoint matching process of SIFT algorithm, we advanced the tracking performance for multiple objects. Based on the experimental results, we found that the proposed tracking system has a robust tracking operation in the complicated environments that multiple objects are frequently overlapped in various of directions.

A Spherical Model Based Keypoint Descriptor and Matching Algorithm for Omnidirectional Images

Omnidirectional images generally have nonlinear distortion in radial direction. Unfortunately, traditional algorithms such as scale-invariant feature transform (SIFT) and Descriptor-Nets (D-Nets) do not work well in matching omnidirectional images just because they are incapable of dealing with the distortion. In order to solve this problem, a new voting algorithm is proposed based on the spherical model and the D-Nets algorithm. Because the spherical-based keypoint descriptor contains the distortion information of omnidirectional images, the proposed matching algorithm is invariant to distortion. Keypoint matching experiments are performed on three pairs of omnidirectional images, and comparison is made among the proposed algorithm, the SIFT and the D-Nets. The result shows that the proposed algorithm is more robust and more precise than the SIFT, and the D-Nets in matching omnidirectional images. Comparing with the SIFT and the D-Nets, the proposed algorithm has two main advantages: (a) there are more real matching keypoints; (b) the coverage range of the matching keypoints is wider, including the seriously distorted areas.

A new spin image based registration algorithm of 3D surrounding model in low-dimensional feature space

Most recently, cooperation and coordination among different robots (CCDR), e.g., the air robot and the ground robot, has gradually been a new researching topic in the field of robotics. Among several challenging problems in CCDR, surrounding model registration is very important and difficult, because the models from different robots are usually of different scale and obtained from completely different viewpoints. Currently, very little algorithms have been reported to be feasible for this problem, wherein spin-image based scheme has achieved much attention. However, researches have showed that spin-image based methods present disadvantages in computational efficiency and robustness. Therefore, in this paper, a new spin image based 3D surrounding model registration algorithm is proposed. The new algorithm is on the basis of a three-dimensional feature space, which is composed by the curvature, the Tsallis entropy of spin image, and the reflection intensity of the laser sensor, and combined with the concept of KD-tree to firstly realize the primary key point matching, i.e., to find the corresponding point candidate set (CPCS). After that, spin-image based corresponding point searching is conducted with respect to each CPCS to obtain the accurate corresponding point relation. The most absorbing advantages of the proposed scheme are as the following two aspects: on one hand, due to the three extra features, the fault corresponding relation can be reduced effectively and thus the algorithm precision and robustness can be improved greatly; on the other hand, the CPCS obtained by using the KD-tree method in the constructed low-dimensional feature space contains much less points and thus the computational burden due to spin-image searching is reduced greatly. Finally, in order to verify the feasibility and validity of the proposed algorithm, experiments are conducted and the results are analyzed.

SURF-based Human Tracking Algorithm with On-line Update of Object Model

Abstract The ability to robustly track a human is an essential prerequisite to an increasing number of applications that needs to interact with a human user. This paper presents a robust vision based algorithm to track a human in a dynamic environment using interest point-based method. The tracking algorithm is expected to cope with changes in pose, scale, illumination as well as camera motion. The interest point based (e.g. SURF) tracking methods suffer from the limitation of unavailability of sufficient number of matching key points for the target in all frames of a running video. One solution to this problem is to have an object model which contains SURF features for all possible poses and scaling factors. So an object model with all possible descriptors could be created off-line and could be used for detecting the target in each and every frame. However, such a scheme can not be used for tracking an object online. In order to overcome this problem, we propose a new approach which update the object model online and have sufficient matching key points for the target in case of change in the pose as well as scaling. Experimental results are provided to show the efficacy of the algorithm.

Automatic GCP Extraction of Fully Polarimetric SAR Images

This paper presents a method for automatic extraction of ground control points (GCPs) of fully polarimetric synthetic aperture radar (SAR) (PolSAR) images obtained from various satellites with different viewing angles. The scale-invariant feature transform (SIFT) algorithm is applied to extract candidate GCPs, where two-way keypoint matching eliminates improbable correspondence keypoints. Minimizing the root-mean-square error (rmse) also removes matching points with large rmse through a pseudoaffine transformation. In addition, information entropy and spatial dispersion quality constraints enable quantification of the spatial distribution of the GCPs. In accordance with full polarization, applying the SIFT-OCT algorithm (SIFT algorithm with the first scale-space octave skipped) to PolSAR data is examined. The total power (TP) image represents a combination of the characteristics of all four polarization images [horizontal transmitting and horizontal receiving (HH), horizontal transmitting and vertical receiving (HV), vertical transmitting and horizontal receiving (VH), and vertical transmitting and vertical receiving (VV)]. Therefore, GCP extraction using a TP image rather than each polarization image is proposed in order to maximize the accuracy of GCP extraction for all of the polarization data, as the TP image generates the highest signal-to-noise ratio (SNR) value. The SNR in conjunction with the matching correlation surface is used as an indicator of the reliability and accuracy of GCP extraction. After successfully applying the method to Advanced Land Observing Satellite/Phased Array type L-band Synthetic Aperture Radar and Japanese Earth Resources Satellite-1 SAR images, the GCP matching accuracy is further improved by using geometric calibration.

A Biometric Based: 3-D Ear Recognition System Combining Local and Holistic Features

This paper introduces an improved ear recognition approach based on 3 dimensional keypoint matching and combining local and holistic features. At first, the 3D keypoints are detected using the shape index image. The system consists of four primary steps: i) ear image segmentation; ii) local feature extraction and matching; iii) holistic feature extraction and matching; and iv) combination of local and holistic features at the match score level. For the segmentation purpose, we use an efficient skin segmentation algorithm, to localize a rectangular region containing the ear. For the local feature extraction and representation purpose, we use the Sparse Representation based Localized Feature Extraction. For the holistic matching component, we introduce a voxelization scheme for holistic ear representation. The match scores obtained from both the local and holistic matching components are combined to generate the final match scores.

Image matching based local Delaunay triangulation and affine invariant geometric constraint

Finding correspondences between two sets of visual features of 2D image is a key problem in computer vision tasks. In practice situation, objects often undergo occlusion, background clutter, illumination and 3D viewpoint changes which increase the difficulties of matching, thus a robust matching method is needed to tackle with these problems. In this paper, we propose a robust approach to image matching based on Hessian affine region detector and local Delaunay triangulation and affine invariant geometric constraint. Firstly, Hessian affine keypoints with Scale Invariant Feature Transform(SIFT) descriptors are extracted for model and scene image, then matching the Hessian affine keypoints to get initial matched keypoints based on Euclidean distance of their (SIFT) descriptors. Third, Delaunay triangulation of the initial matched Hessian affine keypoints. Based on unique topological structure of Delaunay triangulation of a set of keypoints, we divide the keypoints into two classes, one class with the same number of neighbor triangles, the other class with different number of neighbor triangles, from the two classes of keypoints, we obtained matched triangles by local Delaunay triangulation. There are may have wrong matched triangles, we adopt local affine invariant geometric constraint to refine the matched triangles. Finally, get the final matched keypoints from the refined matched triangles. By using three pairs of images, the experimental results indicate that the proposed method can get higher correctness of image matching than RANSAC based method.

Multiexposure and multifocus image fusion with multidimensional camera shake compensation

Multiexposure image fusion algorithms are used for enhancing the perceptual quality of an image captured by sensors of limited dynamic range. This is achieved by rendering a single scene based on multiple images captured at different exposure times. Similarly, multifocus image fusion is used when the limited depth of focus on a selected focus setting of a camera results in parts of an image being out of focus. The solution adopted is to fuse together a number of multifocus images to create an image that is focused throughout. A single algorithm that can perform both multifocus and multiexposure image fusion is proposed. This algorithm is a new approach in which a set of unregistered multiexposure/focus images is first registered before being fused to compensate for the possible presence of camera shake. The registration of images is done via identifying matching key-points in constituent images using scale invariant feature transforms. The random sample consensus algorithm is used to identify inliers of SIFT key-points removing outliers that can cause errors in the registration process. Finally, the coherent point drift algorithm is used to register the images, preparing them to be fused in the subsequent fusion stage. For the fusion of images, a new approach based on an improved version of a wavelet-based contourlet transform is used. The experimental results and the detailed analysis presented prove that the proposed algorithm is capable of producing high-dynamic range (HDR) or multifocus images by registering and fusing a set of multiexposure or multifocus images taken in the presence of camera shake. Further, comparison of the performance of the proposed algorithm with a number of state-of-the art algorithms and commercial software packages is provided. In particular, our literature review has revealed that this is one of the first attempts where the compensation of camera shake, a very likely practical problem that can result in HDR image capture using handheld devices, has been addressed as a part of a multifocus and multiexposure image enhancement system.

Editorial Issue 24.5

This issue contains 5 regular papers. Yuanfeng Zhu, Ajay Sundar Ramakrishnan, Bernd Hamann, and Michael Neff, from University of California in Davis, USA, present a new system for automatically generating three dimensional animations of piano performance, given an input midi music file. A graph theory-based motion planning method is used to decide which set of fingers should strike the piano keys for each chord. Anticipating the progression of the music, the positions of unused fingers are calculated to make possible efficient fingering of future notes. Initial key poses of the hands, including those for complex piano techniques such as crossovers and arpeggio, are determined based on the finger positions and piano theory. An optimization method is used to refine these poses, producing a natural and minimal energy pose sequence. The second paper by Mathieu Perriollat, from VI-Technology in Grenoble, France and Adrien Bartoli, from Université d'Auvergne in Clermont Ferrand, France, deals with developable surface reconstruction from real observations. Most of the existing developable surface parameterizations do not handle boundaries or are driven by overly large parameter sets. The main contribution of this paper is a generative model of bounded developable surfaces that solves these two issues. The model is governed by intuitive parameters whose number depends on the actual deformation and includes the `flat shape boundary'. The authors also propose a 3D reconstruction method well adapted to the use of key point matches over multiple images. The third paper by Fabien Tence from Virtualys in Brest, France, and Laurent Gaubert, Julien Soler, Pierre De Loor, and Cédric Buche, from Université de Bretagne Occidentale in Plouzané, France, propose a method for video games to create programs whose behaviors cannot be told apart from players when observed playing the game. To achieve this goal, the authors choose models using Markov chains to generate the behaviors by imitation. They propose a new model, called CHAMELEON, to enhance expressiveness and the associated imitation learning algorithm. They first organize the sensors and motors by semantic refinement and add a focus mechanism in order to improve the believability. Then, they integrate an algorithm to learn the topology of the environment which tries to best represent the use of the environment by the players. In the fourth paper, MingQi Yu and HongYan Quan, from China Normal University in Shanghai, China, put forward a hierarchical method of fluid surface modeling in natural landscapes. The proposed method produces a visually plausible surface geometry with the texture from a single video image recorded by a standard video device. In contrast with the conventional physically-based fluid simulation, the new method computes preliminary results using empirical method and adopts Stokes wave model to obtain the reconstruction result. The authors illustrate the working of system with a wide range of possible scene, and a qualitative evaluation of their method is provided to verify the quality of the surface geometry. The experiment shows that the method can meet the requirement of real-time performance and the reality of the fluid. The last paper by Oktar Ozgen, Marcelo Kallmann, Carlos Coimbra, from University of California at Merced, USA, and Selcuk Sumengen and Selim Balcisoy, from Sabanci University, Istanbul, Turkey, describes a new method based on the use of fractional differentiation in order to improve the efficiency of simulations based on Smoothed Particle Hydrodynamics (SPH). The proposed approach is based on the observation that the effects requiring a high concentration of particles are most often produced from colliding flows, and therefore by achieving a better modeling of this behavior with the use of fractional derivatives similar high-quality results can be achieved with a lower number of particles. As a result, the new method can be used to reduce the resolution without significant loss of quality, or to improve the quality of the simulation in the current chosen resolution.

Tracking of facial deformations in multi-image sequences with elimination of rigid motion of the head

The paper deals with measurement of human facial deformations from synchronized image sequences taken with multiple calibrated cameras from different viewpoints. SIFT (Scale Invariant Feature Transform) keypoints are utilized as image feature points in the first place to determine spatial and temporal correspondences between images. If no temporal match is found for an image point by keypoint matching, then the tracking of the point is switched to least squares matching provided the point has one or more spatial corresponding points in the other views of the previous frame. For this purpose, a new method based on affine multi-image least squares matching is proposed where multiple spatial and temporal template images are simultaneously matched against each search image and part of the spatial template images also change during adjustment. A new method based on analyzing temporal changes in the image coordinates of the tracked points in multiple views is then presented for detecting the 3-D points which move only rigidly between consecutive frames. These points are used to eliminate the effect of rigid motion of the head and to obtain the changes in the 3-D points and in the corresponding image points due to pure deformation of the face. The methods are thoroughly tested with three multi-image sequences of four cameras including also quite large changes of facial deformations. The test results prove that the proposed affine multi-image least squares matching yields better results than another method using only fixed templates of the previous frame. The elimination of the effect of rigid motion works well and the points where the face is deforming can be correctly detected and the true deformation estimated. A method based on a novel adaptive threshold is also proposed for automated extraction and tracking of circular targets on a moving calibration object.

High Performance Online Image Search with GPUs on Large Image Databases

The authors propose an online image search engine based on local image keypoint matching with GPU support. State-of-the-art models are based on bag-of-visual-words, which is an analogy of textual search for visual search. In this work, thanks to the vector computation power of the GPU, the authors utilize real values of keypoint descriptors and realize real-time search at keypoint level. By keeping the identities of each keypoint, closest keypoints are accurately retrieved. Image search has different characteristics than textual search. The authors implement one-to-one keypoint matching, which is more natural for images. The authors utilize GPUs for every basic step. To demonstrate practicality of GPU-extended image search, the authors also present a simple bag-of-visual-words search technique with full-text search engines. The authors explain how to implement one-to-one keypoint matching with text search engine. Proposed methods lead to drastic performance and precision improvement, which is demonstrated on datasets of different sizes.