Original Scale Invariant Feature Transform Research Articles

An evaluation of conventional and deep learning‐based image‐matching methods on diverse datasets

AbstractImage matching plays an important role in photogrammetry, computer vision and remote sensing. Modern deep learning‐based methods have been proposed for image matching; however, whether they will surpass and take the place of the conventional handcrafted methods in the remote sensing field still remains unclear. A comprehensive evaluation on stereo remote sensing images is also lacking. This paper comprehensively evaluates the performance of conventional and deep learning‐based image‐matching methods by dividing the matching process into feature point extraction, description and similarity measure on various datasets, including images captured from close‐range indoor and outdoor scenarios, unmanned aerial vehicles (UAVs) and satellite platforms. Different combinations of the three steps are evaluated. The experimental results reveal that, first, the performance of the different combinations varies between individual datasets, and it is difficult to determine the best combination. Second, by using more comprehensive indicators on all of the datasets, that is, the average rank and absolute rank, the combination of scale‐invariant feature transform (SIFT), ContextDesc and the nearest neighbour distance ratio (NNDR), and also the original SIFT, achieve the best results, and are recommended for use in remote sensing. Third, the deep learning‐based Sub‐SuperPoint extractor obtains a good performance, and is second only to SIFT. The learning based ContextDesc descriptor is as effective as the SIFT descriptor, and the learning based SuperGlue matcher is not as stable as NNDR, but leads to a few top‐performing combinations. Finally, the handcrafted methods are generally faster than the deep learning‐based methods, but the efficiency of the latter is acceptable. We conclude that although a full deep learning‐based method/combination has not yet beaten the conventional methods, there is still much room for improvement with the deep learning‐based methods because large‐scale aerial and satellite training datasets remain to be constructed, and specific methods for remote sensing images remain to be developed. The performance of the different combinations of feature extractor, descriptor and similarity measure varies between individual datasets. The combination of SIFT, ContextDesc and NNDR, and also the original SIFT, achieve the best results when using more comprehensive indicators on all the datasets. For extractor, the learning based Sub‐SuperPoint is second only to SIFT; for descriptor, learning‐based ContextDesc is as effective as the SIFT descriptor; and for matcher, learning‐based SuperGlue is not as stable as NNDR.

Point Cloud Plane Segmentation-Based Robust Image Matching for Camera Pose Estimation

The mainstream image matching method for recovering the motion of the camera is based on local feature matching, which faces the challenges of rotation, illumination, and the presence of dynamic objects. In addition, local feature matching relies on the distance between descriptors, which easily leads to lots of mismatches. In this paper, we propose a new robust image matching method for camera pose estimation, called IM_CPE. It is a novel descriptor matching method combined with 3-D point clouds for image matching. Specifically, we propose to extract feature points based on a pair of matched point cloud planes, which are generated and segmented based on depth images. Then, the feature points are matched based on the distance between their corresponding 3-D points on the point cloud planes and the distance between their descriptors. Moreover, the robustness of the matching can be guaranteed by the centroid distance of the matched point cloud planes. We evaluate the performance of IM_CPE using four well-known key point extraction algorithms, namely Scale-Invariant Feature Transform (SIFT), Speed Up Robust Feature (SURF), Features from Accelerated Segment Test (FAST), and Oriented FAST and Rotated Brief (ORB), with four sequences from the TUM RGBD dataset. According to the experimental results, compared to the original SIFT, SURF, FAST, and ORB algorithms, the NN_mAP performance of the four key point algorithms has been improved by 11.25%, 13.98%, 16.63%, and 10.53% on average, respectively, and the M.Score has also been improved by 25.15%, 23.05%, 22.28%, and 11.05% on average, respectively. The results show that the IM_CPE can be combined with the existing key points extraction algorithms and the IM_CPE can significantly improve the performance of these key points algorithms.

An Optimized Scale-Invariant Feature Transform Using Chamfer Distance in Image Matching

Scale-Invariant Feature Transform is an image matching algorithm used to match objects of two images by extracting the feature points of target objects in each image. Scale-Invariant Feature Transform suffers from long processing time due to embedded calculations which reduces the overall speed of the technique. This research aims to enhance SIFT processing time by imbedding Chamfer Distance Algorithm to find the distance between image descriptors instead of using Euclidian Distance Algorithm used in SIFT. Chamfer Distance Algorithm requires less computational time than Euclidian Distance Algorithm because it selects the shortest path between any two points when the distance is computed. To validate and evaluate the enhanced algorithm, A data set with (412) images including: (100) images with different degrees of rotation, (100) images with different intensity levels, (112) images with different measurement levels and (100) distorted images to different degrees were used; these images were applied according to four different criteria. The simulation results showed that the enhanced SIFT outperforms the ORB and the original Scale-Invariant Feature Transform in term of the processing time, and it reduces the overall processing time of the classical SIFT by (41%).

An R-SIFT Image Matching Intelligent Algorithm Applied to Hardware Trojan Detection

Malicious modification, deletion or addition of some modules in the integrated circuits (ICs) will cause the chip to be attacked, which is called Hardware Trojans (HTs). Reverse engineering (RE) is a classic destructive method and widely used in HTs detection. However, RE is very time-consuming and error prone. In this paper, based on RE, we propose the R-SIFT algorithm to match image and reformulate the Trojan detection problem as change detection problem. For the R-SIFT algorithm, the ratio of exponentially weighted averages (Roewa) operator is introduced into the scale-invariant feature transform (SIFT) algorithm. Experiments show that the proposed method has 7 to 56 times more matching points than the original SIFT can be effectively applied to HTs detection.

Combination of SIFT and Canny Edge Detection for Registration Between SAR and Optical Images

Scale-invariant feature transform (SIFT) has been successfully used for optical image registration, but it cannot produce satisfying results when directly applied to synthetic aperture radar (SAR) images. In the present study, a novel method is proposed for registration between SAR and optical images. First, candidate keypoints are detected using SIFT algorithm. Then, Canny edge detection algorithm is adopted to remove the edge points which are wrong candidate points. Next, SIFT descriptors are generated from these correct keypoints. Last, the fast library for approximate nearest neighbors (FLANNs) algorithm is applied to search matching points in high-dimensional space. Experimental results show that the proposed approach is significantly more accurate and much faster than the original SIFT algorithm.

Secure outsourcing SIFT: Efficient and Privacy-Preserving Image Feature Extraction in the Encrypted Domain

Multimedia data needs huge storage space, and application of multimedia data needs powerful capability of computing. Cloud computing can help owner of multimedia data to deal with it. But, multimedia data on cloud may reveal privacy of data owner, such as sex, hobbies, address, looks, and so on. Data owner can encrypt multimedia data for confidentiality before uploading it to cloud. However, encrypted multimedia data makes its utilization difficult. In this paper, we first discover pre-existing schemes have problems of huge storage space, security and low efficiency due to their inefficient and insecure algorithms. Then, we provide an effective and practical privacy-preserving scale-invariant feature transform (SIFT) scheme for encrypted image. It uses leveled homomorphic encryption based on our new encoding schemes, our new homomorphic comparison, division and derivative encryption. Our new secure SIFT scheme can realize higher computing efficiency, greatly reduce communication costs and interactive times between user and server, and perform correct feature key point detection, accurate feature point description and image matching. We evaluate security and efficiency of our new secure SIFT scheme, and compare our new secure SIFT scheme with other schemes in detail. The result shows that it is closest to the original SIFT algorithm.

Automatic adaptation of SIFT for robust facial recognition in uncontrolled lighting conditions

The scale invariant feature transform (SIFT), which was proposed by David Lowe, is a powerful method that extracts and describes local features called keypoints from images. These keypoints are invariant to scale, translation, and rotation, and partially invariant to image illumination variation. Despite their robustness against these variations, strong lighting variation is a difficult challenge for SIFT‐based facial recognition systems, where significant degradation of performance has been reported. To develop a robust system under these conditions, variation in lighting must be first eliminated. Additionally, SIFT parameter default values that remove unstable keypoints and inadequately matched keypoints are not well‐suited to images with illumination variation. SIFT keypoints can also be incorrectly matched when using the original SIFT matching method. To overcome this issue, the authors propose propose a method for removing the illumination variation in images and correctly setting SIFT's main parameter values (contrast threshold, curvature threshold, and match threshold) to enhance SIFT feature extraction and matching. The proposed method is based on an estimation of comparative image lighting quality, which is evaluated through an automatic estimation of gamma correction value. Through facial recognition experiments, the authors find significant results that clearly illustrate the importance of the proposed robust recognition system.

Illumination invariant feature point matching for high-resolution planetary remote sensing images

Despite its success with regular close-range and remote-sensing images, the scale-invariant feature transform (SIFT) algorithm is essentially not invariant to illumination differences due to the use of gradients for feature description. In planetary remote sensing imagery, which normally lacks sufficient textural information, salient regions are generally triggered by the shadow effects of keypoints, reducing the matching performance of classical SIFT. Based on the observation of dual peaks in a histogram of the dominant orientations of SIFT keypoints, this paper proposes an illumination-invariant SIFT matching method for high-resolution planetary remote sensing images. First, as the peaks in the orientation histogram are generally aligned closely with the sub-solar azimuth angle at the time of image collection, an adaptive suppression Gaussian function is tuned to level the histogram and thereby alleviate the differences in illumination caused by a changing solar angle. Next, the suppression function is incorporated into the original SIFT procedure for obtaining feature descriptors, which are used for initial image matching. Finally, as the distribution of feature descriptors changes after anisotropic suppression, and the ratio check used for matching and outlier removal in classical SIFT may produce inferior results, this paper proposes an improved matching procedure based on cross-checking and template image matching. The experimental results for several high-resolution remote sensing images from both the Moon and Mars, with illumination differences of 20°–180°, reveal that the proposed method retrieves about 40%–60% more matches than the classical SIFT method. The proposed method is of significance for matching or co-registration of planetary remote sensing images for their synergistic use in various applications. It also has the potential to be useful for flyby and rover images by integrating with the affine invariant feature detectors.

Optimization of SIFT algorithm for fast-image feature extraction in line-scanning ophthalmoscope

The Scale Invariant Feature Transform (SIFT) algorithm is utilized broadly in image registration to improve image qualities. However, the algorithm’s complexity reduces its efficiency in biology study and usually requires real-time. In this article, we present an improved SIFT technique in software architecture for matching sequences of images taken from a line-scanning ophthalmoscope (LSO). The method generates the Gaussian Scale-space pyramid in frequency domain to complete the SIFT feature detector more quickly. A novel SIFT descriptor invariable with rotation and illumination is then created to reduce calculation time, implementing the original SIFT method, our improved SIFT method, and the graphic processing unit (GPU) version of our improved SIFT method. The experiments have shown that the improved SIFT is almost 2–3 times faster than the original while maintaining more robust performance, and the GPU implementation of the improved SIFT is 20 times faster than central processing unit (CPU) implementation and achieves acceleration at real-time as expected. Although tested on an LSO system, the improved SIFT method does not rely on the acquisition setup. As a result, this method can be applied to other imaging instruments, e.g., adaptive optics to increase their resolution in agreement.

An improved SIFT algorithm in the application of close-range Stereo image matching

As unmanned aerial vehicle (UAV) remote sensing is applied in small area aerial photogrammetry surveying, disaster monitoring and emergency command, 3D urban construction and other fields, the image processing of UAV has become a hot topic in current research. The precise matching of UAV image is a key problem, which affects the subsequent processing precision directly, such as 3D reconstruction and automatic aerial triangulation, etc. At present, SIFT (Scale Invariant Feature Transform) algorithm proposed by DAVID G. LOWE as the main method is, is widely used in image matching, since its strong stability to image rotation, shift, scaling, and the change of illumination conditions. It has been successfully applied in target recognition, SFM (Structure from Motion), and many other fields. SIFT algorithm needs the colour images to be converted into grayscale images, detects extremum points under different scales and uses neighbourhood pixels to generate descriptor. As we all know that UAV images with rich colour information, the SIFT algorithm improved through combining with the image colour information in this paper, the experiments are conducted from matching efficiency and accuracy compared with the original SIFT algorithm. The results show that the method which proposed in this paper decreases on the efficiency, but is improved on the precision and provides a basis choice for matching method.

Securing SIFT: Privacy-Preserving Outsourcing Computation of Feature Extractions Over Encrypted Image Data.

Advances in cloud computing have greatly motivated data owners to outsource their huge amount of personal multimedia data and/or computationally expensive tasks onto the cloud by leveraging its abundant resources for cost saving and flexibility. Despite the tremendous benefits, the outsourced multimedia data and its originated applications may reveal the data owner's private information, such as the personal identity, locations, or even financial profiles. This observation has recently aroused new research interest on privacy-preserving computations over outsourced multimedia data. In this paper, we propose an effective and practical privacy-preserving computation outsourcing protocol for the prevailing scale-invariant feature transform (SIFT) over massive encrypted image data. We first show that the previous solutions to this problem have either efficiency/security or practicality issues, and none can well preserve the important characteristics of the original SIFT in terms of distinctiveness and robustness. We then present a new scheme design that achieves efficiency and security requirements simultaneously with the preservation of its key characteristics, by randomly splitting the original image data, designing two novel efficient protocols for secure multiplication and comparison, and carefully distributing the feature extraction computations onto two independent cloud servers. We both carefully analyze and extensively evaluate the security and effectiveness of our design. The results show that our solution is practically secure, outperforms the state-of-the-art, and performs comparably with the original SIFT in terms of various characteristics, including rotation invariance, image scale invariance, robust matching across affine distortion, and addition of noise and change in 3D viewpoint and illumination.

Robust Face Recognition System in Video using Hybrid Scale Invariant Feature Transform

Face recognition plays a significant role in the research field of biometric and computer vision. The important goal of an efficient Face Recognition (FR) system is to have negligible misclassification rate. In video-based face recognition system, the illumination and pose variation problems are predominant. Most of the efficient FR systems are developed for controlled or indoor environment, hence they fail to give accurate recognition in outdoor environment of different illumination variation. Other challenges include occlusion and facial expression. The illumination problem is handled by Histogram Equalization in existing methods. The original Scale Invariant Feature Transform (SIFT) also works well only for pose variation and fails to produce satisfactory results under varying illumination. Hence Hybrid Scale Invariant Feature Transform (HSIFT) with Weighting Factor in feature matching is proposed in this paper which uses a fixed facial landmark localization technique and orientation assignment of SIFT to extract illumination and pose invariant features. The extracted features are then matched using Fast Library for Approximation of Nearest Neighbor (FLANN). The proposed method has been implemented in OpenCV to give a recognition rate of 98% and 95.5% in YouTube celebrity and Extended Yale B dataset respectively.

Compressed matching for feature vectors

The problem of compressing a large collection of feature vectors is investigated, so that object identification can be processed on the compressed form of the features. The idea is to perform matching of a query image against an image database, using directly the compressed form of the descriptor vectors, without decompression. Specifically, we concentrate on the Scale Invariant Feature Transform (SIFT), a known object detection method, as well as on Dense SIFT and PHOW features, that contain, for each image, about 300 times as many vectors as the original SIFT. Given two feature vectors, we suggest achieving our goal by compressing them using a lossless encoding by means of a Fibonacci code, for which the pairwise matching can be done directly on the compressed files. In our experiments, this approach improves the processing time and incurs only a small loss in compression efficiency relative to standard compressors requiring a decoding phase.

Fuzzy zoning for feature matching technique in 3D reconstruction of nasal endoscopic images

3D reconstruction from nasal endoscopic images greatly supports an otolaryngologist in examining nasal passages, mucosa, polyps, sinuses, and nasopharyx. In general, structure from motion is a popular technique. It consists of four main steps; (1) camera calibration, (2) feature extraction, (3) feature matching, and (4) 3D reconstruction. Scale Invariant Feature Transform (SIFT) algorithm is normally used for both feature extraction and feature matching. However, SIFT algorithm relatively consumes computational time particularly in the feature matching process because each feature in an image of interest is compared with all features in the subsequent image in order to find the best matched pair. A fuzzy zoning approach is developed for confining feature matching area. Matching between two corresponding features from different images can be efficiently performed. With this approach, it can greatly reduce the matching time. The proposed technique is tested with endoscopic images created from phantoms and compared with the original SIFT technique in terms of the matching time and average errors of the reconstructed models. Finally, original SIFT and the proposed fuzzy-based technique are applied to 3D model reconstruction of real nasal cavity based on images taken from a rigid nasal endoscope. The results showed that the fuzzy-based approach was significantly faster than traditional SIFT technique and provided similar quality of the 3D models. It could be used for creating a nasal cavity taken by a rigid nasal endoscope.

Hexagonal scale invariant feature transform (H-SIFT) for facial feature extraction

Feature transformation and key-point identification is the solution to many local feature descriptors. One among such descriptor is the Scale Invariant Feature Transform (SIFT). A small effort has been made for designing a hexagonal sampled SIFT feature descriptor with its applicability in face recognition tasks. Instead of using SIFT on square image coordinates, the proposed work makes use of hexagonal converted image pixels and processing is applied on hexagonal coordinate system. The reason of using the hexagonal image coordinates is that it gives sharp edge response and highlights low contrast regions on the face. This characteristic allows SIFT descriptor to mark distinctive facial features, which were previously discarded by original SIFT descriptor. Furthermore, Fisher Canonical Correlation Analysis based discriminate procedure is outlined to give a more precise classification results. Experiments performed on renowned datasets revealed better performances in terms of feature extraction in robust conditions. All Rights Reserved © 2015 Universidad Nacional Autonoma de Mexico, Centro de Ciencias Aplicadas y Desarrollo Tecnologico. This is an open access item distributed under the Creative Commons CC License BY-NC-ND 4.0

Tracking feature extraction techniques with improved SIFT for video identification

This paper presents a method for tracking of object movements and detecting of feature to identify video content using improved Scale-Invariant Feature Transform (SIFT). SIFT can robustly identify objects even among clutter and under partial occlusion, because the SIFT feature descriptor is invariant to uniform scaling, orientation, and also partially invariant to affine distortion and illumination changes. Even if the video drops frames or attacked, our method can extract the features. In our method we detect the video features from tracking the object's movement and make a dataset with feature sequences to identify video. In contrast to the existing tracking techniques, our method recognized reliable object coordinate. The developed algorithm will be an essential part of a completely tracking and identification system. To evaluate the performance of the proposed approach, we was experimenting with several genres of video. Compare with the original SIFT algorithm, we reducing up to 5 % in processing time was achieved for matching. Also appoint the position of the object area in tracking method make the proposed method automatic, fast and effective.

SAR Image Registration Using Phase Congruency and Nonlinear Diffusion-Based SIFT

The scale-invariant feature transform (SIFT) algorithm has been widely applied to optical image registration. However, mostly because of multiplicative speckle noise, SIFT has a limited performance when directly applied to synthetic aperture radar (SAR) image. In this letter, a novel SAR image registration method is proposed, which is based on the combination of SIFT, nonlinear diffusion, and phase congruency. In our proposed algorithm, the multiscale representation of a SAR image is generated by nonlinear diffusion, since it better preserves edges in the image as opposed to Gaussian smoothing, which is used in the original SIFT. To reduce the influence of multiplicative speckle noise, the ratio of exponential weighted average operator is used to compute the gradient information in the construction of nonlinear diffusion scale space. Moreover, phase congruency information is utilized to remove the erroneous keypoints within the initial keypoints. Experimental results on multipolarization, multiband, and multitemporal SAR images indicate that our algorithm can improve the match performance compared to the SIFT-based method, which leads to a subpixel accuracy for all the tested image pairs.

Using an improved SIFT algorithm and fuzzy closed-loop control strategy for object recognition in cluttered scenes.

Partial occlusions, large pose variations, and extreme ambient illumination conditions generally cause the performance degradation of object recognition systems. Therefore, this paper presents a novel approach for fast and robust object recognition in cluttered scenes based on an improved scale invariant feature transform (SIFT) algorithm and a fuzzy closed-loop control method. First, a fast SIFT algorithm is proposed by classifying SIFT features into several clusters based on several attributes computed from the sub-orientation histogram (SOH), in the feature matching phase only features that share nearly the same corresponding attributes are compared. Second, a feature matching step is performed following a prioritized order based on the scale factor, which is calculated between the object image and the target object image, guaranteeing robust feature matching. Finally, a fuzzy closed-loop control strategy is applied to increase the accuracy of the object recognition and is essential for autonomous object manipulation process. Compared to the original SIFT algorithm for object recognition, the result of the proposed method shows that the number of SIFT features extracted from an object has a significant increase, and the computing speed of the object recognition processes increases by more than 40%. The experimental results confirmed that the proposed method performs effectively and accurately in cluttered scenes.

Automatic Indonesian's Batik Pattern Recognition Using SIFT Approach

Abstract Batik is a traditional clothwith unique patterns applied to fabric using a wax-resist dyeing technique. Aside from preserving this rich cultural heritage, the automated recognition of Batik patterns would enable many interesting applications. This paper introduces an approach to batik pattern recognition using the Scale Invariant Feature Transform (SIFT) as a feature extraction method. The challenging issues that arise are due to the highly symmetrical and repetitive properties of batik patterns. The Hough transform, as an evidence-based method of object detection, is applied to handle mismatched keypoints resulting from symmetrical and repetitive patterns of batik. On a collection of 120 batik images generated from 20 basic batik patterns, the proposed method showsan improvement over the original SIFT matching method with an equal error rate of 8.47%.

Classification of 3D Multicellular Organization in Phase Microscopy for High Throughput Screening of Therapeutic Targets.

The current trend in high throughput screening is the utilization of more complex model systems that mimic both structural and functional properties of cellular processes in vivo. In this context, 3D cell culture models have emerged as effective systems to study tumor initiation and cancer behavior, where colony organization represents distinct phenotypic signatures that enable differentiation of cancer cells in culture using phase imaging and in the absence of clinical markers. If the colony organization can be classified into different phenotypes, it will enable rapid drug screening using phase microscopy. In this paper, we propose a novel method based on locality-constrained dictionary learning for the discrimination of aberrant colony organization in phase images, which encodes original SIFT (Scale-Invariant Feature Transform) features into high dimensional sparse codes with locality-preserving landmark points on the nonlinear manifold, and summarizes the sparse features at various locations and scales through spatial pyramid matching for robust representation. Experimental results demonstrate the significant improvement of performance, compared to the state-of-art in the field.