Keypoint Matching Research Articles

Attitude Mounting Misalignment Estimation Method for the Calibration of UAV LiDAR System by using a TIN-based Corresponding Model

In traditional attitude mounting misalignment estimation methods for the calibration of unmanned autonomous vehicle (UAV) based light detection and ranging (LiDAR) system, signalized targets and iterative corresponding models are required, which makes it highly cost and computationally time-consuming. This paper presents an attitude mounting misalignment estimation (AMME) method for the calibration of UAV LiDAR system. The proposed method is divided into the coarse registration of LiDAR strips and the estimation of the attitude mounting misalignment. Firstly, 3D keypoints are extracted in the point clouds using the scale-invariant feature transform (SIFT) algorithm. Afterwards, the point feature transform (PFH) descriptor is used for 3D keypoint matching. Then, the coarse registration is executed. In the second part of the contribution, the systematic errors in the attitude mounting misalignment are estimated by incorporating the proposed triangular irregular network (TIN) corresponding model into the calibration modelling. Using the TIN-based corresponding model saves time and cost for AMME method. Furthermore, it provides two important effects: practical and computational, as no designed calibration boards, segmentation and iterative matching are needed. The performance of the proposed method is demonstrated under an UAV LiDAR data onboarded with lightweight navigation sensors. The experimental results show the efficacy of the method in comparison with a state-of-the-art method.

Attitude Mounting Misalignment Estimation Method for the Calibration of UAV LiDAR System by using a TIN-based Corresponding Model

In traditional attitude mounting misalignment estimation methods for the calibration of unmanned autonomous vehicle (UAV) based light detection and ranging (LiDAR) system, signalized targets and iterative corresponding models are required, which makes it highly cost and computationally time-consuming. This paper presents an attitude mounting misalignment estimation (AMME) method for the calibration of UAV LiDAR system. The proposed method is divided into the coarse registration of LiDAR strips and the estimation of the attitude mounting misalignment. Firstly, 3D keypoints are extracted in the point clouds using the scale-invariant feature transform (SIFT) algorithm. Afterwards, the point feature transform (PFH) descriptor is used for 3D keypoint matching. Then, the coarse registration is executed. In the second part of the contribution, the systematic errors in the attitude mounting misalignment are estimated by incorporating the proposed triangular irregular network (TIN) corresponding model into the calibration modelling. Using the TIN-based corresponding model saves time and cost for AMME method. Furthermore, it provides two important effects: practical and computational, as no designed calibration boards, segmentation and iterative matching are needed. The performance of the proposed method is demonstrated under an UAV LiDAR data onboarded with lightweight navigation sensors. The experimental results show the efficacy of the method in comparison with a state-of-the-art method.

Composite attacks‐based copy‐move image forgery detection using AKAZE and FAST with automatic contrast thresholding

Copy-move image forgery is one of the most popular image tampering technique which can be performed for vicious purposes. In this forgery technique, selected region is copied and pasted at different locations on the same image to produce a manipulated image. Such forgery is denigratory as it can alter the image content by hiding or appending visual information. In this study, the authors propose a novel keypoint-based technique to detect forged images sustaining composite attacks consisting of various combinations of geometrical and post-processing operations. In the authors' method, AKAZE and FAST techniques are used to extract keypoints from the image. Non-maximal value suppression with automatic contrast thresholding is performed during FAST keypoint extraction. SIFT and DAISY descriptors are computed corresponding to extracted keypoints. PCA is applied over SIFT and DAISY descriptors to discard lower components which are sensitive to distortions occurred in images. They apply a correlation-based nearest neighbour search technique to detect similarity among keypoint descriptors. HDBSCAN algorithm is applied to obtain matched keypoint clusters. Further, RANSAC algorithm is utilised for removal of keypoint outliers. In comparison to state-of-the-art techniques, their approach achieve high F-measure (%) and low FPR (%) for image-level as well as pixel-level copy-move forgery detection.

Enhancing collaborative road scene reconstruction with unsupervised domain alignment

Scene reconstruction and visual localization in dynamic environments such as street scenes are a challenge due to the lack of distinctive, stable keypoints. While learned convolutional features have proven to be robust to changes in viewing conditions, handcrafted features still have advantages in distinctiveness and accuracy when applied to structure from motion. For collaborative reconstruction of road sections by a car fleet, we propose to use multimodal domain adaptation as a preprocessing step to align images in their appearance and enhance keypoint matching across viewing conditions while preserving the advantages of handcrafted features. Training a generative adversarial network for translations between different illumination and weather conditions, we evaluate qualitative and quantitative aspects of domain adaptation and its impact on feature correspondences. Combined with a multi-feature discriminator, the model is optimized for synthesis of images which do not only improve feature matching but also exhibit a high visual quality. Experiments with a challenging multi-domain dataset recorded in various road scenes on multiple test drives show that our approach outperforms other traditional and learning-based methods by improving completeness or accuracy of structure from motion with multimodal two-domain image collections in eight out of ten test scenes.

Electrical fault detection in three-phase induction motor using deep network-based features of thermograms

In this paper, an automatic method is proposed for detecting the operating faults in three-phase induction motors based on thermal images. If these faults are not detected or fixed on time, they can lead to permanent motor failure. This is why non-invasive and non-destructive experiments are significantly considered. In this paper, first, the region of interest is detected in the thermograms using SIFT-based key-points matching. Then, these images are transformed into representative feature vectors based on a pre-trained convolutional neural network. Then, the training vector samples are clustered into cold and hot clusters by K-means. For each cluster, an SVM-based classifier is trained. The test feature vector samples are clustered and mapped into classes using the corresponding trained SVM-based classifiers. Evaluating the proposed method on the datasets including real thermal images, shows that this algorithm can detect 100% of the faults of the induction motor.

A Method to Detect and Track Moving Airplanes from a Satellite Video

In recent years, satellites capable of capturing videos have been developed and launched to provide high definition satellite videos that enable applications far beyond the capabilities of remotely sensed imagery. Moving object detection and moving object tracking are among the most essential and challenging tasks, but existing studies have mainly focused on vehicles. To accurately detect and then track more complex moving objects, specifically airplanes, we need to address the challenges posed by the new data. First, slow-moving airplanes may cause foreground aperture problem during detection. Second, various disturbances, especially parallax motion, may cause false detection. Third, airplanes may perform complex motions, which requires a rotation-invariant and scale-invariant tracking algorithm. To tackle these difficulties, we first develop an Improved Gaussian-based Background Subtractor (IPGBBS) algorithm for moving airplane detection. This algorithm adopts a novel strategy for background and foreground adaptation, which can effectively deal with the foreground aperture problem. Then, the detected moving airplanes are tracked by a Primary Scale Invariant Feature Transform (P-SIFT) keypoint matching algorithm. The P-SIFT keypoint of an airplane exhibits high distinctiveness and repeatability. More importantly, it provides a highly rotation-invariant and scale-invariant feature vector that can be used in the matching process to determine the new locations of the airplane in the frame sequence. The method was tested on a satellite video with eight moving airplanes. Compared with state-of-the-art algorithms, our IPGBBS algorithm achieved the best detection accuracy with the highest F1 score of 0.94 and also demonstrated its superiority on parallax motion suppression. The P-SIFT keypoint matching algorithm could successfully track seven out of the eight airplanes. Based on the tracking results, movement trajectories of the airplanes and their dynamic properties were also estimated.

Human Facial Feature Matching based on Motion-Smoothness Constraint

Face recognition is a practical and important problem. In our life, face recognition problems exist everywhere. How to identify the face in the image accurately and quickly is an important and urgent topic. In this paper, an efficient and accurate face recognition method based on keypoints feature matching is proposed to extract the keypoints features of the face image, and then the nearest neighbor method is used to obtain a large number of keypoints rough matching. Then the correct matching of the keypoints is determined by the motion smoothness constraint of the original keypoints matching. Finally, the matching of the face is determined according to the number of keypoints matching between the two input face images. Some experiments show that this method is suitable for face recognition with expression change and pose change.

Orientation-independent Feature Matching (OIFM) for Multimodal Retinal Image Registration

The analysis of fundus images in ophthalmology can be greatly facilitated by data integration of multimodal retinal images, which can be achieved via image registration based on the matching of keypoints represented by certain descriptors. However, the matching results offered by conventional feature descriptors may be substantially compromised due to their inconsistent estimates of the main orientation for keypoints in multimodal images. In this paper, we propose an orientation-independent feature matching (OIFM) method for better matching of feature points in multimodal retinal images. The keypoints detected in the images are firstly represented with a new circular neighborhood-based feature descriptor, allowing for the rotation of the neighborhood around the keypoints can be achieved by circularly shifting the elements in the descriptor. Then, the proposed feature descriptor is applied for the matching of keypoints, whose distance is measured after the compensation of the orientation discrepancy. The proposed OIFM method has two distinct characteristics. First, the feature descriptor no longer relies on the calculation of main orientations, and instead it is formulated by stacking the feature vectors of the keypoint neighborhood in a circular order with a conveniently assigned reference. Second, orientation independence is achieved during the matching stage where the feature points are aligned in a global manner, leading the OIFM method more robust to rotation and content variations among multimodal retinal images. Experimental results on a total of 160 pairs of multimodal retinal images show that the proposed OIFM method outperforms the conventional algorithms in terms of registration accuracy and robustness.

Weakly supervised learning for image keypoint matching using graph convolutional networks

Matching between two sets of features from a pair of images is a fundamental and critical step in most computer vision tasks. Existing attempts typically establish a set of putative correspondences with the nearest-neighbor rule in feature spaces and then try to find a subset of reliable matches. However, when there are large camera angles, repetitive structures, and illumination changes existing in the two images of the same scene, recently proposed feature matching approaches do not work well to find good correspondences, especially with a higher proportion of false-positive matches in the putative set. To address these problems, we propose a novel weakly supervised Graph Convolutional Siamese Network Matcher, called GCSNMatcher, to learn the correct correspondences for image feature matching. In particular, GCSNMatcher can directly work on unstructured keypoint sets and further exploit geometric information among sparse interest points by constructing dynamic neighborhood graph structures to enhance the ability of the feature representation of each keypoint. With channel-wised symmetric aggregation operations in our graph convolutional neural networks, the performance of our matcher does not vary under different permutations of unordered keypoint sets. Empirical studies on Yahoo’s YFCC100M benchmark dataset demonstrate that our matcher can give a more robust performance for image matching tasks than those state-of-the-art methods, even when it is trained on small datasets.

Long-term real time object tracking based on multi-scale local correlation filtering and global re-detection

This paper investigates long-term visual object tracking which is a complex problem in computer vision community and big data analysis, due to the variation of the target and the surrounding environment. A novel tracking algorithm based on local correlation filtering and global keypoint matching is proposed to solve problems occurred during long-term tracking such as occlusion, target-losing, etc. The algorithm consists of two major components: (1) local object tracking module, and (2) global losing re-detection module. The local tracking module optimizes the conventional correlation filtering algorithm. Firstly, the Color Name feature is applied to increase the color sensitivity. Secondly, a scale traversal is employed to accommodate target scale changes. In the global losing re-detection module, the target losing judgment and global re-detection is realized by keypoint feature models of foreground and background. The proposed tracker achieves the 1st place in the VTB50 test set with 81.3% precision and 61.3% success rate, which outperforms other existing state-of-the-art trackers by over 10%. And it achieves the 2nd place in our Chasing-Car test set with a higher real-time performance 43.2 fps. The experimental results show that the proposed tracker has higher accuracy and robustness when dealing with situations like object deformation, occlusion and target-losing, etc.

A New Object Tracking Framework for Interest Point Based Feature Extraction Algorithms

This paper presents a novel object tracking framework for interest point based feature extracting algorithms. The proposed framework uses the feature extracting algorithm without making any changes and it relies on outlier detection, object modelling, and object tracking. At first, the keypoints are extracted by using a feature extraction algorithm. Then, incorrect keypoint matches are detected by the DBScan algorithm. The second step of our tracking framework is object modelling. The object model is defined as a bounding box. The box model has six points and each of these points has its own Gaussian model. Finally, the Gaussian model is performed for object tracking. In object tracking, the old five values are retained to detect incorrect position information. Thus, while the object movements are softened, the instant deviations are eliminated also. Our interest point based object tracking framework (IPBOT) works with any interest point based feature extracting algorithm. Thus, a new algorithm can be added to the object tracking framework with a short integration process. The experiment results show that the proposed tracker significantly improves the success rate of the object tracking.

Abrupt-motion-aware lightweight visual tracking for unmanned aerial vehicles

Visual tracking for unmanned aerial vehicles (UAVs) is a hot research topic for the wide applications of UAVs. As UAVs are high-altitude and high-freedom platforms, small targets in UAV tracking sequences are often under the attribute of large-scale location change due to the abrupt motion of the platform. Currently, many visual tracking methods based on the local search hypothesis have been widely researched on low-speed moving platforms. However, these methods cannot be directly used on the UAV platform, because targets will appear in any position of the new frame. To address this problem, we propose an abrupt-motion-aware visual tracking method in this paper. Because of the high power consumption of deep learning models, the proposed method is a lightweight tracker for small UAVs without the deep learning framework. Our method consists of three major components: abrupt motion estimation, object tracking and model updating. Abrupt motion often leads to abnormal changes in the response map of trackers. Thus, by analyzing the changes of tracking response maps, the abrupt motion can be detected efficiently. When abrupt motion happens, keypoint matching will be adaptively implemented to estimate the ego-motion and skipped otherwise. Then, the target location is predicted by the correlation filter tracker in a local search region. Moreover, according to the confidence analysis, an adaptive model update strategy is designed to alleviate the model noise caused by the short-term occlusion. Experimental results confirm the robustness and the accuracy of our method on challenging sequences and show the comparative performance of the proposed method against several state-of-the-art lightweight methods.

Mapping erosion and deposition in an agricultural landscape: Optimization of UAV image acquisition schemes for SfM-MVS

Structure-from-motion (SfM) and multi-view stereo (MVS) algorithms coupled with the use of unmanned aerial vehicles (UAVs) have become a popular tool in the geosciences for modelling complex landscapes on-demand allowing for high-resolution topographic change-detection studies to be conducted at minimal cost. To identify the effects of UAV image orientation on the accuracy of SfM-MVS 3D surface models, we tested four different UAV image acquisition schemes that incorporated both nadir and oblique imagery of an agricultural field. The coupling of nadir and oblique imaging angles led to the highest surface model accuracy in the absence of ground control points (GCPs; vertical RMSE: 0.047 m, horizontal RMSE: 0.019 m), while with a normative distribution of GCPs the nadir-only image sets had similar accuracy metrics (vertical RMSE 0.028 m, horizontal RMSE 0.017 m) to surface models generated with nadir and oblique imaging angles (vertical RMSE 0.028 m, horizontal RMSE 0.013 m). Homologous keypoint matching between nadir and oblique imagery was poor when the survey conditions were bright and the surface texture of the field was homogeneous, leading to broad-scale vertical noise in the generated surface models. Results indicate that a nadir-only image set accompanied with a dense deployment of GCPs is the most ideal for SfM-MVS agricultural 3D surface reconstructions. The diachronic analysis of surface models generated from nadir-only image sets were able to detect surface-change >0.040 m in depth (i.e., rill and gully erosion, depositional zones) and were comparable to results obtained from a terrestrial laser scanner. Stable GCPs should be used where possible to ensure precise co-registration between subsequent UAV surveys.

A Flexible and Efficient Real-Time ORB-Based Full-HD Image Feature Extraction Accelerator

This article proposes a hardware accelerator for image feature extraction based on the oriented features from accelerated segment test (FAST) and rotated binary robust independent elementary features (ORB) algorithm. The architecture adopts a hybrid workflow to process three scales in parallel as an octave, and multiple octaves are processed in serial by time-sharing the DRAM. In order to support any image resolution, a block-wise dataflow is used on the serial-parallel architecture. The overlapped data between two blocks is reused. As a result, the on-chip memory is limited to 1.47 Mb, and the DRAM bandwidth is compressed by 33%. Besides, 3×3 nonmaximum suppression with heap sorting is applied to balance keypoint distribution in 2-D. It improves the valid keypoint match ratio by 7.09%. In addition, parallel processing in keypoint detection increases 2× throughout. Moreover, approximate computing and superscalar processing reduce the timing cost in orientation estimation and descriptor generation by 92.78% and 67% respectively. Compared to the non-optimized baseline architecture, the proposed architecture saves the total timing cost by 82.4%. The accelerator is implemented in a Xilinx MPSoC, and it achieves 108 fps on full-HD images at 200 MHz under 873 mW.

Automated Visual Inspection of Multiple Missing Fastening Bolts on the Sides of High-Speed Train Body in Railway Maintenance

The rapid development of China Railway High-speed (CRH) has provided safe and comfortable transportation for many people. The fastening bolts (FB) on the sides of CRH fall off occasionally, which might cause serious traffic accidents. However, the automatic inspection of multiple missing FBs on the sides of CRH is seldom conducted. In this study, an automated visual inspection system based on space-scale normalization is proposed to inspect the status of FBs on the sides of CRH. To locate the missing FBs in sequence images, the exact alignment of the target image and its reference image must be ensured. This problem can be solved by normalizing the target image according to its reference image. First, the FBs are located in a specific CRH region, which is called the region of interest (ROI). The target image is cropped according to the size of the ROI. Second, the keypoints are extracted and matched by using the gradient-enhanced scale-invariant feature transform (GESIFT). Furthermore, the horizontal shift between the target image and its reference image is obtained during keypoint matching. Third, the target image and its neighboring image are stitched according to the horizontal shift. Then, the target image is resegmented according to its reference image to complete the normalization of the target image. Finally, the normalized target image can be exactly aligned with its reference image. We propose an image subtraction method to locate the potential missing FBs. The FBs that may be lost are finally determined according to prior knowledge, and the state of the FBs is determined by using high-level image understanding knowledge. Experimental results show that our presented method performs excellently in monitoring the missing FBs on the sides of CRH.

Local Feature Descriptor Indexing for Image Matching and Object Detection in Real-Time Applications

Local feature extraction and description algorithms can be used to address a large variety of computer vision problems, including pattern recognition, frame stitching and 3D reconstruction. One of the most computationally expensive and time-consuming stages of any method based on local features is keypoint matching. This paper discusses possible ways to optimize matching for frame alignment and object detection applications using Vantage-Point tree indexing to optimize pairwise keypoint matching, and image keypoint graph indexing to optimize pose estimation.

Toward Visualization and Searching: A Dual-Purpose Video Coding Framework

To offer more powerful video-enabled applications, it is increasingly more critical not only to visualize the decoded video but also to provide efficient searching capabilities for similar content. Video surveillance and personal communication are critical application examples asking for these dual visualization and searching functionalities. However, the current video coding solutions are strongly biased towards visualization needs. In this context, this paper introduces a novel dual-purpose video coding framework targeting both visualization and searching needs by adopting a hybrid coding approach where the usual pixel-based coding approach is combined with an innovative feature-based coding approach. In a particular implementation of this dual-purpose video coding framework, some frames are coded using a set of keypoint matches, which not only allow decoding for visualization, but also provide the decoder valuable feature-related information, extracted at the encoder from the original frames, instrumental for efficient searching. Experimental results for video surveillance and personal communications scenarios show competitive performance regarding the state-of-the-art HEVC standard both in terms of visualization and searching performances.

Face quality analysis of single-image super-resolution based on SIFT

Single-image super-resolution (SISR) aims at improving image quality, and there so far exist many SISR algorithms to hallucinate super-resolution (super-res) image from simulated low-res image. In order to evaluate SISR algorithms, objective image quality assessment (IQA), e.g., full reference IQA and no-reference IQA, and subjective quality are usually estimated. However, the objective IQA usually does not well match with the subjective quality. This paper therefore introduces a new measurement based on SIFT key-points. Both descriptors and locations of SIFT key-points are used to detect the matched SIFT key-points between super-res image and its high-res label image. The more the matched SIFT key-points are, the closer super-res image should be to its high-res label image, that is the SISR algorithm is able to recover more SIFT key-points. Both simulated low-res faces and real low-res face are employed to validate the evaluation strategy. The normalization of the number of SIFT key-points is proposed and mean opinion score from 30 raters are collected to evaluate SISR algorithms. The experimental results show that the objective IQA based on SIFT key-points are able to effectively evaluate SISR algorithms, and can well match with the subjective IQA.

Vision technique for deflection measurements based on laser positioning

This paper proposes a vision-based target-free deflection measurement method of bridge. This method eliminates the trouble of installing artificial targets on the structure. The proposed method utilizes multi-point laser projection as a reference point, which can eliminate the error caused by vibration at the camera position, and also avoids the difficulty of finding the background building as a reference point. The proposed method controls the measurement accuracy by adjusting the size of the captured video area. Through the method of camera scanning, large-scale multi-point deflection measurement is carried out under the premise of certain precision. The video laser spot reference point (LSRP) before loading is selected by the image segmentation algorithm. The Oriented features from accelerated cegment Test (FAST) and rotated binary robust independent elementary features (BRIEF) (ORB) feature detection and extraction algorithm is used to detect and extract key features in the region of interest (ROI) near the LSRP. The extracted key feature points calculate the displacement of each frame of the entire video by using the Lucas-Kanade Tracking (KLT) algorithm, which ensures the maximum number of matches of key points between consecutive images. The mismatched outliers are discarded by using the Random Sampling Consistency (RANSAC) algorithm. The accuracy of the proposed method was verified by an indoor and outdoor beam experiment. In the experiment, the results obtained by the proposed method are compared with the vertical displacement values of the corresponding positions acquired by the displacement meter obtained by the high-precision DH3816N dynamic strain test system to verify the validity and accuracy of the proposed method.

Quantitative evaluation of multiple-point simulations using image segmentation and texture descriptors

Continuous growth of multiple-point simulation algorithms for modeling environmental variables necessitates a straightforward, reliable, robust, and distinctive method for evaluating the quality of output images. A good simulation method should produce realizations consistent with the training image (TI). Moreover, it should be capable of producing diverse realizations to effectively model the variability of real fields. In this paper, the pattern innovation capability is evaluated by estimating the coherence map using keypoint detection and matching, without assuming any access to the simulation process. Local binary patterns, as distinctive and effective texture descriptors, are also employed to evaluate the consistency of realizations with the TI. Our proposed method provides absolute measures in the interval [0,1], allowing MPS algorithms to be evaluated on their own. Experiments show that the produced scores are consistent with human perception and robust for different realizations obtained using the same method, allowing for a reliable judgment using a few realizations. While a human observer is highly sensitive to discontinuities and insensitive to verbatim copies, the proposed method considers both factors simultaneously.