Interest Point Matching Research Articles

UW Deep SLAM-CNN Assisted Underwater SLAM

Abstract Underwater simultaneous localization and mapping (SLAM) poses significant challenges for modern visual SLAM systems. The integration of deep learning networks within computer vision offers promising potential for addressing these difficulties. Our research draws inspiration from deep learning approaches applied to interest point detection and matching, single image depth prediction and underwater image enhancement. In response, we propose 3D-Net, a deep learning-assisted network designed to tackle these three tasks simultaneously. The network consists of three branches, each serving a distinct purpose: interest point detection, descriptor generation, and depth prediction. The interest point detector and descriptor generator can effectively serve as a front end for a classical SLAM system. The predicted depth information is akin to a virtual depth camera, opening up possibilities for various applications. We provide quantitative and qualitative evaluations to illustrate some of these potential uses. The network was trained in in several steps, using in-air datasets and followed by generated underwater datasets. Further, the network is integrated into feature-based SALM systems ORBSLAM2 and ORBSSLAM3, providing a comprehensive assessment of its effectiveness for underwater navigation.

High resolution and real-time measurement of 2D fatigue crack propagation using an advanced digital image correlation

This paper developed a real-time and non-destructive 2D fatigue crack propagation tracing method based on an optimized digital image correlation (DIC) approach by introducing an image in-situ processing algorithm and a fatigue crack propagation monitoring algorithm. The in-situ processing algorithm coupled with optimizing the speeded up robust features (SURF) was applied to correct the shifted images during fatigue crack propagation tests, while the fatigue crack propagation monitoring algorithm was developed to detect the true crack tip and measure crack length. By optimizing SURF algorithm, the construction time of feature descriptor for interest point matching was reduced to 0.0066 s, 55 times shorter than the case using traditional SURF. In conjunction with the fatigue crack identification, this enables the real-time measurement of fatigue crack length with the resolution as high as 5.2 μm for a measurement region of 15.85 × 10.57 mm2. Compared to current DIC, the proposed algorithm either enables characterizing the crack tip and crack propagation path, or can be programmed as user-defined subroutine and integrated into commercial DIC software for example MatchID. As a result, the experimental measurements obtained by the proposed approach on the fatigue crack propagation tests of an Aluminium alloy exhibited a good agreement with simulation results based on the extended finite element method (XFEM), indicated its promising industrial prospect.

A review of multimodal image matching: Methods and applications

Multimodal image matching, which refers to identifying and then corresponding the same or similar structure/content from two or more images that are of significant modalities or nonlinear appearance difference, is a fundamental and critical problem in a wide range of applications, including medical, remote sensing and computer vision. An increasing number and diversity of methods have been proposed over the past decades, particularly in this deep learning era, due to the challenges in eliminating modality variance and geometrical deformation that intrinsically exist in multimodal image matching. However, a comprehensive review and analysis of traditional and recent trainable methods and their applications in different research fields are lacking. To this end and in this survey, we first introduce two general frameworks, saying area- and feature-based, in terms of their core components, taxonomy, and procedure details. Second, we provide a comprehensive review of multimodal image matching methods from handcrafted to deep methods for each research field according to their imaging nature, including medical, remote sensing and computer vision. Extensive experimental comparisons of interest point detection, description and matching, and image registration are performed on various datasets containing common types of multimodal image pairs that we collected and annotated. Finally, we briefly introduce and analyze several typical applications to reveal the significance of multimodal image matching and provide insightful discussions and conclusions to these multimodal image matching approaches, and simultaneously deliver their future trends for researchers and engineers in related research areas to achieve further breakthroughs.

Tourism recommendation system based on semantic clustering and sentiment analysis

Numerous number of tourism attractions along with a huge amount of information about them on web and social platforms have made the decision-making process for selecting and visiting them complicated. In this regard, the tourism recommendation systems have become interesting for tourists, but challenging for designers because they should be able to provide personalized services. This paper introduces a tourism recommendation system that extracts users' preferences in order to provide personalized recommendations. To this end, users' reviews on tourism social networks are used as a rich source of information to extract their preferences. Then, the comments are preprocessed, semantically clustered, and sentimentally analyzed to detect a tourist's preferences. Similarly, all users aggregated reviews about an attraction are utilized to extract the features of these points of interest. Finally, the proposed recommendation system, semantically compares the preferences of a user with the features of attractions to suggest the most matching points of interest to the user. In addition, the system utilizes the vital contextual information of time, location, and weather to filter unsuitable items and increase the quality of suggestions regarding the current situation. The proposed recommendation system is developed by Python and evaluated on a dataset gathered from TripAdvisor platform. The evaluation results show that the proposed system improves the f-measure criterion in comparison with the previous systems.

Image mosaicking using improved auto-sorting algorithm and local difference-based harris features

Image mosaicking is an image processing technique which is useful for tiling images. Image Mosaicing stitches many correlated images to get a picture of a greater field of view. General-purpose cameras, which have a low field of view, can not create images with a higher field of view while mosaicking can help us achieve it. One important step in an image mosaicking framework is the auto-sorting algorithm, which is to be performed to minimize registration errors in the mosaic image. Another step in mosaicking is the detection of interest points for matching of the source images obtained after auto-sorting. However, in the presence of noisy and pseudo-periodic structures in the source images, the existing auto-sorting methods generally produce distortions in the final mosaic image. Secondly, most of the popular interest point detection algorithms do not specifically consider computational issues. So, this work mainly addresses the above-mentioned problems which are generally encountered during image mosaicking. The problem of image auto-sorting can be partially solved by adopting a phase correlation strategy. In our method, the sorting procedure is further improved by deploying the structural similarity index (SSIM) measure instead of using the phase correlation. The issue of high time complexity of conventional corner detectors is reduced by using our proposed local difference operation in place of standard Sobel edge detector. Experimental results show the efficacy of the proposed method.

Word Spotting Based on Bispace Similarity for Visual Information Retrieval in Handwritten Document Images

Retrieving information from a huge collection of ancient handwritten documents is important for indexing, interpreting, browsing, and searching documents in various domains. Word spotting approaches are widely used in this context but have several limitations related to the complex properties of handwriting. These can appear at several steps: interest point detection, description, and matching. This article proposes a new word spotting approach for the word retrieval in handwritten document, which mainly leverages the properties of image gradients for visual features detection and description. The proposed approach is based on the combination of spatial relationships with textural information to design a more accurate matching. The experimental results of the proposed approach demonstrate a higher performance over the Jeremy Bentham dataset, evaluated following the recent benchmarks of ICDAR 2015 Competition on Keyword Spotting for Handwritten Documents.

Multimedia based fast face recognition algorithm of speed up robust features

For the problem that the HAAR descriptors in the speed up robust feature (SURF) algorithm cannot make full use of the information around the feature points, the K-Mean clustering technology is used in this paper to improve the SURF, thus proposing a new face recognition algorithm. Firstly, the problem that the main direction is too dependent on the direction of the local area is avoided by expanding the scope of the main direction; then the information around the subblock that is masked by the 3 × 3 window template is made full use of to construct a descriptor with stronger recognition ability; finally, the problems of excessive time consumed and incorrect matching of interest points are solved by introducing the K-Mean clustering idea. The results of the experiment on FERET and Yale face database show that the proposed algorithm has higher recognition rate and efficiency than other face recognition techniques.

GA-ORB: A New Efficient Feature Extraction Algorithm for Multispectral Images Based on Geometric Algebra

Feature extraction, including feature detection and description, plays an important role in many computer vision applications. The feature extraction for multispectral images is especially challenging because the information is contained in both spectral and spatial spaces. Previous works, such as scale invariant feature transform (SIFT) algorithm based on geometric algebra (GA) theory (GA-SIFT) and speeded up robust features (SURF) algorithm based on GA theory (GA-SURF), have tried to extract features for multispectral images based on GA to simultaneously capture relations among multiple channels. However, those methods are hard to be applied in real-time tasks. Fortunately, oriented fast and rotated brief (ORB) is a feature extraction method faster than SIFT and SURF. But, the existing ORB algorithms are not capable of detecting features for multispectral images directly. In this paper, we propose a novel feature extraction method, geometric algebra based oriented fast and rotated brief (GA-ORB), for multispectral images based on the theory GA. First, the scale information in both spectral and spatial spaces of multispectral images is obtained in GA, where the inherent spectral structures can be retained successfully. Then, referring to the ORB, the images are computed in different scales and the interest points are detected and described in the GA space. The experimental results show that the GA-ORB outperforms some previous algorithms with respect to distinctiveness and robustness in extracting and matching interest points, and it can be computed much faster.

Enhancement of sparse 3D reconstruction using a modified match propagation based on particle swarm optimization

Sparse 3D reconstruction, based on interest points detection and matching, does not allow to obtain a suitable 3D surface reconstruction because of its incapacity to recover a cloud of well distributed 3D points on the surface of objects/scenes. In this work, we present a new approach to retrieve a 3D point cloud that leads to a 3D surface model of quality and in a suitable time. First of all, our method uses the structure from motion approach to retrieve a set of 3D points (which correspond to matched interest points). After that, we proposed an algorithm, based on the match propagation and the use of particle swarm optimization (PSO), which significantly increases the number of matches and to have a regular distribution of these matches. It takes as input the obtained matches, their corresponding 3D points and the camera parameters. Afterwards, at each time, a match of best ZNCC value is selected and a set of these neighboring points is defined. The point corresponding to a neighboring point and its 3D coordinates are recovered by the minimization of a nonlinear cost function by the use of PSO algorithm respecting the constraint of photo-consistency. Experimental results show the feasibility and efficiency of the proposed approach.

On an algorithm for human action recognition

Human action recognition which needs video processing in real time, requires large memory size and execution time. This work proposes a local maxima of difference image (LMDI) based interest point detection technique, random projection tree with overlapping split and modified voting score for human action recognition. In LMDI based interest point detection method, difference images are obtained using consecutive frame differencing technique and next, 3D peak detection is applied on the bunch of calculated difference images. Histogram of oriented gradients and histogram of optical flow as local features are extracted by defining a block of size 16 × 16 around each of the interest point. These local features are then indexed by random projection trees. Overlapping split is used during tree structuring to reduce failure probability. Hough voting technique is applied on testing video to compute highest similarity matching score with individual training classes. In addition to Hough voting score, the number of matched interest points of a single query video with each training class, is considered for recognition. The proposed method is evaluated on segmented UT-interaction dataset, J-HMDB dataset and UCF101 dataset. The experimental results indicate that the proposed technique provides better performance compared to earlier reported techniques.

A LiDAR data-based camera self-calibration method

To find the intrinsic parameters of a camera, a LiDAR data-based camera self-calibration method is presented here. Parameters have been estimated using particle swarm optimization (PSO), enhancing the optimal solution of a multivariate cost function. The main procedure of camera intrinsic parameter estimation has three parts, which include extraction and fine matching of interest points in the images, establishment of cost function, based on Kruppa equations and optimization of PSO using LiDAR data as the initialization input. To improve the precision of matching pairs, a new method of maximal information coefficient (MIC) and maximum asymmetry score (MAS) was used to remove false matching pairs based on the RANSAC algorithm. Highly precise matching pairs were used to calculate the fundamental matrix so that the new cost function (deduced from Kruppa equations in terms of the fundamental matrix) was more accurate. The cost function involving four intrinsic parameters was minimized by PSO for the optimal solution. To overcome the issue of optimization pushed to a local optimum, LiDAR data was used to determine the scope of initialization, based on the solution to the P4P problem for camera focal length. To verify the accuracy and robustness of the proposed method, simulations and experiments were implemented and compared with two typical methods. Simulation results indicated that the intrinsic parameters estimated by the proposed method had absolute errors less than 1.0 pixel and relative errors smaller than 0.01%. Based on ground truth obtained from a meter ruler, the distance inversion accuracy in the experiments was smaller than 1.0 cm. Experimental and simulated results demonstrated that the proposed method was highly accurate and robust.

Efficient invariant interest point detector using Bilateral-Harris corner detector for object recognition application

Interest point detection plays a significant role in computer vision applications. The most commonly used interest point detector algorithm is scale invariant feature transform (SIFT). The use of Gaussian filter in the SIFT algorithm fails to match interest points on the edge and it also causes blur annoyance in the rescaling process. To overcome this failure Bilateral-Harris Corner Detector (BHCD) has been proposed in this paper. In the proposed BHCD, a Bilateral filter preserves edges by smoothening and removing noise in an image. Accuracy in localization of interest points are improved by using the proposed dynamic blur metric calculation. The Harris corner has been added to get stable and reliable interest point detection. The proposed BHCD has been simulated for the evaluation criteria such as repeatability and matching score. Extensive experimental results show that the proposed method is more robust to illumination, scaling, rotation, compression and viewpoint changes. The experimental evaluation for BHCD has been carried for the object recognition benchmark datasets COIL-100, ZuBud, Caltech-101. The proposed BHCD achieves highest recognition rate compared to the other state-of-the-art methods.

Fusion of appearance and motion-based sparse representations for multi-shot person re-identification

We present in this paper a multi-shot human re-identification system from video sequences based on interest points (IPs) matching. Our contribution is to take advantage of the complementary of person’s appearance and style of its movement that leads to a more robust description with respect to various complexity factors. The proposed contributions include person’s description and features matching. For person’s description, we propose to exploit a fusion strategy of two complementary features provided by appearance and motion description. We describe motion using spatiotemporal IPs, and use spatial IPs for describing the appearance. For feature matching, we use Sparse Representation (SR) as a local matching method between IPs. The fusion strategy is based on the weighted sum of matched IPs votes and then applying the rule of majority vote. This approach is evaluated on a large public dataset, PRID-2011. The experimental results show that our approach clearly outperforms current state-of-the-art.

AUTOMATIC RECTIFICATION OF BUILDING FAÇADES

Abstract. Focusing mainly on the case of (near-)planar building façades, a methodology for their automatic projective rectification is described and evaluated. It relies on a suitably configured, calibrated stereo pair of an object expected to contain a minimum of vertical and/or horizontal lines for the purposes of levelling. The SURF operator has been used for extracting and matching interest points. The coplanar points have been separated with two alternative methods. First, the fundamental matrix of the stereo pair, computed using robust estimation, allows estimating the relative orientation of the calibrated pair; initial parameter values, if needed, may be estimated via the essential matrix. Intersection of valid points creates a 3D point set in model space, to which a plane is robustly fitted. Second, all initial point matches are directly used for robustly estimating the inter-image homography of the pair, thus directly selecting all image matches referring to coplanar points; initial values for the relative orientation parameters, if needed, may be estimated from a decomposition of the inter-image homography. Finally, all intersected coplanar model points yield the object-to-image homography to allow image rectification. The in-plane rotation required to finalize the transformation is found by assuming that rectified images contain sufficient straight linear segments to form a dominant pair of orthogonal directions which correspond to horizontality/verticality in 3D space. In our implementation, image edges from Canny detector are used in linear Hough Transform (HT) resulting in a 2D array (ρ, θ) with values equal to the sum of pixels belonging to the particular line. Quantization parameter values aim at absorbing possible slight deviations from collinearity due to thinning or uncorrected lens distortions. By first imposing a threshold expressing the minimum acceptable number of edge-characterized pixels, the resulting HT is accumulated along the ρ-dimension to give a single vector, whose values represent the number of lines of the particular direction. Since here the dominant pair of orthogonal directions has to be found, all vector values are added with their π/2-shifted counterpart. This function is then convolved with a 1D Gaussian function; the optimal angle of in-plane rotation is at the maximum value of the result. The described approach has been successfully evaluated with several building façades of varying morphology by assessing remaining line convergence (projectivity), skewness and deviations from horizontality/verticality. Mean estimated deviation from a metric result was 0°.2. Open questions are also discussed.

Efficient 3D scene abstraction using line segments

Extracting 3D information from a moving camera is traditionally based on interest point detection and matching. This is especially challenging in urban indoor- and outdoor environments, where the number of distinctive interest points is naturally limited. While common Structure-from-Motion (SfM) approaches usually manage to obtain the correct camera poses, the number of accurate 3D points is very small due to the low number of matchable features. Subsequent Multi-view Stereo approaches may help to overcome this problem, but suffer from a high computational complexity. We propose a novel approach for the task of 3D scene abstraction, which uses straight line segments as underlying features. We use purely geometric constraints to match 2D line segments from different images, and formulate the reconstruction procedure as a graph-clustering problem. We show that our method generates accurate 3D models with low computational costs, which makes it especially useful for large-scale urban datasets.

Guest Editorial Special Issue on Mobile Biometrics

Guest Editorial Special Issue on Mobile Biometrics

Incremental Multi-view 3D Reconstruction Starting from Two Images Taken by a Stereo Pair of Cameras

In this paper, we present a new method for multi-view 3D reconstruction based on the use of a binocular stereo vision system constituted of two unattached cameras to initialize the reconstruction process. Afterwards , the second camera of stereo vision system (characterized by varying parameters) moves to capture more images at different times which are used to obtain an almost complete 3D reconstruction. The first two projection matrices are estimated by using a 3D pattern with known properties. After that, 3D scene points are recovered by triangulation of the matched interest points between these two images. The proposed approach is incremental. At each insertion of a new image, the camera projection matrix is estimated using the 3D information already calculated and new 3D points are recovered by triangulation from the result of the matching of interest points between the inserted image and the previous image. For the refinement of the new projection matrix and the new 3D points, a local bundle adjustment is performed. At first, all projection matrices are estimated, the matches between consecutive images are detected and Euclidean sparse 3D reconstruction is obtained. So, to increase the number of matches and have a more dense reconstruction, the Match propagation algorithm, more suitable for interesting movement of the camera, was applied on the pairs of consecutive images. The experimental results show the power and robustness of the proposed approach.

A vision processor with a unified interest point detection and matching hardware for accelerating stereo matching algorithm

In this paper, a unified interest-point detection and matching hardware with an optimized memory architecture is proposed for a real-time stereo-matching system. In order to support a stereo-matching algorithm, the unified datapath in the hardware performs not only interest-point detection and matching algorithms such as features from the accelerated segment test (FAST) and binary robust independent elementary feature (BRIEF) in real time but also a census transform, which is widely used in stereo matching. To achieve maximum performance, we propose two special memory architectures: 1) reconfigurable image memory (RIM) and 2) point cloud index memory system (PCIM). RIM is a unified memory architecture that loads pixel values from a raw image patch. Since FAST, BRIEF, and the census transform have different and complex memory access patterns, the miss rate of the memory access may increase. To optimize the memory operation, RIM can change its memory configuration according to the algorithm. PCIM is a dedicated memory system that utilizes the geometric information of the cameras in order to reduce the off-chip memory bandwidth. Based on the geometric information, PCIM removes most of the redundant candidates. Since PCIM minimizes the off-chip memory bandwidth using a dedicated cache, the performance degradation is negligible compared with the exact-nearest-neighbor method. Area-based stereo matching is accelerated based on the general-purpose computing on graphics processing units (GPGPU) architecture because the search range is adaptively reduced according to the disparity of the matched correspondences. The overall hardware consists of 1.20-M logic gates and consumes a maximum of 185 mW. Interest point detection and matching accelerator achieves 106 frames/s in 1080p full high-definition video (HD) resolution at a 200-MHz operating frequency with 3500 descriptors per image.

A Novel Local Features Based Salient Object Recognition Algorithm via Hybrid SVM-QPSO Model

As the salient objects extraction is of great importance in computer vision and multimedia information retrieval, this paper concentrates on the problem of salient object recognition using local features. Considering the rotational invariance performance of circular region is much better, we exploit a circular region to replace the rectangular region. To implement the salient object detection, the visual object classes should be constructed from training image dataset through SIFT features clustering. Furthermore, for a test image, the object class which the test image belonged to can be detected by interest points matching. Afterwards, the SIFT features clustering and local features matching process can be implemented through the proposed hybrid SVM-QPSO model. To promote the quality of parameter selection in SVM, we utilize the quantum behaved particle swarm optimization technique to select suitable SVM parameters. Finally, experiments are conducted to make performance evaluation using the MSRC dataset. Experimental results show that compared with other methods, the proposed algorithm can effectively detect salient objects in both object detecting precision and computing efficiency.

Line Matching in Wide-Baseline Stereo: A Top-Down Approach.

This paper introduces a new algorithm for matching lines across images that exploit the epipolar geometry and the coplanarity constraints between pairs of lines. In contrast to common treatment in matching of interest points, we use the epipolar geometry to constrain coplanarity conditions between line-pairs. This treatment eliminates the potential matching problems due to the incomplete line observations with nonmatching endpoints. This observation is used to detect a set of candidate line-pair correspondences. These matching pairs are then verified via local homography transforms derived from the neighboring interest point correspondences. This step results in a line affinity matrix which is processed to obtain matching lines. During this process, we do not use appearance models and show that the proposed treatment performs better than the state-ofthe- art appearance and geometry based methods especially for images with wide-baseline.