Feature Point Pairs Research Articles

A Novel Visual SLAM Based on Multiple Deep Neural Networks

The current visual simultaneous localization and mapping (SLAM) systems require the use of matched feature point pairs to estimate camera pose and construct environmental maps. Therefore, they suffer from poor performance of the visual feature matchers. To address this problem, a visual SLAM using deep feature matcher is proposed, which is mainly composed of three parallel threads: Visual Odometry, Backend Optimizer and LoopClosing. In the Visual Odometry, the deep feature extractor with convolutional neural networks is utilized for extracting feature points in each image frame. Then, the deep feature matcher is used for obtaining the corresponding feature landmark pairs. Afterwards, a fusion method based on the last and the reference frame is proposed for camera pose estimation. The Backend Optimizer is designed to execute local bundle adjustment for a part of camera poses and landmarks (map points). While LoopClosing, consisting of a lightweight deep loop closure detector and the same matcher as the one used in Visual Odometry is utilized for loop correction based on pose graph. The proposed system has been tested extensively on most of benchmark KITTI odometry dataset. The experimental results show that our system yields better performance than the existing visual SLAM systems. It can not only run in real-time at a speed of 0.08 s per frame, but also reduce estimation error by at least 0.1 m.

Robust pose estimation for ship block assembly feature based on large-scale scanning

PurposeHull block assembly is a vital task in ship construction. It is necessary to obtain the actual poses of the assembly features to guide further block alignment. Traditional methods use single-point measurement, which is time-consuming and may lead to loss of key information. Thus, large-scale scanning is introduced for data acquisition, and this paper aims to provide a precise and robust method for retrieving poses based on point set registration.Design/methodology/approachThe main problem of point registration is to find the correct transformation between the model and the scene. In this paper, a vote framework based on a new point pair feature is used to calculate the transformation. First, a special edge indicator for multiplate objects is proposed to determine the edges. Subsequently, pair features with an edge description are noted for every point. Finally, a voting scheme based on agglomerative clustering is implemented to determine the optimal transformation.FindingsThe proposed method not only improves registration efficiency but also maintains high accuracy compared to several commonly used approaches. In particular, for objects composed of plates, the results of pose estimation are more promising because of the compact pair feature. The multiple ship longitudinal localization experiment validates the effectiveness in real scan applications.Originality/valueThe proposed edge description performs a better detection for the edges of multiplate objects. The pair feature incorporating the edge indicator is more discriminative than the original template, resulting in better robustness to outliers, noise and occlusions.

A robust video stabilization method for camera shoot in mobile devices using GMM-based motion estimator

The application of video stabilization methods to mobile devices with low computing power is an important issue. In this study, we aim to provide a robust video stabilization solution for such mobile devices with limited computing resources. We first propose a fast feature extraction method based on a block-wise gradient. Then, we extract motion information through feature point pairs and rebuild the camera path. Finally, the data were smoothed and the corresponding compensation matrix was calculated to obtain a stable video. However, large foreground objects may lead to incorrect motion estimation. Thus, we propose a fast foreground segmentation method based on GMM for jitter scenes to eliminate foreground features and obtain a more accurate camera trajectory. The experimental results show that the proposed method is robust against large foreground objects and exhibits good real-time performance.

A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic Picking

An apple-picking robot is now the most widely accepted method in the substitution of low-efficiency and high-cost labor-intensive apple harvesting. Although most current research on apple-picking robots works well in the laboratory, most of them are unworkable in an orchard environment due to unsatisfied apple positioning performance. In general, an accurate, fast, and widely used apple positioning method for an apple-picking robot remains lacking. Some positioning methods with detection-based deep learning reached an acceptable performance in some orchards. However, apples occluded by apples, leaves, and branches are ignored in these methods with detection-based deep learning. Therefore, an apple binocular positioning method based on a Mask Region Convolutional Neural Network (Mask R-CNN, an instance segmentation network) was developed to achieve better apple positioning. A binocular camera (Bumblebee XB3) was adapted to capture binocular images of apples. After that, a Mask R-CNN was applied to implement instance segmentation of apple binocular images. Then, template matching with a parallel polar line constraint was applied for the stereo matching of apples. Finally, four feature point pairs of apples from binocular images were selected to calculate disparity and depth. The trained Mask R-CNN reached a detection and segmentation intersection over union (IoU) of 80.11% and 84.39%, respectively. The coefficient of variation (CoV) and positioning accuracy (PA) of binocular positioning were 5.28 mm and 99.49%, respectively. The research developed a new method to fulfill binocular positioning with a segmentation-based neural network.

Image Stitching Based on Color Difference and KAZE with a Fast Guided Filter

Image stitching is of great importance for multiple fields such as moving object detection and tracking, ground reconnaissance and augmented reality. To ameliorate the stitching effect and alleviate the mismatch rate, an effective image stitching algorithm based on color difference and an improved KAZE with a fast guided filter is proposed. Firstly, the fast guided filter is introduced to reduce the mismatch rate before feature matching. Secondly, the KAZE algorithm based on improved random sample consensus is used for feature matching. Then, the color difference and brightness difference of the overlapping area are calculated to make an overall adjustment to the original images so as to improve the nonuniformity of the splicing result. Finally, the warped images with color difference compensation are fused to obtain the stitched image. The proposed method is evaluated by both visual effect mapping and quantitative values. In addition, the proposed algorithm is compared with other current popular stitching algorithms. The results show that the proposed algorithm is superior to other algorithms in terms of the quantity of feature point pairs, the matching accuracy, the root mean square error and the mean absolute error.

Cross-Viewpoint Template Matching Based on Heterogeneous Feature Alignment and Pixel-Wise Consensus for Air- and Space-Based Platforms

Template matching is the fundamental task in remote sensing image processing of air- and space-based platforms. Due to the heterogeneous image sources, different scales and different viewpoints, the realization of a general end-to-end matching model is still a challenging task. Considering the abovementioned problems, we propose a cross-view remote sensing image matching method. Firstly, a spatial attention map was proposed to solve the problem of the domain gap. It is produced by two-dimensional Gaussian distribution and eliminates the distance between the distributed heterogeneous features. Secondly, in order to perform matching at different flight altitudes, a multi-scale matching method was proposed to perform matching on three down-sampling scales in turn and confirm the optimal result. Thirdly, to improve the adaptability of the viewpoint changes, a pixel-wise consensus method based on a correlation layer was applied. Finally, we trained the proposed model based on weakly supervised learning, which does not require extensive annotation but only labels one pair of feature points of the template image and search image. The robustness and effectiveness of the proposed methods were demonstrated by evaluation on various datasets. Our method accommodates three types of template matching with different viewpoints, including SAR to RGB, infrared to RGB, and RGB to RGB.

Wheeled Robot Visual Odometer Based on Two-dimensional Iterative Closest Point Algorithm

According to the two-dimensional motion characteristics of planar motion wheeled robot, the visual odometer was dimensionally reduced in this study. In the feature point matching part of visual odometer, the contour constraint was used to filter out the mismatched feature point pairs (abbreviated as FPP). This method could also filter out the matched FPP, and the feature of FPP was correct color image matches, however, their depth image error was large. This offered higher quality matched FPP for the subsequent interframe motion estimation. Dimension reduction was performed in the interframe motion estimation part, and the two-dimensional Iterative Closest Point (ICP) algorithm was used for camera motion estimation. The experiments indicated that the proposed algorithm effectively improved the computational speed and precision of planar motion wheeled robot visual odometer. This research indicates that the dimension reduction processing of ICP algorithm can effectively improve the operation speed and calculation accuracy of planar motion wheeled robot visual odometry, which provides a good reference and data support for the subsequent research of wheeled robot visual odometry in the future.

A Pose Estimation Method Combining Instance Segmentation and Point Pair Features

For the traditional pose estimation method based on point pair features, both the preprocessing of scene point cloud and the construction of point pair features of scene point cloud have serious time-consuming problems, which cannot meet the needs of actual industry. Therefore, this paper proposes a pose estimation method that combines instance segmentation and point pair features. Therefore, this paper proposes a pose estimation method that combines instance segmentation and point pair features. First, use the Mask R-CNN-based instance segmentation network to obtain the location of the target object in the two-dimensional image of the scene; then, obtain the local point cloud data of the space where the target object is located from this position and the depth information of the scene; finally, the local the point cloud data is used as the scene point cloud based on point pair feature pose estimation to perform feature matching with the point cloud of the target object.

Automatic drift compensation for nanoscale imaging using feature point matching

An implementation of drift compensation for imaging at the nanoscale is presented. The method is based on computer vision techniques and hence applicable to any microscope that generates images through a computer interface. The algorithm extracts and matches pairs of feature points from consecutive images to compute and compensate for probe–sample misalignments over time. The protocol also applies selection rules that enable it to withstand significant changes in image contrast. We demonstrate our fully automatic implementation by continuously imaging the same area of a Si(100) surface at the atomic scale with scanning probe microscopy over a period of 25 h at room temperature, showing that the method is robust even under the presence of non-linear drift or spontaneous changes of the probe apex. We apply our method to study the movement of pairs of tin atoms confined within a half-unit cell of the Si(111)-(7 × 7) surface and estimate the energy barrier for their diffusion at room temperature.

Vision geometry-based UAV flocking

AbstractA distributed UAV (unmanned aerial vehicle) flocking control method based on vision geometry is proposed, in which only monocular RGB (red, green, blue) images are used to estimate the relative positions and velocities between drones. It does not rely on special visual markers and external infrastructure, nor does it require inter-UAV communication or prior knowledge of UAV size. This method combines the advantages of deep learning and classical geometry. It adopts a deep optical flow network to estimate dense matching points between two consecutive images, uses segmentation technology to classify these matching points into background and specific UAV, and then maps the classified matching points to Euclidean space based on the depth map information. In 3D matching points, also known as 3D feature point pairs, each of their classifications is used to estimate the rotation matrix, translation vector, velocity of the corresponding UAV, as well as the relative position between drones, based on RANSAC and least squares method. On this basis, a flocking control model is constructed. Experimental results in the Microsoft Airsim simulation environment show that in all evaluation metrics, our method achieves almost the same performance as the UAV flocking algorithm based on ground truth cluster state.

DIC measurement method for large rotation based on improved grid-based motion statistics.

As a noncontact optical measurement method, the digital image correlation (DIC) method can provide full-field displacement and strain measurement during object deformation. In the case of small rotation deformation, the traditional DIC method can obtain accurate deformation measurement results. However, when the object rotates at a large angle, the traditional DIC method cannot obtain the extreme value of the correlation function, resulting in the occurrence of decorrelation. In order to address the issue, a full-field deformation measurement DIC method based on improved grid-based motion statistics is proposed for large rotation angles. First, the speeded up robust features algorithm is applied to extract and match the feature point pairs between the reference image and the deformed image. Furthermore, an improved grid-based motion statistics algorithm is proposed to eliminate the wrong matching point pairs. Then, the deformation parameters of the feature point pairs obtained by the affine transformation are taken as the initial deformation value for DIC calculation. Finally, the intelligent gray-wolf optimization algorithm is used to obtain the accurate displacement field. The effectiveness of the proposed method is proved by simulation and practical experiments, and the comparative experiments show that the proposed method is faster and more robust.

Methodological Research on Image Registration Based on Human Brain Tissue In Vivo

As one of the critical steps in brain imaging analysis and processing, brain image registration plays a significant role. In this paper, we proposed a technique of human brain image registration based on tissue morphology in vivo to address the problems of previous image registration. First, different feature points were extracted and combined, including those at the boundary of different brain tissues and those of the maximum or minimum from the original image. Second, feature points were screened through eliminating their wrong matching pairs between moving image and reference image. Finally, the remaining matching pairs of feature points were used to generate the model parameters of spatial transformation, with which the brain image registration can be finished by combining interpolation techniques. Results showed that compared with the Surf, Demons, and Sift algorithms, the proposed method can perform better not only for four quantitative indicators (mean square differences, normalized cross correlation, normalized mutual information and mutual information) but also in spatial location, size, appearance contour, and registration details. The findings may suggest that the proposed method will be of great value for brain image reconstruction, fusion, and statistical comparison analysis.

Feature extraction algorithm of an irregular small celestial body in a weak light environment.

This study focuses on creating a crater-matching algorithm to improve the matching rate and address the phenomenon of insufficient feature extraction and mismatching of irregular celestial objects and crater edge information on the dim surface of celestial bodies images. These images were captured by the detector's navigation camera. In order to improve the brightness and clarity of the images, the target images were filtered, denoised, and image-enhanced using the bilateral filtering method and improved histogram equalization algorithm, successively. Then, the enhanced image was extracted and matched using the ORB feature point detection algorithm based on scale invariance, and the feature point mismatch was processed by the Hamming distance screening method. The simulation results revealed that the optimization algorithm effectively improved the imaging quality of the target image in dark and weak light environments, increased the number of feature points extracted, reduced the mismatch of effective feature point pairs, and improved the matching rate.

3D Recognition Algorithm Based on Curvature Point Pair Features

3D Recognition Algorithm Based on Curvature Point Pair Features

Improved SURF method in digital image correlation for estimation of large rotation angle

Digital image correlation (DIC) is widely used in optical measurement techniques with the advantages of high precision, robustness and expansibility. The conventional DIC, however, is highly dependent on the accuracy of initial parameters, and there are reports [1,2] that it may fail when the rotation angle of the target specimen exceeds 7°. The combination of Speeded-Up Robust Features (SURF) with Random Sample Consensus (RANSAC) algorithm has been applied to solve this problem with limited success due to the effects of residual mismatches. An enhanced algorithm based on SURF is proposed in this study to address this problem. The transformation model between two images is obtained by RANSAC similar to existing methods. Then an angle compensation strategy is proposed to estimate the rotation angle of the deformed image to form the compensated image. Finally, an iterative process is proposed to refine the initial parameters from the reference and deformed images with a new convergence threshold. In addition, an evaluation criterion developed from a multiple ring template is proposed to compute the similarity of feature point pairs to improve the speed of convergence and sampling accuracy. Results from numerical simulation and experimental analysis show that the accuracy and robustness of the proposed method are significantly improved. Further application of the proposed method to monitor the rotating blades of a scaled model wind turbine illustrates further the capability of the proposed method.

A Monocular-Visual SLAM System with Semantic and Optical-Flow Fusion for Indoor Dynamic Environments.

A static environment is a prerequisite for the stable operation of most visual SLAM systems, which limits the practical use of most existing systems. The robustness and accuracy of visual SLAM systems in dynamic environments still face many complex challenges. Only relying on semantic information or geometric methods cannot filter out dynamic feature points well. Considering the problem of dynamic objects easily interfering with the localization accuracy of SLAM systems, this paper proposes a new monocular SLAM algorithm for use in dynamic environments. This improved algorithm combines semantic information and geometric methods to filter out dynamic feature points. Firstly, an adjusted Mask R-CNN removes prior highly dynamic objects. The remaining feature-point pairs are matched via the optical-flow method and a fundamental matrix is calculated using those matched feature-point pairs. Then, the environment's actual dynamic feature points are filtered out using the polar geometric constraint. The improved system can effectively filter out the feature points of dynamic targets. Finally, our experimental results on the TUM RGB-D and Bonn RGB-D Dynamic datasets showed that the proposed method could improve the pose estimation accuracy of a SLAM system in a dynamic environment, especially in the case of high indoor dynamics. The performance effect was better than that of the existing ORB-SLAM2. It also had a higher running speed than DynaSLAM, which is a similar dynamic visual SLAM algorithm.

Fast plane extraction method based on the point pair feature

Fast plane extraction method based on the point pair feature

Outlier removal and feature point pairs optimization for piecewise linear transformation in the co-registration of very high-resolution optical remote sensing imagery

Co-registration of very high-resolution images is a challenging and essential preprocessing step in remote sensing image analysis. Piecewise linear (PL) transformation has been shown to accurately handle the registration of very high-resolution images with various local deformations. However, the PL transformation has strict requirements on the structure of the local triangulated irregular network (TIN) and the position accuracy of feature point pairs (FPPs). Existing outlier removal methods barely consider the local TIN structure of FPPs, thus failing to meet the requirements of the PL transformation. Moreover, the inevitably remaining outliers lead to lower local registration accuracy. This paper proposes an outlier removal method and an FPPs optimization method to improve the registration accuracy of the PL transformation. First, we developed a triangle-area representation ratio of the triangulated irregular network (TIN-TARR) descriptor to identify the outliers. Then, we divided these outliers into three types and removed them iteratively. Finally, we further optimized the FPPs by maximizing the similarity between the fixed and the registered images to improve the registration accuracy of the PL transformation. The proposed method was validated using three pairs of TripleSat-2 images with a spatial resolution of 3.2 m, one pair of WorldView-2 images with a spatial resolution of 1.6 m, and one pair of aerial images with a spatial resolution of 0.3 m. The results demonstrated that: (1) The obvious deformations in each registered image were eliminated after the outlier removal using TIN-TARR. (2) The FPPs optimization significantly improved the registration accuracy of the PL transformation. The average registration errors in five tests were 0.542, 0.389, 0.456, 0.420, and 0.592 pixels, respectively. (3) Different similarity metrics had various effects on the FPPs optimization, and structural similarity was the most effective measurement. (4) The proposed co-registration framework presented better registration accuracy than three state-of-the-art methods, especially in images with land cover changes and shadows.

Efficient MSPSO Sampling for Object Detection and 6-D Pose Estimation in 3-D Scenes

The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">point pair feature</i> (PPF) is widely used in manufacturing for estimating 6-D poses. The key to the success of PPF matching is to establish correct 3-D correspondences between the object and the scene, i.e., finding as many valid similar point pairs as possible. However, efficient sampling of point pairs has been overlooked in existing frameworks. In this article, we propose a revised PPF matching pipeline to improve the efficiency of 6-D pose estimation. Our basic idea is that the valid scene reference points are lying on the object’s surface and the previously sampled reference points can provide prior information for locating new reference points. The novelty of our approach is a new sampling algorithm for selecting scene reference points based on the multisubpopulation particle swarm optimization guided by a probability map. We also introduce an effective pose clustering and hypotheses verification method to obtain the optimal pose. Moreover, we optimize the progressive sampling for multiframe point clouds to improve processing efficiency. The experimental results show that our method outperforms previous methods by 6.6%, 3.9% in terms of accuracy on the public DTU and LineMOD datasets, respectively. We further validate our approach by applying it in a real robot grasping task.

Method for extrinsic parameters calibration of Pan-Tilt camera using intersecting lines from two images

Pan-Tilt (PT) cameras are widely used in video surveillance because of larger field of view. The camera’s intrinsic parameters remain unchanged and the extrinsic parameters calibration of the PT camera should be addressed when it comes to visual geometric measurement. In this paper, the unknown parameters of the PT camera are only reduced to the rotation angles in horizontal and vertical directions, and a method for extrinsic parameters calibration using intersecting lines from two images is presented. An improved feature point matching algorithm is proposed to obtain the natural feature point pairs between images acquired before and after rotation, and direct linear transformation method is employed to obtain the rough rotation angles by using the natural feature point pairs. Based on rotation constraint, the optimization function established by the projections of the intersections of the straight lines in natural scene is then used to calculate the refined extrinsic parameters. Computer simulations and real experiments prove the effectiveness of the improved feature point matching algorithm and the proposed calibration method. Experimental data reflects the extrinsic parameters estimated by our method are close to reference values from high-precision stepper motors, and the average deviation of Euler angles representing rotation matrix and translations from our method relative to reference values does not exceed 0.03° and 0.27 mm, respectively.