Keypoint Pairs Research Articles

Discrete Cycle-Consistency Based Unsupervised Deep Graph Matching

We contribute to the sparsely populated area of unsupervised deep graph matching with application to keypoint matching in images. Contrary to the standard supervised approach, our method does not require ground truth correspondences between keypoint pairs. Instead, it is self-supervised by enforcing consistency of matchings between images of the same object category. As the matching and the consistency loss are discrete, their derivatives cannot be straightforwardly used for learning. We address this issue in a principled way by building our method upon the recent results on black-box differentiation of combinatorial solvers. This makes our method exceptionally flexible, as it is compatible with arbitrary network architectures and combinatorial solvers. Our experimental evaluation suggests that our technique sets a new state-of-the-art for unsupervised graph matching.

Improving ellipse fitting via multi-scale smoothing and key-point searching

Fast and efficient fitting of accurate ellipses from data points has many applications in pattern recognition, machine vision, and robotics. However, the fitting accuracy may significantly degrade in the existence of outliers, such as the least-squares-based approaches. Despite robust methods attaining more accurate results than the least-squares manner under the contamination of outliers, they typically require the careful tuning of the hyper-parameters for good results. To mitigate the outlier disturbance, in this paper, we propose a conceptually simple yet quite useful preprocessing framework for high-precision ellipse fitting. Firstly, we leverage multi-scale operators to shrink the input image, by which a large number of outliers can be removed, followed by the smoothing of the sub-image to further improve the data quality. Then, we propose a key-point searching method to enhance the fitting precision via the analysis of the discrete pixel data in images. We prove that key-point-based ellipse fitting gives the upper bound of the approximation error generated by other sampled points with the same ellipse. Based on the key-point pairs inside and outside the ellipse, we further calculate their barycentric points and then perform fitting on these points to attain high-precision ellipses. We conduct extensive experiments on synthetic and real-world images to validate the proposed method and compare it with representative state-of-the-art approaches. Quantitative and qualitative results demonstrate that our method has more accurate and robust performance than competitors. Additionally, we employ the proposed method to compared approaches as a preprocessing step. Experiments demonstrate that our method is effective to significantly improve their fitting accuracy. Our source code is freely available at https://github.com/ChengQian09/MSKPF.

Unsupervised Non-rigid Histological Image Registration Guided by Keypoint Correspondences Based on Learnable Deep Features with Iterative Training.

Histological image registration is a fundamental task in histological image analysis. It is challenging because of substantial appearance differences due to multiple staining. Keypoint correspondences, i.e., matched keypoint pairs, have been introduced to guide unsupervised deep learning (DL) based registration methods to handle such a registration task. This paper proposes an iterative keypoint correspondence-guided (IKCG) unsupervised network for non-rigid histological image registration. Fixed deep features and learnable deep features are introduced as keypoint descriptors to automatically establish keypoint correspondences, the distance between which is used as a loss function to train the registration network. Fixed deep features extracted from DL networks that are pre-trained on natural image datasets are more discriminative than handcrafted ones, benefiting from the deep and hierarchical nature of DL networks. The intermediate layer outputs of the registration networks trained on histological image datasets are extracted as learnable deep features, which reveal unique information for histological images. An iterative training strategy is adopted to train the registration network and optimize learnable deep features jointly. Benefiting from the excellent matching ability of learnable deep features optimized with the iterative training strategy, the proposed method can solve the local non-rigid large displacement problem, an inevitable problem usually caused by misoperation, such as tears in producing tissue slices. The proposed method is evaluated on the Automatic Non-rigid Histology Image Registration (ANHIR) website and AutomatiC Registration Of Breast cAncer Tissue (ACROBAT) website. It ranked 1st on both websites as of August 6th, 2024.

Robust Posegraph Optimization Using Proximity Points

This paper proposes a robust posegraph optimization (PGO) method for posegraphs with keypoints. In the conventional PGO formulation, a loop constraint is defined between a pair of nodes, whereas in the proposed method, it is defined between a pair of keypoints. In this manner, robust PGO based on switch variables can be realized in a more fine-grained manner. Loop constraint is defined based on the unique geometric property of proximity point, and implemented as a new edge type of the g2o solver. The proposed method is compared with other robust PGO methods using real world data recorded in Nakanoshima Robot Challenge 2021.

A Novel Copy–Move Forgery Detection Algorithm via Gradient-Hash Matching and Simplified Cluster-Based Filtering

Copy–move forgery is one of the most frequently used methods for producing fake digital images. Current algorithms for copy–move forgery detection (CMFD) cannot combine high accuracy and fast speed. Motivated by the observation, we propose a novel CMFD algorithm whose workflow is as follows. First, we use a keypoint-extraction method with the lowest contrast threshold to extract more keypoints from the input image. Second, a new technique, gradient-hash matching, finds pairs of similar keypoints quickly and effectively using a hash table, where the hash value is computed using gradients of keypoints. Subsequently, a new method called simplified cluster-based filtering exploits the density pattern of keypoints in the copy–move regions to remove false matching keypoint pairs. Finally, image matting is applied to indicate the forgery regions vividly. Extensive experiments show that not only the new algorithm is better than the state-of-the-art algorithms in terms of computation correctness, but also its computation time is drastically less. Commonly only about half time is needed. The relative time saving is even higher when images are larger. Different algorithms modules are compared through experiments to choose the best combination.

SD-CapsNet: A Siamese Dense Capsule Network for SAR Image Registration with Complex Scenes

SAR image registration is the basis for applications such as change detection, image fusion, and three-dimensional reconstruction. Although CNN-based SAR image registration methods have achieved competitive results, they are insensitive to small displacement errors in matched point pairs and do not provide a comprehensive description of keypoint information in complex scenes. In addition, existing keypoint detectors are unable to obtain a uniform distribution of keypoints in SAR images with complex scenes. In this paper, we propose a texture constraint-based phase congruency (TCPC) keypoint detector that uses a rotation-invariant local binary pattern operator (RI-LBP) to remove keypoints that may be located at overlay or shadow locations. Then, we propose a Siamese dense capsule network (SD-CapsNet) to extract more accurate feature descriptors. Then, we define and verify that the feature descriptors in capsule form contain intensity, texture, orientation, and structure information that is useful for SAR image registration. In addition, we define a novel distance metric for the feature descriptors in capsule form and feed it into the Hard L2 loss function for model training. Experimental results for six pairs of SAR images demonstrate that, compared to other state-of-the-art methods, our proposed method achieves more robust results in complex scenes, with the number of correctly matched keypoint pairs (NCM) at least 2 to 3 times higher than the comparison methods, a root mean square error (RMSE) at most 0.27 lower than the compared methods.

A Novel Image Copy–Move Forgery Detection Algorithm Using the Characteristics of Local Descriptors

The image copy–move forgery can achieve the realistic effects without leaving any obvious manipulation traces because the copied source and pasted target belong to the homogenous sources. Image copy–move forgery detection (CMFD) is still a serious challenge issue for researchers, e.g. computational efficiency, occlusive (smoothing) object detection, and partial object detection, under various post-processing and geometrical transformations. This paper proposes a novel CMFD scheme using the characteristics of local descriptors. First, the proposed method deeply analyzes the structure and mines the inherent characteristics of the local descriptor (SURF) for feature extraction in the coarse and smooth regions. Subsequently, the proposed method takes the kernel features as coarse feature matching to decrease matching costs. Then, the found candidate keypoints with a relatively small quantity are used with complete features to perform fine keypoint matching for getting suspicious candidate keypoint pairs. Furthermore, based on the deep mining of the inherent local descriptor characteristics, the proposed method can indicate the forgery region localizations through keypoint pairs, even the forgeries suffering from various geometrical transformations. Then, the proposed method proposes adaptive dual-filtering algorithms of [Formula: see text]-means relying on keypoint characteristics, and spatial distance for hierarchical progress of feature filtering. Finally, Delaunay Triangulation relies on the true-positive keypoint pairs to perform the forgery region matting effectively and efficiently. The experimental results demonstrate that the proposed method can achieve the best F1 scores compared to the state-of-the-art methods in most cases, especially in anti-scaling transformations.

Topology-controllable Implicit Surface Reconstruction Based on Persistent Homology

In geometric design, reconstructing an implicit surface with high-quality geometry and expected topology from point clouds has been a challenging problem. However, traditional topological invariants, such as Betti numbers, are not easily used in controlling topology in implicit surface reconstruction. Persistent homology provides a quantitative measurement, i.e. , persistence diagram (PD), and allows tracking of pairs of key points that generate and destroy topological holes. In this study, we propose a topology-controllable implicit surface reconstruction method from point clouds based on persistent homology. Specifically, given a point cloud with normals, the signed distance field is first constructed, and then a B-spline function represented distance function is generated by fitting the signed distance values through progressive iterative approximation. By designing a topological target function using the persistent pairs in PDs according to topological priors, the control coefficients of the B-spline function are optimized to extract an implicit surface, i.e. , an iso-surface of the B-spline function, with expected topology. Experiments show that the proposed method can reconstruct surfaces with higher topological quality than other reconstruction methods. Moreover, the proposed method can be used to edit the topology of an implicit surface. • The topology-controllable framework for implicit B-spline surface reconstruction improves geometric quality and ensures the desired topology of the extracted surface. • Principles of designing the topological target function for implicit surface reconstruction are proposed. • Reconstruction on sparse point clouds and the application of topology editing are explored.

Rethinking of learning-based 3D keypoints detection for large-scale point clouds registration

The main solution for large-scale point clouds registration is to first obtain a set of matched 3D keypoint pairs and then accomplish the point cloud registration task based on these matched keypoint pairs. However, at present, many methods study the feature descriptors in the point clouds registration task, but few methods discuss the 3D keypoints detection issues. The commonly used 3D keypoints detection strategy is the voxel-grid-based downsampling method, and the detected 3D keypoints are usually with a relatively huge amount and also with no explicit geometrical properties, which finally leads to a low inlier ratio. In this study, we rethink the 3D keypoints detection problem for large-scale point clouds with deep learning. Specifically, we discuss four kinds of 3D keypoints detection methods based on the joint keypoint detection and description learning framework D3Feat, and carry out extensive analyses on both the indoor large-scale point clouds dataset 3DMatch and the outdoor large-scale point clouds dataset KITTI Odometry. Experimental results demonstrate that the Multi-layer Perceptron (MLP) based method achieves the best inlier ratios under the different numbers of extracted 3D keypoints on both the indoor and outdoor large-scale point clouds. Further, we test these four kinds of keypoints detection methods under the application of large-scale point clouds registration, and the MLP-based method also achieves the state-of-the-art registration performance.

Application of Sparse-Channel Lidar Sensors on Viewpoint-Invariant Loop Closing Task

Loop closing is an essential task for SLAM system, which can reduce the pose drifts and map distortion. However, challenges from viewpoint changes and sparse-channel lidar are still open problems. In this paper, a novel efficient and viewpoint-invariant loop closing method is presented, which well fits sparse-channel lidars. To keep the metric information of 3D point clouds, elevation maps are explored to represent sparse point clouds, based on which a novel and viewpoint-invariant key point feature detection method is proposed. To improve efficiency, Bag of Words technique is applied to accelerate the process of scene description and matching. In addition, a strict geometric consistency check of matched key point pairs is proposed to reject false positive detections, which makes our method robust to scenes with similar layouts in urban environments. Extensive experiments on challenging public datasets verify the validity and superiority of the proposed method.

A Large-Scale Mouse Pose Dataset for Mouse Pose Estimation

Mouse pose estimations have important applications in the fields of animal behavior research, biomedicine, and animal conservation studies. Accurate and efficient mouse pose estimations using computer vision are necessary. Although methods for mouse pose estimations have developed, bottlenecks still exist. One of the most prominent problems is the lack of uniform and standardized training datasets. Here, we resolve this difficulty by introducing the mouse pose dataset. Our mouse pose dataset contains 40,000 frames of RGB images and large-scale 2D ground-truth motion images. All the images were captured from interacting lab mice through a stable single viewpoint, including 5 distinct species and 20 mice in total. Moreover, to improve the annotation efficiency, five keypoints of mice are creatively proposed, in which one keypoint is at the center and the other two pairs of keypoints are symmetric. Then, we created simple, yet effective software that works for annotating images. It is another important link to establish a benchmark model for 2D mouse pose estimations. We employed modified object detections and pose estimation algorithms to achieve precise, effective, and robust performances. As the first large and standardized mouse pose dataset, our proposed mouse pose dataset will help advance research on animal pose estimations and assist in application areas related to animal experiments.

GKNet: Grasp keypoint network for grasp candidates detection

Contemporary grasp detection approaches employ deep learning to achieve robustness to sensor and object model uncertainty. The two dominant approaches design either grasp-quality scoring or anchor-based grasp recognition networks. This paper presents a different approach to grasp detection by treating it as keypoint detection in image-space. The deep network detects each grasp candidate as a pair of keypoints, convertible to the grasp representation g = {x,y,w,θ} T , rather than a triplet or quartet of corner points. Decreasing the detection difficulty by grouping keypoints into pairs boosts performance. To promote capturing dependencies between keypoints, a non-local module is incorporated into the network design. A final filtering strategy based on discrete and continuous orientation prediction removes false correspondences and further improves grasp detection performance. GKNet, the approach presented here, achieves a good balance between accuracy and speed on the Cornell and the abridged Jacquard datasets (96.9% and 98.39% at 41.67 and 23.26 fps). Follow-up experiments on a manipulator evaluate GKNet using four types of grasping experiments reflecting different nuisance sources: static grasping, dynamic grasping, grasping at varied camera angles, and bin picking. GKNet outperforms reference baselines in static and dynamic grasping experiments while showing robustness to varied camera viewpoints and moderate clutter. The results confirm the hypothesis that grasp keypoints are an effective output representation for deep grasp networks that provide robustness to expected nuisance factors.

Multi-Hypothesis Topological Isomorphism Matching Method for Synthetic Aperture Radar Images with Large Geometric Distortion

The dramatic undulations of a mountainous terrain will introduce large geometric distortions in each Synthetic Aperture Radar (SAR) image with different look angles, resulting in a poor registration performance. To this end, this paper proposes a multi-hypothesis topological isomorphism matching method for SAR images with large geometric distortions. The method includes the Ridge-Line Keypoint Detection (RLKD) and Multi-Hypothesis Topological Isomorphism Matching (MHTIM). Firstly, based on the analysis of the ridge structure, a ridge keypoint detection module and a keypoint similarity description method are designed, which aim to quickly produce a small number of stable matching keypoint pairs under large look angle differences and large terrain undulations. The keypoint pairs are further fed into the MHTIM module. Subsequently, the MHTIM method is proposed, which uses the stability and isomorphism of the topological structure of the keypoint set under different perspectives to generate a variety of matching hypotheses, and iteratively achieves the keypoint matching. This method uses both local and global geometric relationships between two keypoints, hence it achieving better performance compared with traditional methods. We tested our approach on both simulated and real mountain SAR images with different look angles and different elevation ranges. The experimental results demonstrate the effectiveness and stable matching performance of our approach.

Cross-frame keypoint-based and spatial motion information-guided networks for moving vehicle detection and tracking in satellite videos

Deep learning methods have achieved the state-of-the-art performance of object detection and tracking in natural images, such as keypoint-based detectors and appearance/motion-based trackers. However, for small and blurry moving vehicles in satellite videos, keypoint-based detectors cause the missed detection of keypoints and incorrect keypoint matching. In terms of multi-object tracking, it is difficult to track the crowded similar vehicles stably only by using the appearance or motion information. To address these problems, a novel deep learning framework is proposed for moving vehicle detection and tracking in the satellite videos. It is comprised of the cross-frame keypoint-based detection network (CKDNet) and spatial motion information-guided tracking network (SMTNet). In CKDNet, a customized cross-frame module is designed to assist the detection of keypoints by exploiting complementary information between frames. Furthermore, CKDNet improves keypoint matching by incorporating size prediction around the keypoints and defining the soft mismatch suppression for out-of-size keypoint pairs. Based on high-quality detection, SMTNet can track the densely-packed vehicles effectively by constructing two-branch long short-term memories. It extracts not only spatial information of the same frame by considering the relative spatial relationship of neighboring vehicles, but also motion information among consecutive frames by calculating the movement velocity. Especially, it regresses virtual positions for missed or occluded vehicles and keeps on tracking these vehicles while they reappear. Experimental results on Jilin-1 and SkySat satellite videos demonstrate the effectiveness of the proposed detection and tracking methods.

A novel copy-move forgery detection algorithm via two-stage filtering

With the wide application of simple image editing software, forgery images have become severe social problems with extremely damaging effects. The copy-move forgery technique is commonly used to temper an image by copying some regions of an image and pasting them somewhere in the same image. In this paper, a novel algorithm is proposed for the detection of copy-move forgery. First, the keypoints of the input image are computed using a keypoint extraction algorithm. Second, a keypoint-matching algorithm is used to match similar keypoints as the candidate keypoint pairs. Third, a proposed novel Two-Stage Filtering algorithm, including the Grid-Based Filter and the Clustering-Based Filter, is applied to filter out most of the false matching keypoint pairs. Subsequently, image matting is achieved by the Delaunay triangulation algorithm so that the marked areas indicate the forgery regions. Three copy-move forgery datasets are used to compare the performances of the proposed algorithm with some state-of-the-art algorithms. The experimental results demonstrate that the proposed algorithm's overall performance is superior to other solutions for detecting copy-move forgery images.

Copy Move Forgery Detection based on double matching

Copy Move is a technique widespreadly used in digital image tampering, meaning Copy Move Forgery Detection (CMFD) is still a significant research. In this paper, a novel CMFD method is proposed, including double matching process and region localizing process. In double matching process, the first matching is conducted on Delaunay triangles consisting of Local Intensity Order Pattern (LIOP) keypoints, to find the approximate location of suspicious regions. In order to find sufficient keypoint pairs, the existing set of matching triangles is expanded by adding their neighbors iteratively, covering the whole tampered regions, and the second matching with a looser threshold is conducted on the vertices. In the region localizing process, considering the case of multiple copies, Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is used to classify the keypoint pairs described in a new model. Experimental results indicate that the proposed method, with good robustness, outperforms some state-of-the-art methods.

A Novel Algorithm for Detecting Pedestrians on Rainy Image.

Pedestrian detection is widely used in cooperative vehicle infrastructure systems. Traditional pedestrian detection methods perform sufficiently well under sunny scenarios and obtain trustworthy traffic data. However, the detection drastically decreases under rainy scenarios. This study proposes a pedestrian detection algorithm with a de-raining module that improves detection accuracy under various rainy scenarios. Specifically, this algorithm determines the density information of rain and effectively removes rain streaks through the de-raining module. Then the algorithm detects pedestrians as a pair of keypoints through the pedestrian detection module to solve the problem of occlusion. Furthermore, a new pedestrian dataset containing rain density labels is established and used to train the algorithm. For the scenarios of light, medium, and heavy rain, extensive experiments on synthetic datasets demonstrate that the proposed algorithm increases AP (average precision) of pedestrian detection by 21.1%, 48.1%, and 60.9%. Moreover, the proposed algorithm performs well on real datasets and achieves improvements over the state-of-the-art methods, which reveals that the proposed algorithm can significantly improve the accuracy of pedestrian detection in rainy scenarios.

C-FCN: Corners-based fully convolutional network for visual object detection

Object detection has achieved significantly progresses in recent years. Proposal-based methods have become the mainstream object detectors, achieving excellent performance on accurate recognition and localization of objects. However, region proposal generation is still a bottleneck. In this paper, to address the limitations of conventional region proposal network (RPN) that defines dense anchor boxes with different scales and aspect ratios, we propose an anchor-free proposal generator named corner region proposal network (CRPN) which is based on a pair of key-points, including top-left corner and bottom-right corner of an object bounding box. First, we respectively predict the top-left corners and bottom-right corners by two sibling convolutional layers, then we obtain a set of object proposals by grouping strategy and non-maximum suppression algorithm. Finally, we further merge CRPN and fully convolutional network (FCN) into a unified network, achieving an end-to-end object detection. Our method has been evaluated on standard PASCAL VOC and MS COCO datasets using a deep residual network. Experiment results present that the proposed method outperforms previous detectors in the term of precision. Additionally, it runs with a speed of 76 ms per image on a single GPU by using ResNet-50 as the backbone, which is faster than other detectors.

Axis bound registration of pan-tilt RGB-D scans for fast and accurate reconstruction

Abstract A fast and accurate algorithm is presented to register scans from an RGB-D camera, which rotates and scans the entire scene in an automated fashion on a pan-tilt platform. The proposed algorithm, Axis Bound Registration, exploits the movement of the camera that is bound by the two rotation axes of pan-tilt servos, so as to realize fast and accurate registration of acquired point clouds. The rotation parameters, including the rotation axes, pan-tilt transformations and the servo control mechanism, are calibrated beforehand. Subsequently, fast global registration can be performed during online operation with transformation matrices formed by the calibrated rotation axes and angles. In local registration, features are extracted and matched between two scenes. For robust registration, false-positive correspondences are rejected based on the distances of pre-oriented keypoint pairs, namely the circle of deviation constraint. Then, a more accurate registration can be achieved by minimizing the residual distances between correspondence pairs, while estimated transformations are bound to the rotation axes. Results of comparative experiments validate that the proposed method outperforms state-of-the-art algorithms of various approaches based on camera calibration, global registration, and simultaneous-localization-and-mapping in terms of root-mean-square error and computation time.

A Simple and Efficient Image Stabilization Method for Coastal Monitoring Video Systems

Fixed video camera systems are consistently prone to importune motions over time due to either thermal effects or mechanical factors. Even subtle displacements are mostly overlooked or ignored, although they can lead to large geo-rectification errors. This paper describes a simple and efficient method to stabilize an either continuous or sub-sampled image sequence based on feature matching and sub-pixel cross-correlation techniques. The method requires the presence and identification of different land-sub-image regions containing static recognizable features, such as corners or salient points, referred to as keypoints. A Canny edge detector ( C E D ) is used to locate and extract the boundaries of the features. Keypoints are matched against themselves after computing their two-dimensional displacement with respect to a reference frame. Pairs of keypoints are subsequently used as control points to fit a geometric transformation in order to align the whole frame with the reference image. The stabilization method is applied to five years of daily images collected from a three-camera permanent video system located at Anglet Beach in southwestern France. Azimuth, tilt, and roll deviations are computed for each camera. The three cameras showed motions on a wide range of time scales, with a prominent annual signal in azimuth and tilt deviation. Camera movement amplitude reached up to 10 pixels in azimuth, 30 pixels in tilt, and 0.4° in roll, together with a quasi-steady counter-clockwise trend over the five-year time series. Moreover, camera viewing angle deviations were found to induce large rectification errors of up to 400 m at a distance of 2.5 km from the camera. The mean shoreline apparent position was also affected by an approximately 10–20 m bias during the 2013/2014 outstanding winter period. The stabilization semi-automatic method successfully corrects camera geometry for fixed video monitoring systems and is able to process at least 90% of the frames without user assistance. The use of the C E D greatly improves the performance of the cross-correlation algorithm by making it more robust against contrast and brightness variations between frames. The method appears as a promising tool for other coastal imaging applications such as removal of undesired high-frequency movements of cameras equipped in unmanned aerial vehicles (UAVs).