Correspondence Matching Research Articles

High-accuracy 3D reconstruction of step edge with ray aliasing based on projective parallel single-pixel imaging

Optical 3D measurement technology especially fringe projection profilometry (FPP) is widely utilized in industrial production. However, invalid mixed phases are common when measuring step edges in industrial parts which leads to outliers in reconstructed point cloud and ultimately causes reduction of valid point cloud and dimensional measurement errors. This paper first analyzes reasons for failure of step edge measurement in FPP and illustrates feasibility of parallel single-pixel imaging (PSI) applied to eliminate ray aliasing of step edge. Further for a good balance of accuracy and efficiency, we propose an optimized projective parallel single-pixel imaging (pPSI) method combing with edge detection in wrapped phase generated by pPSI patterns and adaptive bi-direction projection strategy to accurately identify direct correspondence matching points of step edge. Experimental results verified that the proposed method can achieve high accuracy of 3D reconstruction on step edge while taking less measurement time.

Multi-scale transformer network for super-resolution of visible and thermal air images

Reference image-based Super-Resolution (RefSR) is introduced to improve the quality of a Low-resolution (LR) input image by leveraging the additional information provided by a High-Resolution (HR) reference image (Ref). While existing RefSR methods focus on thermal or visible flows separately, they often struggle to enhance the resolution of small objects such as Mini/Micro UAVs (Unmanned Aerial Vehicle) due to the resolution disparities between the input and reference images. To cope with these challenges when dealing with UAV early detection in context of video surveillance, we propose ThermoVisSR, a multiscale texture transformer for enhancing the Super-Resolution (SR) of visible and thermal images of Mini/Micro UAVs. Our approach tries to reconstruct the fine details of these objects while preserving their approximation (the body form and color of the different scene objects) already contained in the LR image. Hence, our model is divided up into two streams dealing separately with approximation and detail reconstruction. In the first one, we introduce a Convolution Neural Network (CNN) fusion backbone to extract the Low-Frequency (LF) approximation from the original LR image pairs. In the second one and to extract the details from the Ref image, our approach involves blending features from both visible and thermal sources to make the most of what each offer. Subsequently, we introduce the High-Frequency Texture Transformer (HFTT) across various resolutions of the merged features to ensure an accurate correspondence matching and significant transfer of High-Frequency (HF) patches from Ref to LR images. Moreover, to adapt the injection to the different bands well, we incorporate the separable software decoder (SSD) into the HFTT allowing to capture channel-specific details during the reconstruction phase. We validated our approach using a newly created dataset of Air images of Mini/Micro UAVs. Experimental results demonstrate that the proposed model consistently outperforms the state-of-the-art approaches on both qualitative and quantitative assessments.

Effective descriptor extraction strategies for correspondence matching in coronary angiography images

The importance of 3D reconstruction of coronary arteries using multiple coronary angiography (CAG) images has been increasingly recognized in the field of cardiovascular disease management. This process relies on the camera matrix’s optimization, needing correspondence info for identical point positions across two images. Therefore, an automatic method for determining correspondence between two CAG images is highly desirable. Despite this need, there is a paucity of research focusing on image matching in the CAG images. Additionally, standard deep learning image matching techniques often degrade due to unique features and noise in CAG images. This study aims to fill this gap by applying a deep learning-based image matching method specifically tailored for the CAG images. We have improved the structure of our point detector and redesigned loss function to better handle sparse labeling and indistinct local features specific to CAG images. Our method include changes to training loss and introduction of a multi-head descriptor structure leading to an approximate 6% improvement. We anticipate that our work will provide valuable insights into adapting techniques from general domains to more specialized ones like medical imaging and serve as an improved benchmark for future endeavors in X-ray image-based correspondence matching.

Towards Robust Image Stitching: An Adaptive Resistance Learning against Compatible Attacks

Image stitching seamlessly integrates images captured from varying perspectives into a single wide field-of-view image. Such integration not only broadens the captured scene but also augments holistic perception in computer vision applications. Given a pair of captured images, subtle perturbations and distortions which go unnoticed by the human visual system tend to attack the correspondence matching, impairing the performance of image stitching algorithms. In light of this challenge, this paper presents the first attempt to improve the robustness of image stitching against adversarial attacks. Specifically, we introduce a stitching-oriented attack (SoA), tailored to amplify the alignment loss within overlapping regions, thereby targeting the feature matching procedure. To establish an attack resistant model, we delve into the robustness of stitching architecture and develop an adaptive adversarial training (AAT) to balance attack resistance with stitching precision. In this way, we relieve the gap between the routine adversarial training and benign models, ensuring resilience without quality compromise. Comprehensive evaluation across real-world and synthetic datasets validate the deterioration of SoA on stitching performance. Furthermore, AAT emerges as a more robust solution against adversarial perturbations, delivering superior stitching results. Code is available at: https://github.com/Jzy2017/TRIS.

FPRF: Feed-Forward Photorealistic Style Transfer of Large-Scale 3D Neural Radiance Fields

We present FPRF, a feed-forward photorealistic style transfer method for large-scale 3D neural radiance fields. FPRF stylizes large-scale 3D scenes with arbitrary, multiple style reference images without additional optimization while preserving multi-view appearance consistency. Prior arts required tedious per-style/-scene optimization and were limited to small-scale 3D scenes. FPRF efficiently stylizes large-scale 3D scenes by introducing a style-decomposed 3D neural radiance field, which inherits AdaIN’s feed-forward stylization machinery, supporting arbitrary style reference images. Furthermore, FPRF supports multi-reference stylization with the semantic correspondence matching and local AdaIN, which adds diverse user control for 3D scene styles. FPRF also preserves multi-view consistency by applying semantic matching and style transfer processes directly onto queried features in 3D space. In experiments, we demonstrate that FPRF achieves favorable photorealistic quality 3D scene stylization for large-scale scenes with diverse reference images.

Point cloud registration with quantile assignment

Point cloud registration is a fundamental problem in computer vision. The problem encompasses critical tasks such as feature estimation, correspondence matching, and transformation estimation. The point cloud registration problem can be cast as a quantile matching problem. We refined the quantile assignment algorithm by integrating prevalent feature descriptors and transformation estimation methods to enhance the correspondence between the source and target point clouds. We evaluated the performances of these descriptors and methods with our approach through controlled experiments on a dataset we constructed using well-known 3D models. This systematic investigation led us to identify the most suitable methods for complementing our approach. Subsequently, we devised a new end-to-end, coarse-to-fine pairwise point cloud registration framework. Finally, we tested our framework on indoor and outdoor benchmark datasets and compared our results with state-of-the-art point cloud registration methods.

Single-shot global localization via graph-theoretic correspondence matching

ABSTRACT This paper describes a method of single-shot global localization based on graph-theoretic matching of instances between a query and a prior map. The proposed framework employs correspondence matching based on the maximum clique problem (MCP). The framework is potentially applicable to other map and/or query modalities thanks to the graph-based abstraction of the problem, while many existing global localization methods rely on a query and the dataset in the same modality. We implement it with a semantically labeled 3D point cloud map, and a semantic segmentation image as a query. Leveraging the graph-theoretic framework, the proposed method realizes global localization exploiting only the map and the query. The method shows promising results on multiple large-scale simulated maps of urban scenes.

A low-cost close-range photogrammetric surface scanner

IntroductionA low-cost, close-range photogrammetric surface scanner is proposed, made from Computer Numerical Control (CNC) components and an off-the-shelf, consumer-grade macro camera.MethodsTo achieve micrometer resolution in reconstruction, accurate and photorealistic surface digitization, and retain low manufacturing cost, an image acquisition approach and a reconstruction method are proposed. The image acquisition approach uses the CNC to systematically move the camera and acquire images in a grid tessellation and at multiple distances from the target surface. A relatively large number of images is required to cover the scanned surface. The reconstruction method tracks keypoint features to robustify correspondence matching and uses far-range images to anchor the accumulation of errors across a large number of images utilized.Results and discussionQualitative and quantitative evaluation demonstrate the efficacy and accuracy of this approach.

Correction to: Core sample consensus method for two-view correspondence matching

Correction to: Core sample consensus method for two-view correspondence matching

Exploring Neighbor Correspondence Matching for Multiple-hypotheses Video Frame Synthesis

Video frame synthesis, which consists of interpolation and extrapolation , is an essential video processing technique that can be applied to various scenarios. However, most existing methods cannot handle small objects or large motion well, especially in high-resolution videos such as 4K videos. To eliminate such limitations, we introduce a neighbor correspondence matching (NCM) algorithm for flow-based frame synthesis. Since the current frame is not available in video frame synthesis, NCM is performed in a current-frame-agnostic fashion to establish multi-scale correspondences in the spatial-temporal neighborhoods of each pixel. Based on the powerful motion representation capability of NCM, we propose a heterogeneous coarse-to-fine scheme for intermediate flow estimation. The coarse-scale and fine-scale modules are trained progressively, making NCM computationally efficient and robust to large motions. We further explore the mechanism of NCM and find that neighbor correspondence is powerful, since it provides multiple-hypotheses motion information for synthesis. Based on this analysis, we introduce a multiple-hypotheses estimation process for video frame extrapolation, resulting in a more robust framework, NCM-MH. Experimental results show that NCM and NCM-MH achieve 31.63 and 28.08 dB for interpolation and extrapolation on the most challenging X4K1000FPS benchmark, outperforming all the other state-of-the-art methods that use two reference frames as input.

Using outlier elimination to assess learning-based correspondence matching methods

Recently, deep learning (DL) technology has been widely used in correspondence matching. The learning-based models are usually trained on benign image pairs with partial overlaps. Since DL model is usually data-dependent, non-overlapping images may be used as poison samples to fool the model and produce false registrations. In this study, we propose an outlier elimination-based assessment method (OEAM) to assess the registrations of learning-based correspondence matching method on partially overlapping and non-overlapping images. OEAM first eliminates outliers based on spatial paradox. Then OEAM implements registration assessment in two streams using the obtained core correspondence set. If the cardinality of the core set is sufficiently small, the input registration is assessed as a low-quality registration. Otherwise, it is assessed to be of high quality, and OEAM improves its registration performance using the core set. OEAM is a post-processing technique imposed on learning-based method. The comparison experiments are implemented on outdoor (YFCC100M) and indoor (SUN3D) datasets using four deep learning-based methods. The experimental results on registrations of partially overlapping images show that OEAM can reliably infer low-quality registrations and improve performance on high-quality registrations. The experiments on registrations of non-overlapping images demonstrate that learning-based methods are vulnerable to poisoning attacks launched by non-overlapping images, and OEAM is robust against poisoning attacks crafted by non-overlapping images.

Textureless Deformable Object Tracking with Invisible Markers.

Tracking and reconstructing deformable objects with little texture is challenging due to the lack of features. Here we introduce "invisible markers" for accurate and robust correspondence matching and tracking. Our markers are visible only under ultraviolet (UV) light. We build a novel imaging system for capturing videos of deformed objects under their original untouched appearance (which may have little texture) and, simultaneously, with our markers. We develop an algorithm that first establishes accurate correspondences using video frames with markers, and then transfers them to the untouched views as ground-truth labels. In this way, we are able to generate high-quality labeled data for training learning-based algorithms. We contribute a large real-world dataset, DOT, for tracking deformable objects with little or no texture. Our dataset has about one million video frames of various types of deformable objects. We provide ground truth tracked correspondences in both 2D and 3D. We benchmark state-of-the-art methods on optical flow and deformable object reconstruction using our dataset, which poses great challenges. By training on DOT, their performance significantly improves, not only on our dataset, but also on other unseen data.

COMPREHENSIVE POTHOLE DETECTION SYSTEM FOR ROAD MAINTENANCE AND SAFETY USING IMAGE PROCESSING AND STEREO VISION

This paper introduces a systematic approach to pothole detection, emphasizing its significance in road maintenance and safety. The proposed system is comprised of meticulously designed modules that collectively contribute to achieving accurate and effective results. Commencing with data collection through cameras capturing road areas of interest, the images undergo manual cropping and resizing for standardization. The core principle of the system revolves around image enhancement, involving grayscale conversion, application of blurring techniques, and adjustments to brightness and contrast. Automatic thresholding extracts essential information encoded within pixels, paving the way for precise edge detection refined through morphological operations. The focus then shifts to pothole detection, incorporating a stereo camera setup to calculate depth and disparity of road surfaces. Critical steps, including image rectification, correspondence matching, and disparity calculations, contribute to the accurate identification and delineation of potholes using depth thresholds. Subsequent modules employ K-Means clustering to segregate regions of interest from the image background, and post-processing steps, including filtering, masking, and refinement, fine-tune the images. Utilizing the HSV color space for grayscale image refinement and the connected component method to isolate white pixel objects further enhance the system's capability. The final step involves the addition of bounding boxes around identified potholes, streamlining their identification process. This comprehensive methodology ensures effective image processing, precise pothole detection, and contributes significantly to road safety and maintenance efforts.

A correspondence selection method based on same object and same position constraints

Establishing robust correspondences between two images is important for computer version tasks. However, in the real scene incorrect correspondences are inevitable no matter what kind of correspondence matching algorithms are adopted due to some complex factors, such as illumination, occlusion, and so on. To reduce the number of incorrect correspondences, an algorithm with the same object and same position constraints (SOSPC), is proposed to remove wrong correspondences from the given putative correspondences in this paper. The algorithm is based on the fact that in the given image pairs correct correspondences locate at the same position on the same objects. To select the correspondences on the same objects, an object matching method based on the correspondences selected by GMS is proposed. To select the correspondences on the correct positions, an iterative fundamental matrix estimation method based on clustering is presented. The experimental results have validated the effectiveness of the same object and the same position constraints, and the method achieves the state-of-art performance on five datasets.

Rotation-equivariant correspondence matching based on a dual-activation mixer

Learning-based correspondence matching methods have become the mainstream techniques in many computer vision and robotics applications due to their robustness to large illumination and viewpoint changes. However, it is difficult for conventional convolutional neural networks (CNNs) to extract rotation-equivariant local features. Recent work has shown that CNNs combined with group-equivariant architectures are surprisingly effective at matching correspondences even when the images are rotated to a dramatic extent. However, the inherent shape (square) of convolution kernels causes the performance bottleneck of such rotation-equivariant CNNs. To address this issue, we propose an adaptive dual rotation-equivariant correspondence matching algorithm, which performs stably at all angles. We mathematically analyze the effectiveness of our proposed rotation-equivariant correspondence matching approach and its performance with respect to different convolution kernels. Extensive experiments on the Rotated-HPatches, SIM2E, and MegaDepth datasets demonstrate the effectiveness of our proposed algorithm.

Unifying Flow, Stereo and Depth Estimation.

We present a unified formulation and model for three motion and 3D perception tasks: optical flow, rectified stereo matching and unrectified stereo depth estimation from posed images. Unlike previous specialized architectures for each specific task, we formulate all three tasks as a unified dense correspondence matching problem, which can be solved with a single model by directly comparing feature similarities. Such a formulation calls for discriminative feature representations, which we achieve using a Transformer, in particular the cross-attention mechanism. We demonstrate that cross-attention enables integration of knowledge from another image via cross-view interactions, which greatly improves the quality of the extracted features. Our unified model naturally enables cross-task transfer since the model architecture and parameters are shared across tasks. We outperform RAFT with our unified model on the challenging Sintel dataset, and our final model that uses a few additional task-specific refinement steps outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow, stereo and depth datasets, while being simpler and more efficient in terms of model design and inference speed.

A point cloud registration method based on multiple-local-feature matching

A dual-view point cloud registration method has been proposed to enhance the accuracy of point-cloud registration by utilizing multiple local feature descriptors. The method focuses on studying the eigenvalues of the point cloud set at various scales and the impact of descriptors that consist of corresponding curvature radii in reducing misregistration in fast point feature histogram (FPFH). The optimal scale size and number were determined through experimental analysis. SIFT-3D key points of point clouds were extracted to calculate their FPFH and complete initial correspondence matching. Next, a multi-neighborhood local feature descriptor was designed to filter correspondence. A large number of incorrect corresponding point pairs were removed after the initial FPFH matching, and the remaining correct point pairs were used for rough registration. Finally, the improved ICP algorithm was used to reduce registration errors and achieve more accurate dual-view registration. According to the experiments of the public point cloud data set, the dual-view point cloud registration algorithm implemented by the method has good accuracy, high robustness, and more reliable point cloud registration results, which provides a theoretical basis for 3D point-cloud reconstruction.

E3CM: Epipolar-constrained cascade correspondence matching

Accurate and robust correspondence matching is of utmost importance for various 3D computer vision tasks. However, traditional explicit programming-based methods often struggle to handle challenging scenarios, and deep learning-based methods require large well-labeled datasets for network training. In this article, we introduce Epipolar-Constrained Cascade Correspondence (E3CM), a novel approach that addresses these limitations. Unlike traditional methods, E3CM leverages pre-trained convolutional neural networks to match correspondence, without requiring annotated data for any network training or fine-tuning. Our method utilizes epipolar constraints to guide the matching process and incorporates a cascade structure for progressive refinement of matches. We extensively evaluate the performance of E3CM through comprehensive experiments and demonstrate its superiority over existing methods. To promote further research and facilitate reproducibility, we make our source code publicly available at https://mias.group/E3CM/.

SACF-Net: Skip-Attention Based Correspondence Filtering Network for Point Cloud Registration

Rigid registration is a transformation estimation problem between two point clouds. The two point clouds captured may partially overlap owing to different viewpoints and acquisition times. Some previous correspondence matching based methods utilize an encoder-decoder network to carry out partial-to-partial registration task and adopt a skip-connection structure to convey information between the encoder and decoder. However, equally revisiting them with skip-connection may introduce the information redundancy, and limit the feature learning ability of the entire network. To address these problems, we propose a skip-attention based correspondence filtering network ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SACF-Net</i> ) for point cloud registration. A novel feature interaction mechanism is designed to utilize both low-level geometric information and high-level context-aware information to enhance the original pointwise matching map. Additionally, a skip-attention based correspondence filtering method is proposed to selectively revisits features in the encoder at different resolutions, allowing the decoder to extract high-quality correspondences within overlapping regions. We conduct comprehensive experiments on indoor and outdoor scene datasets, and the results show that the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SACF-Net</i> yields unprecedented performance improvements.

De-redundancy in wireless capsule endoscopy video sequences using correspondence matching and motion analysis

De-redundancy in wireless capsule endoscopy video sequences using correspondence matching and motion analysis