Correspondence Matching Research Articles

Core sample consensus method for two-view correspondence matching

Core sample consensus method for two-view correspondence matching

Deep-Learning-Based Precision Visual Tracking

In this paper, we target the “precision visual tracking” problem, which is the precise tracking of a target point on a moving object. Compared with the abundant efforts in object-level tracking, which aims at accurately predicting the bounding box (or the mask) of the object, much less attention has been drawn to precision visual tracking. To this end, we present a template tracking framework that consists of two parts, template matching and template updating. For template matching, we trained a deep architecture to directly estimate the projective transformation that deforms the template to the search image. For template updating, we came up with a systematic strategy to update the initial and new templates. To avoid drift build-ups, we incorporate a fast-running dense correspondence matching module into the template update step. The proposed method was extensively tested on both synthetic and real data. To generate the synthetic data, we created a dataset of 480 image sequences. Each image sequence comes with the trajectory ground truth of a target point moving across all the frames. We compared the proposed method with six comparative approaches on this dataset. The tracking accuracy achieved by the proposed method was as follows: around 44% of the tracking errors were less than one pixel, about 78% of them were less than three pixels, and only about 14% of them exceeded five pixels. The proposed method is fast, running at 14 frames-per-second (fps) on [Formula: see text] image sequences on a workstation equipped with a Nvidia RTX 2080Ti graphic card. Qualitative results also show that the proposed method is applicable to real-world image sequences. The related code, pre-trained models, and the test data will be made publicly available at https://github.com/XM-Peng/Precision-Visual-Tracking/ .

KISS-ICP: In Defense of Point-to-Point ICP – Simple, Accurate, and Robust Registration If Done the Right Way

Robust and accurate pose estimation of a robotic platform, so-called sensor-based odometry, is an essential part of many robotic applications. While many sensor odometry systems made progress by adding more complexity to the ego-motion estimation process, we move in the opposite direction. By removing a majority of parts and focusing on the core elements, we obtain a surprisingly effective system that is simple to realize and can operate under various environmental conditions using different LiDAR sensors. Our odometry estimation approach relies on point-to-point ICP combined with adaptive thresholding for correspondence matching, a robust kernel, a simple but widely applicable motion compensation approach, and a point cloud subsampling strategy. This yields a system with only a few parameters that in most cases do not even have to be tuned to a specific LiDAR sensor. Our system performs on par with state-of-the-art methods under various operating conditions using different platforms using the same parameters: automotive platforms, UAV-based operation, vehicles like segways, or handheld LiDARs. We do not require integrating IMU data and solely rely on 3D point clouds obtained from a wide range of 3D LiDAR sensors, thus, enabling a broad spectrum of different applications and operating conditions. Our open-source system operates faster than the sensor frame rate in all presented datasets and is designed for real-world scenarios.

Easy, bias-free Bayesian hierarchical modeling of the psychometric function using the Palamedes Toolbox.

A hierarchical Bayesian method is proposed that can be used to fit multiple psychometric functions (PFs) simultaneously across conditions and subjects. The method incorporates the generalized linear model and allows easy reparameterization of the parameters of the PFs, for example, to constrain parameter values across conditions or to code for experimental effects (e.g., main effects and interactions in a factorial design). Simulations indicate that fitting PFs for multiple conditions and observers simultaneously using the hierarchical structure effectively eliminates bias and improves precision in parameter estimates relative to fitting PFs individually in each condition. The method is further validated by analyzing human psychophysical data obtained in an experiment investigating the effect of attention on correspondence matching in an ambiguous long-range motion display. The method converges successfully, even for experiments that use a low number of trials per subject, without the need for fine-tuning by the user and while using the default essentially uninformative priors. The latter may make the method more acceptable to those critical of applying informative priors. The method is implemented in the freely downloadable Palamedes Toolbox, which also includes routines that graphically display the fitted psychometric functions alongside the data, and derive and display posterior distributions of parameters, summary statistics, and diagnostic measures. Overall, these features make hierarchical Bayesian modeling of PFs easily available to researchers who wish to use Bayesian statistics but lack the expertise to implement these methods themselves.

Can a Tablet Game That Boosts Kindergarten Phonics Advance 1st Grade Reading?

We report the effectiveness of using the tablet-game Kalulu Phonics immediately after intervention in kindergarten and on national evaluations the year after. In a previous intervention testing the software with 1st graders, fluency and comprehension were boosted, but only when used in concert with reading instruction at the start of the year. Here, we asked whether a similar intervention would be more efficient if it started a year earlier, in kindergarten. Forty classes (1092 = children) were randomized into playing Kalulu Phonics or the matched Kalulu Numbers control game for the first half of the year, with reversed assignments in the second half. Ten non-randomized business-as-usual classes also participated. In a cross-over effect, children who used the phonics version improved in letter naming, grapheme-phoneme correspondence (GPC) matching and fluency. Students with the number version improved in number knowledge. In a longitudinal follow-up, all intervention participants maintained an advantage in phoneme awareness and GPC matching at the start of 1st grade, but this advantage failed to translate into school literacy gains by mid-year. No longitudinal benefits were found for numbers. Our results support using tablet-based aids in spurring early reading skills but question the possibility that a short-term intervention may address the challenges of long-term educational goals.

Calibrating a Three-Viewpoints Thermal Camera with Few Correspondences

This paper proposes the approach to calibration of a thermal camera that has three viewpoints. This work estimates the intrinsic parameters of the camera by establishing geometric relationship between real-world scene points and image points. This work estimates the extrinsic parameters, i.e., rotation and translation of the camera using vanishing points. The present work is different from the existed work on the calibration from three perspectives. The first one is that this work calibrates thermal camera with three viewpoints whose relative rotation is greater than 180-degree. The second one is that the viewpoints shares few overlapped areas due to the structure of the camera, so the proposed work is more challenging and difficult than the previous work on the calibration. The third one is that projected image of a scene requires quality enhancement for the calibration. Since few overlapped images are given a priori, feature matching cannot be used for the calibration. Instead of using correspondence matching, vanishing points are exploited to estimate relative rotations of cameras. The intrinsic parameters are estimated by capturing thermal-checkerboard followed by image processing. Experimental results show the accuracy and stability of the proposed approach for estimation of the intrinsic and the extrinsic parameters.

Low-cost structured light imaging of regional volume changes for use in assessing mechanical ventilation

BackgroundOptimal setting of mechanical ventilators is critical for improving outcomes. Accurate, predictive lung mechanics models are effective in optimizing MV settings, but only at a global level as they cannot estimate regional lung volume ventilation to assess the potential of local distension or under-ventilation. This study presents a low-cost structured light system for non-contact high resolution chest motion measurement to estimate regional lung volume changes. MethodsThe system consists of a structured light projector and two cameras. A new pattern is designed to extract motion from sub-regions of the chest surface, and an efficient feature is proposed to provide a fast and accurate correspondence matching between two views. Reconstruction of 3D surface points is based on the matched points and stereo method. Asymmetric distribution of tidal volume into left and right lungs is estimated based on reconstructed regional chest expansion. A proof-of-concept experiment using a dummy model and two test lungs connected to a ventilator to provide differential chest expansion is conducted under tidal volumes of 400 ml, 500 ml and 600 ml, with results compared to the widely-used SURF and ORB methods. ResultsCompared to the SURF and ORB methods, the proposed method is more computationally efficient with ∼40% less computational time cost, and higher accuracy for dense point correspondence. Finally, the proposed method estimated the region lung volumes with the maximum error of 8 ml under 600 ml tidal volume, indicating a good accuracy. ConclusionsSurface reconstruction results in a proof-of-concept experiment with differential chest expansion show good performance for the proposed pattern and method in extracting the key information for regional chest expansion. The proposed method is generalizable, with potential for use in other applications.

Feature-Based Correspondence Filtering Using Structural Similarity Index for Visual Odometry

The stereo correspondence problem is one of the most pre-eminent problems in a stereo vision system. With the right correspondence, a stereo vision system can help cap over diverse problems, while on the other hand, a wrong correspondence can be costly. While the performance of a feature-based correspondence approach is exceptional, the method can still produce wrong correspondences. This work presents an amalgam of feature-based and correlation-based correspondence, where the local pixels around a feature pair are compared using Structural SIMilarity index (SSIM), enhancing the correspondences, and a semantic-based filtering module, which further filters the obtained corresponding features using semantic data whenever detected in both the stereo image pair. While approaches in the literature are focused towards finding better features and their representation, the proposed approach advocates that correlation-based verification of the features can filter out bad correspondences, and in addition, aided by semantic-level filtering. These two modules establish the novelty of the work. The proposed correspondence matching algorithm is used to solve the problem of Visual Odometry to let a low-cost robot compute its pose in a novel environment. The experimental results show adequate filtering of wrong feature correspondence wherein, different environments with different lighting conditions were also considered. The proposed approach outperformed numerous state-of-the-art approaches available in the literature. The visual odometry algorithm using the proposed correspondence matching is compared against classical methods and a deep learning method, and it is observed that the proposed approach delivers lower trajectory error values in most scenarios on the KITTI dataset sequences.

Delaunay walk for fast nearest neighbor: accelerating correspondence matching for ICP

Point set registration algorithms such as Iterative Closest Point (ICP) are commonly utilized in time-constrained environments like robotics. Finding the nearest neighbor of a point in a reference 3D point set is a common operation in ICP and frequently consumes at least 90% of the computation time. We introduce a novel approach to performing the distance-based nearest neighbor step based on Delaunay triangulation. This greedy algorithm finds the nearest neighbor of a query point by traversing the edges of the Delaunay triangulation created from a reference 3D point set. Our work integrates the Delaunay traversal into the correspondences search of ICP and exploits the iterative aspect of ICP by caching previous correspondences to expedite each iteration. An algorithmic analysis and comparison is conducted showing an order of magnitude speedup for both serial and vector processor implementation.

Relative Pose Estimation for RGB-D Human Input Scans via Implicit Function Reconstruction

To achieve a promising performance on relative pose estimation for RGB-D scans, a considerable overlap between two RGB-D inputs is often required for most existing methods. However, in many practical applications for human scans, we often have to estimate the relative poses under arbitrary overlaps, which is challenging for existing methods. To deal with this problem, this paper presents a novel end-to-end and coarse-to-fine optimization method. Our method is self-supervision which firstly combines implicit function reconstruction with differentiable render for RGB-D human input scans at arbitrary overlaps in relative pose estimation. The insight is to take advantage of the underlying human geometry prior as much as possible. First of all, for stable coarse poses, we utilize the implicit function reconstruction to dig out abundant hidden cues from unseen regions in the initialization module. To further refine the poses, the differentiable render is leveraged to establish a self-supervision mechanism in the optimization module, which is independent of standard pipelines for feature extracting and accurate correspondence matching. More importantly, our proposed method is flexible to be extended to multiview input scans. The results and evaluations demonstrate that our optimization module is robust for real-world noisy inputs, and our approach outperforms considerably than standard pipelines in non-overlapping setups.

DSMA: Reference-Based Image Super-Resolution Method Based on Dual-View Supervised Learning and Multi-Attention Mechanism

Reference image based super-resolution methods (RefSR) have made rapid and remarkable progress in the field of image super-resolution (SR) in recent years by introducing additional high-resolution (HR) images to enhance the recovery of low-resolution (LR) images. The existing RefSR methods can rely on implicit correspondence matching to transfer the HR texture from the reference image (Ref) to compensate for the information loss in the input image. However, the differences between low-resolution input images and high-resolution reference images still affects the effective utilization of Ref images, so it is an important challenge to make full use of the information in Ref images to improve the SR performance. In this paper, we propose an image super-resolution method based on dual-view supervised learning and multi-attention mechanism (DSMA). It enhances the learning of important detail features of Ref images and weakens the interference of noisy information by introducing the multi-attention mechanism, while employing dual-view supervision to motivate the network to learn more accurate feature representations. Quantitative and qualitative experiments on these benchmarks, i.e., CUFED5, Urban100 and Manga109, show that DSMA outperforms the state-of-the-art baselines with significant improvements.

Reference-based Image and Video Super-Resolution via C2-Matching.

Reference-based Super-Resolution (Ref-SR) has recently emerged as a promising paradigm to enhance a low-resolution (LR) input image or video by introducing an additional high-resolution (HR) reference image. Existing Ref-SR methods mostly rely on implicit correspondence matching to borrow HR textures from reference images to compensate for the information loss in input images. However, performing local transfer is difficult because of two gaps between input and reference images: the transformation gap (e.g. scale and rotation) and the resolution gap (e.g. HR and LR). To tackle these challenges, we propose C2-Matching in this work, which performs explicit robust matching crossing transformation and resolution. 1) To bridge the transformation gap, we propose a contrastive correspondence network, which learns transformation-robust correspondences using augmented views of the input image. 2) To address the resolution gap, we adopt teacher-student correlation distillation, which distills knowledge from the easier HR-HR matching to guide the more ambiguous LR-HR matching. 3) Finally, we design a dynamic aggregation module to address the potential misalignment issue between input images and reference images. In addition, to faithfully evaluate the performance of Reference-based Image Super-Resolution (Ref Image SR) under a realistic setting, we contribute the Webly-Referenced SR (WR-SR) dataset, mimicking the practical usage scenario. We also extend C2-Matching to Reference-based Video Super-Resolution (Ref VSR) task, where an image taken in a similar scene serves as the HR reference image. Extensive experiments demonstrate that our proposed C2-Matching significantly outperforms state of the arts by up to 0.7dB on the standard CUFED5 benchmark and also boosts the performance of video super-resolution by incorporating the C2-Matching component into Video SR pipelines. Notably, C2-Matching also shows great generalizability on WR-SR dataset as well as robustness across large scale and rotation transformations. Codes and datasets are available at https://github.com/yumingj/C2-Matching.

Latent Space Semantic Supervision Based on Knowledge Distillation for Cross-Modal Retrieval.

As an important field in information retrieval, fine-grained cross-modal retrieval has received great attentions from researchers. Existing fine-grained cross-modal retrieval methods made several improvements in capturing the fine-grained interplay between vision and language, failing to consider the fine-grained correspondences between the features in the image latent space and the text latent space respectively, which may lead to inaccurate inference of intra-modal relations or false alignment of cross-modal information. Considering that object detection can get the fine-grained correspondences of image region features and the corresponding semantic features, this paper proposed a novel latent space semantic supervision model based on knowledge distillation (L3S-KD), which trains classifiers supervised by the fine-grained correspondences obtained from an object detection model by using knowledge distillation for image latent space fine-grained alignment, and by the labels of objects and attributes for text latent space fine-grained alignment. Compared with existing fine-grained correspondence matching methods, L3S-KD can learn more accurate semantic similarities for local fragments in image-text pairs. Extensive experiments on MS-COCO and Flickr30K datasets demonstrate that the L3S-KD model consistently outperforms state-of-the-art methods for image-text matching.

Active Stereo Method for 3D Endoscopes using Deep-layer GCN and Graph Representation with Proximity Information.

Techniques for 3D endoscopic systems have been widely studied for various reasons. Among them, active stereo based systems, in which structured-light patterns are projected to surfaces and endoscopic images of the pattern are analyzed to produce 3D depth images, are promising, because of robustness and simple system configurations. For those systems, finding correspondences between a projected pattern and an original pattern is an open problem. Recently, correspondence estimation by graph neural networks (GCN) using graph-based representation of the patterns were proposed for 3D endoscopic systems. One severe problem of the approach is that the graph matching by GCN is largely affected by the stability of the graph construction process using the detected patterns of a captured image. If the detected pattern is fragmented into small pieces, graph matching may fail and 3D shapes cannot be retrieved. In this paper, we propose a solution for those problems by applying deep-layered GCN and extended graph representations of the patterns, where proximity information is added. Experiments show that the proposed method outperformed the previous method in accuracies for correspondence matching for 3D reconstruction.

Self-Localization of Mobile Robots Using a Single Catadioptric Camera with Line Feature Extraction.

This paper presents a novel self-localization technique for mobile robots using a central catadioptric camera. A unified sphere model for the image projection is derived by the catadioptric camera calibration. The geometric property of the camera projection model is utilized to obtain the intersections of the vertical lines and ground plane in the scene. Different from the conventional stereo vision techniques, the feature points are projected onto a known planar surface, and the plane equation is used for depth computation. The 3D coordinates of the base points on the ground are calculated using the consecutive image frames. The derivation of motion trajectory is then carried out based on the computation of rotation and translation between the robot positions. We develop an algorithm for feature correspondence matching based on the invariability of the structure in the 3D space. The experimental results obtained using the real scene images have demonstrated the feasibility of the proposed method for mobile robot localization applications.

Projector undistortion for high-accuracy fringe projection profilometry

Nowadays, fringe projection profilometry is widely used in optical three-dimensional (3D) measurement. However, due to distortion, epipolar constraints are inaccurate for use in correspondence matching of a projector. This results in low accuracy of final 3D reconstruction. To address the issue, two simple and effective methods for undistorted correspondence matching are proposed in this paper. The first removes distortion from projector correspondences using double epipolar constraints; the second pre-distorts the fringe images for projection, thus the projected fringes become naturally undistorted. Comparative experiments are implemented, in a measurement volume of (50 × 30 × 8) mm. The results show that typical mean reconstruction errors of the two methods drop to 32.9% and 27.6% from that of an existing method based on single epipolar-constraint undistortion.

Brain-inspired model for early vocal learning and correspondence matching using free-energy optimization.

We propose a developmental model inspired by the cortico-basal system (CX-BG) for vocal learning in babies and for solving the correspondence mismatch problem they face when they hear unfamiliar voices, with different tones and pitches. This model is based on the neural architecture INFERNO standing for Iterative Free-Energy Optimization of Recurrent Neural Networks. Free-energy minimization is used for rapidly exploring, selecting and learning the optimal choices of actions to perform (eg sound production) in order to reproduce and control as accurately as possible the spike trains representing desired perceptions (eg sound categories). We detail in this paper the CX-BG system responsible for linking causally the sound and motor primitives at the order of a few milliseconds. Two experiments performed with a small and a large audio database show the capabilities of exploration, generalization and robustness to noise of our neural architecture in retrieving audio primitives during vocal learning and during acoustic matching with unheared voices (different genders and tones).

Quality assessment of digital speckle patterns for the single-shot speckle projection profilometry based on a visualised simulation platform

As a high-speed 3D surface imaging technique, the single-shot speckle projection profilometry (SPP) is widely applied to various fields, where the design of speckle pattern plays a critical role in the fundamental issue of correspondence matching. However, the assessment of speckle patterns in the real imaging system is a time consuming and tedious procedure and their performance might vary with the operators and testing environment. To solve this problem, a visualised simulation platform based on 3ds Max engine is proposed in the paper. Furthermore, the performance of digital speckle patterns (DSPs) on the SPP technique is comprehensively evaluated in terms of 3D reconstruction precision, algorithm efficiency and pattern uniqueness. Firstly, the influence of the noise levels of the projection and imaging processes is considered. Then, the 3D profile reconstruction using five types of speckle pattern is performed in a real experiment. Afterward, a comparative analysis with the optimal DSP is conducted. In addition, the effectiveness of the typical metric of the painted speckle pattern in the field of digital image correlation (DIC) is investigated; results demonstrate its insufficient capability in the SPP field. Moreover, the scaling model of speckle size, influence of speckle deformation and material surface are also discussed to guide the design of DSP. This study can potentially facilitate the selection of high-quality DSPs and the standardisation of the SPP technique. The virtual experiment platform executable file of 3ds Max and the recommended DSP are uploaded as reference.

Structural-Information-Based Robust Corner Point Extraction for Camera Calibration Under Lens Distortions and Compression Artifacts

Previous camera calibration methods often use a checkerboard to capture images and estimate the camera parameters from the correspondences between images and the checkerboard. The corner points in the checkerboard images are used as useful features for correspondence matching. Therefore, it is essential to precisely find the corner points in the checkerboard images. In many previous works, the corner points are extracted assuming that the checkerboard images are not distorted by its lens. Instead, image blurring and Gaussian noise on the images are usually considered, but other cases are not dealt with. However, the captured checkerboard images are often corrupted by lens distortions and compression artifacts, which leads to performance degradation of corner point extraction. Moreover, the corner points are extracted individually in the previous methods without considering their geometric relations. To better handle the corner point extraction problem under lens distortions, in our corner point extraction optimization, the distorted locations of the pixels on checkerboard images are corrected with the camera parameters, and the structural constraints for checkerboard image grids are then applied under the line-to-line mapping. Also, to robustly find the blurred edges between corner points due to JPEG compression, an edge surface model is newly proposed that models the transitions with over- and under-shoots around the blurred edges. Extensive experimental results show that our method significantly outperforms the state-of-the-art method with average 88.3% and 54.3% reduction in RMSE for corner point reprojection and camera parameter estimation, respectively under compression and lens distortions for synthetic and real data.

Depth Measurement Based on Stereo Vision With Integrated Camera Rotation

Measuring the distance is one of the main objectives for sensor development. In this article, we present a new sensor system design for depth recovery. It is based on stereopsis with the integrated rotation of an image sensor. Different from conventional stereo vision systems with multiple cameras, the proposed rotational stereo configuration is built upon a single imaging device. By directing its lateral motion with a circular constraint, the visual parallax is generated from multiple viewpoints. The image pairs captured with the integrated sensor rotation are used for the correspondence matching and disparity computation. Moreover, a variation of multiple-baseline stereo with multiple base-angle implementation is proposed. It provides a flexible baseline setting to improve the depth measurement accuracy. The geometric formulation and computational algorithm are derived and developed. In the experiments, a prototype range sensing device is constructed. The results obtained from the real scene environment have demonstrated the feasibility of the proposed depth measurement approach.