Feature Matching Research Articles

Optical Flow for Spike Camera with Hierarchical Spatial-Temporal Spike Fusion

As an emerging neuromorphic camera with an asynchronous working mechanism, spike camera shows good potential for high-speed vision tasks. Each pixel in spike camera accumulates photons persistently and fires a spike whenever the accumulation exceeds a threshold. Such high-frequency fine-granularity photon recording facilitates the analysis and recovery of dynamic scenes with high-speed motion. This paper considers the optical flow estimation problem for spike cameras. Due to the Poisson nature of incoming photons, the occurrence of spikes is random and fluctuating, making conventional image matching inefficient. We propose a Hierarchical Spatial-Temporal (HiST) fusion module for spike representation to pursue reliable feature matching and develop a robust optical flow network, dubbed as HiST-SFlow. The HiST extracts features at multiple moments and hierarchically fuses the spatial-temporal information. We also propose an intra-moment filtering module to further extract the feature and suppress the influence of randomness in spikes. A scene loss is proposed to ensure that this hierarchical representation recovers the essential visual information in the scene. Experimental results demonstrate that the proposed method achieves state-of-the-art performance compared with the existing methods. The source codes are available at https://github.com/ruizhao26/HiST-SFlow.

Dual-Perspective Knowledge Enrichment for Semi-supervised 3D Object Detection

Semi-supervised 3D object detection is a promising yet under-explored direction to reduce data annotation costs, especially for cluttered indoor scenes. A few prior works, such as SESS and 3DIoUMatch, attempt to solve this task by utilizing a teacher model to generate pseudo-labels for unlabeled samples. However, the availability of unlabeled samples in the 3D domain is relatively limited compared to its 2D counterpart due to the greater effort required to collect 3D data. Moreover, the loose consistency regularization in SESS and restricted pseudo-label selection strategy in 3DIoUMatch lead to either low-quality supervision or a limited amount of pseudo labels. To address these issues, we present a novel Dual-Perspective Knowledge Enrichment approach named DPKE for semi-supervised 3D object detection. Our DPKE enriches the knowledge of limited training data, particularly unlabeled data, from two perspectives: data-perspective and feature-perspective. Specifically, from the data-perspective, we propose a class-probabilistic data augmentation method that augments the input data with additional instances based on the varying distribution of class probabilities. Our DPKE achieves feature-perspective knowledge enrichment by designing a geometry-aware feature matching method that regularizes feature-level similarity between object proposals from the student and teacher models. Extensive experiments on the two benchmark datasets demonstrate that our DPKE achieves superior performance over existing state-of-the-art approaches under various label ratio conditions. The source code and models will be made available to the public.

Cross-Layer and Cross-Sample Feature Optimization Network for Few-Shot Fine-Grained Image Classification

Recently, a number of Few-Shot Fine-Grained Image Classification (FS-FGIC) methods have been proposed, but they primarily focus on better fine-grained feature extraction while overlooking two important issues. The first one is how to extract discriminative features for Fine-Grained Image Classification tasks while reducing trivial and non-generalizable sample level noise introduced in this procedure, to overcome the over-fitting problem under the setting of Few-Shot Learning. The second one is how to achieve satisfying feature matching between limited support and query samples with variable spatial positions and angles. To address these issues, we propose a novel Cross-layer and Cross-sample feature optimization Network for FS-FGIC, C2-Net for short. The proposed method consists of two main modules: Cross-Layer Feature Refinement (CLFR) module and Cross-Sample Feature Adjustment (CSFA) module. The CLFR module further refines the extracted features while integrating outputs from multiple layers to suppress sample-level feature noise interference. Additionally, the CSFA module addresses the feature mismatch between query and support samples through both channel activation and position matching operations. Extensive experiments have been conducted on five fine-grained benchmark datasets, and the results show that the C2-Net outperforms other state-of-the-art methods by a significant margin in most cases. Our code is available at: https://github.com/zenith0923/C2-Net.

Adaptive FSS: A Novel Few-Shot Segmentation Framework via Prototype Enhancement

The Few-Shot Segmentation (FSS) aims to accomplish the novel class segmentation task with a few annotated images. Current FSS research based on meta-learning focuses on designing a complex interaction mechanism between the query and support feature. However, unlike humans who can rapidly learn new things from limited samples, the existing approach relies solely on fixed feature matching to tackle new tasks, lacking adaptability. In this paper, we propose a novel framework based on the adapter mechanism, namely Adaptive FSS, which can efficiently adapt the existing FSS model to the novel classes. In detail, we design the Prototype Adaptive Module (PAM), which utilizes accurate category information provided by the support set to derive class prototypes, enhancing class-specific information in the multi-stage representation. In addition, our approach is compatible with diverse FSS methods with different backbones by simply inserting PAM between the layers of the encoder. Experiments demonstrate that our method effectively improves the performance of the FSS models (e.g., MSANet, HDMNet, FPTrans, and DCAMA) and achieves new state-of-the-art (SOTA) results (i.e., 72.4% and 79.1% mIoU on PASCAL-5i 1-shot and 5-shot settings, 52.7% and 60.0% mIoU on COCO-20i 1-shot and 5-shot settings). Our code is available at https://github.com/jingw193/AdaptiveFSS.

Addressing the generalization of 3D registration methods with a featureless baseline and an unbiased benchmark

Recent 3D registration methods are mostly learning-based that either find correspondences in feature space and match them, or directly estimate the registration transformation from the given point cloud features. Therefore, these feature-based methods have difficulties with generalizing onto point clouds that differ substantially from their training data. This issue is not so apparent because of the problematic benchmark definitions that cannot provide any in-depth analysis and contain a bias toward similar data. Therefore, we propose a methodology to create a 3D registration benchmark, given a point cloud dataset, that provides a more informative evaluation of a method w.r.t. other benchmarks. Using this methodology, we create a novel FAUST-partial (FP) benchmark, based on the FAUST dataset, with several difficulty levels. The FP benchmark addresses the limitations of the current benchmarks: lack of data and parameter range variability, and allows to evaluate the strengths and weaknesses of a 3D registration method w.r.t. a single registration parameter. Using the new FP benchmark, we provide a thorough analysis of the current state-of-the-art methods and observe that the current method still struggle to generalize onto severely different out-of-sample data. Therefore, we propose a simple featureless traditional 3D registration baseline method based on the weighted cross-correlation between two given point clouds. Our method achieves strong results on current benchmarking datasets, outperforming most deep learning methods. Our source code is available on github.com/DavidBoja/exhaustive-grid-search.

Ultradurable Embedded Physically Unclonable Functions.

Physically unclonable functions (PUFs) have attracted growing interest for anticounterfeiting and authentication applications. The practical applications require durable PUFs made of robust materials. This study reports a practical strategy to generate extremely robust PUFs by embedding random features onto a substrate. The chaotic and low-cost electrohydrodynamic deposition process generates random polymeric features over a negative-tone photoresist film. These polymer features function as a conformal photomask, which protects the underlying photoresist from UV light, thereby enabling the generation of randomly positioned holes. Dry plasma etching of the substrate and removal of the photoresist result in the transfer of random features to the underlying silicon substrate. The matching of binary keys and features via different algorithms facilitates authentication of features. The embedded PUFs exhibit extreme levels of thermal (∼1000 °C) and mechanical stability that exceed the state of the art. The strength of this strategy emerges from the PUF generation directly on the substrate of interest, with stability that approaches the intrinsic properties of the underlying material. Benefiting from the materials and processes widely used in the semiconductor industry, this strategy shows strong promise for anticounterfeiting and device security applications.

A reliable unmanned aerial vehicle multi-target tracking system with global motion compensation for monitoring Procapra przewalskii

Procapra przewalskii, which inhabits plateau areas, faces the constant threat of poaching and unpredictable risks that impede its survival. The implementation of a comprehensive, real-time monitoring and tracking system for Procapra przewalskii using artificial intelligence and unmanned aerial vehicle (UAV) technology is crucial to safeguard its existence. Therefore, a UAV multi-object-tracking (MOT) system with global motion compensation (GMC) was proposed in this study. YOLOv7 and Deep SORT were employed for object detection and tracking, respectively. Furthermore, the Kalman filter (KF) in Deep SORT is optimized to enhance the accuracy of object-tracking. Moreover, a novel appearance feature-extraction network (FEN) is introduced to enable more effective multi-scale feature (MSF) extraction. In addition, a GMC module was proposed to align neighboring frames through feature matching. This facilitates the correction of the position of the target in the subsequent frame, mitigating the impact of UAV camera motion on tracking. The results demonstrated the remarkable tracking accuracy of the system. Compared with the Deep SORT model, the proposed system exhibited an increase of 6.4% in MOTA, 2.7% in MOTP, and 7.9% in IDF1. Through a comprehensive evaluation and analysis of real-world tracking scenarios, the system proposed in this study exhibits reliability in complex scenes and holds the potential to significantly enhance the protection of Procapra przewalskii from threats.

DINA: Deformable INteraction Analogy

We introduce deformable interaction analogy (DINA) as a means to generate close interactions between two 3D objects. Given a single demo interaction between an anchor object (e.g. a hand) and a source object (e.g. a mug grasped by the hand), our goal is to generate many analogous 3D interactions between the same anchor object and various new target objects (e.g. a toy airplane), where the anchor object is allowed to be rigid or deformable. To this end, we optimize the pose or shape of the anchor object to adapt it to a new target object to mimic the demo. To facilitate the optimization, we advocate using interaction interface (ITF), defined by a set of points sampled on the anchor object, as a descriptive and robust interaction representation that is amenable to non-rigid deformation. We model similarity between interactions using ITF, while for interaction analogy, we transform the ITF, either rigidly or non-rigidly, to guide the feature matching to the reposing and deformation of the anchor object. Qualitative and quantitative experiments show that our ITF-guided deformable interaction analogy works surprisingly well even with simple distance features compared to variants of state-of-the-art methods that utilize more sophisticated interaction representations and feature learning from large datasets.

Knowledge distillation based on projector integration and classifier sharing

Knowledge distillation can transfer the knowledge from the pre-trained teacher model to the student model, thus effectively accomplishing model compression. Previous studies have carefully crafted knowledge representation, targeting loss function design, and distillation location selection, but there have been few studies on the role of classifiers in distillation. Previous experiences have shown that the final classifier of the model has an essential role in making inferences, so this paper attempts to narrow the gap in performance between models by having the student model directly use the classifier of the teacher model for the final inference, which requires an additional projector to help match features of the student encoder with the teacher's classifier. However, a single projector cannot fully align the features, and integrating multiple projectors may result in better performance. Considering the balance between projector size and performance, through experiments, we obtain the size of projectors for different network combinations and propose a simple method for projector integration. In this way, the student model undergoes feature projection and then uses the classifiers of the teacher model for inference, obtaining a similar performance to the teacher model. Through extensive experiments on the CIFAR-100 and Tiny-ImageNet datasets, we show that our approach applies to various teacher–student frameworks simply and effectively.

Amputation surgery associated with shortened survival in patients with localized extremity bone sarcoma

BackgroundThis study assesses survival rates among patients with localized extremity bone sarcoma who have undergone amputation, pinpointing subpopulations that are disproportionately affected by amputation-related survival disparities. MethodsExamination of data was conducted using the Surveillance, Epidemiology, and End Results (SEER) program, analyzing records of 3765 patients diagnosed with localized extremity bone sarcoma between 2000 and 2019. Of these, 857 received amputations (Amputation cohort), and 2908 underwent limb-sparing surgeries. Propensity score matching, considering demographic and clinical features, was utilized to ensure a fair comparison. ResultsFollowing propensity score matching, the study focused on 1714 cases. The Amputation cohort was observed to have significantly poorer survival rates (Cancer-Specific Survival [CSS]: Hazard Ratio [HR] = 1.28, 95% Confidence Interval [CI]: 1.05–1.55; Overall Survival [OS]: HR = 1.37, 95% CI: 1.15–1.63). Subsequent subgroup analysis indicated that individuals with tumors exceeding 8 cm in size or those located in the lower limbs were notably at a higher risk of shortened survival (for tumors >8 cm – CSS: HR = 1.32, 95% CI: 1.02–1.71; OS: HR = 1.39, 95% CI: 1.09–1.76; for lower limb tumors – CSS: HR = 1.25, 95% CI: 1.01–1.54; OS: HR = 1.33, 95% CI: 1.11–1.61). ConclusionsOur findings demonstrate that patients diagnosed with localized extremity bone sarcoma undergoing amputation exhibit lower survival rates, especially in cases involving tumors of greater size or those situated in the lower limbs. In patient groups where amputation is inevitable, careful follow-up is required after surgical intervention.

FPGA and computer-vision-based atom tracking technology for scanning probe microscopy

Atom tracking technology enhanced with innovative algorithms has been implemented in this study, utilizing a comprehensive suite of controllers and software independently developed domestically. Leveraging an on-board field-programmable gate array (FPGA) with a core frequency of 100 MHz, our system facilitates reading and writing operations across 16 channels, performing discrete incremental proportional-integral-derivative (PID) calculations within 3.4 microseconds. Building upon this foundation, gradient and extremum algorithms are further integrated, incorporating circular and spiral scanning modes with a horizontal movement accuracy of 0.38 pm. This integration enhances the real-time performance and significantly increases the accuracy of atom tracking. Atom tracking achieves an equivalent precision of at least 142 pm on a highly oriented pyrolytic graphite (HOPG) surface under room temperature atmospheric conditions. Through applying computer vision and image processing algorithms, atom tracking can be used when scanning a large area. The techniques primarily consist of two algorithms: the region of interest (ROI)-based feature matching algorithm, which achieves 97.92% accuracy, and the feature description-based matching algorithm, with an impressive 99.99% accuracy. Both implementation approaches have been tested for scanner drift measurements, and these technologies are scalable and applicable in various domains of scanning probe microscopy with broad application prospects in the field of nanoengineering.

Electrically Equivalent Head Tissue Materials for Electroencephalogram Study on Head Surrogates

With the recent advent of smart wearable sensors for monitoring brain activities in real-time, the scopes for using Electroencephalograms (EEGs) and Magnetoencephalography (MEG) in mobile and dynamic environments have become more relevant. However, their application in dynamic and open environments, typical of mobile wearable use, poses challenges. Presently, there is limited clinical data on using EEG/MEG as wearables. To advance these technologies at a time when large-scale clinical trials are not feasible, many researchers have turned to realistic phantom heads to further explore EEG and MEG capabilities. However, to achieve translational results, such phantom heads should have matching geometric features and electrical properties. Here, we have designed and fabricated multilayer chopped carbon fiber–PDMS reinforced composites to represent phantom head tissues. Two types of phantom layers are fabricated, namely seven-layer and four-layer systems with a goal to achieve matching electrical conductivities in each layer. Desired electrical conductivities are obtained by varying the weight fraction of the carbon fibers in PDMS. Then, the prototype system was calibrated and tested with a 32-electrode EEG cap. The test results demonstrated that the phantom effectively generates a variety of scalp potential patterns, achieved through a finite number of internal dipole generators within the phantom sample. This innovative design holds potential as a valuable test platform for assessing wearable EEG technology as well as developing an EEG analysis process.

Learning single and multi-scene camera pose regression with transformer encoders

Contemporary state-of-the-art localization methods perform feature matching against a structured scene model or learn to regress the scene 3D coordinates. The resulting matches between 2D query pixels and 3D scene coordinates are used to estimate the camera pose using PnP and RANSAC, requiring the camera intrinsics for both the query and reference images. An alternative approach is to directly regress the camera pose from the query image. Although less accurate, absolute camera pose regression does not require any additional information at inference time and is typically lightweight and fast. Recently, Transformers were proposed for learning multi-scene camera pose regression, employing encoders to attend to spatially varying deep features while using decoders to embed multiple scene queries at once. In this work, we show that Transformer Encoders can aggregate and extract task-informative latent representations for learning both single- and multi- scene camera pose regression, without Transformer-Decoders. Our approach is shown to reduce the runtime and memory of previous Transformer-based multi-scene solutions, while comparing favorably with contemporary pose regression schemes and achieving state-of-the-art accuracy on multiple indoor and outdoor regression benchmarks. In particular, to the best of our knowledge, our approach is the first absolute regression approach to attain sub-meter average accuracy across outdoor scenes. We make our code publicly available at: https://github.com/yolish/transposenet.

A Vascular Feature Detection and Matching Method Based on Dual-Branch Fusion and Structure Enhancement.

How to obtain internal cavity features and perform image matching is a great challenge for laparoscopic 3D reconstruction. This paper proposes a method for detecting and associating vascular features based on dual-branch weighted fusion vascular structure enhancement. Our proposed method is divided into three stages, including analyzing various types of minimally invasive surgery (MIS) images and designing a universal preprocessing framework to make our method generalized. We propose a Gaussian weighted fusion vascular structure enhancement algorithm using the dual-branch Frangi measure and MFAT (multiscale fractional anisotropic tensor) to address the structural measurement differences and uneven responses between venous vessels and microvessels, providing effective structural information for vascular feature extraction. We extract vascular features through dual-circle detection based on branch point characteristics, and introduce NMS (non-maximum suppression) to reduce feature point redundancy. We also calculate the ZSSD (zero sum of squared differences) and perform feature matching on the neighboring blocks of feature points extracted from the front and back frames. The experimental results show that the proposed method has an average accuracy and repeatability score of 0.7149 and 0.5612 in the Vivo data set, respectively. By evaluating the quantity, repeatability, and accuracy of feature detection, our method has more advantages and robustness than the existing methods.

Multi-view 3D reconstruction based on deep learning: A survey and comparison of methods

An important objective in computer vision is to analyze multiple images and subsequently reconstruct the shape and structure in 3D. Traditional multi-view 3D reconstruction techniques extract and match key features from images with known camera parameters. However, this approach is inefficient and fails to fully exploit the advantages of multi-view information. Advancements in deep learning have revolutionized multi-view 3D reconstruction by enabling end-to-end 3D shape inferencing without the need for sequential feature matching typically found in conventional algorithms. Recent rapid progress in this field necessitates a thorough review of current algorithms and provide insight into method of improving 3D reconstruction performance. This review classifies reconstruction algorithms according to their resultant model, including depth map, voxel, point cloud, mesh, and implicit surface. Additionally, this review encompasses the inclusion of frequently employed network training loss functions for network training, assessment metrics, and the incorporation of 3D datasets. Experimental results are also presented to assess the performance of different algorithms. Finally, the paper concludes with a summary, discussion of challenges, and potential future directions.

Accurate semantic segmentation of small-body craters for navigation

Feature extraction and matching play integral roles in autonomous vision navigation. As a typical morphological feature, craters can be used as navigation landmarks. However, the spin of small bodies introduces substantial variations in illumination and viewing angles within acquired images, thereby posing challenges to feature extraction and matching algorithms. Addressing these challenges, a deep learning-based approach emerges as pivotal in mitigating the limitations of existing navigation algorithms. This paper presents an algorithm designed to segment noteworthy craters and irregular regions on such small bodies, thereby furnishing crucial autonomous navigation data for missions involving uncharted terrains. By seamlessly integrating deep learning techniques with conventional computer vision-based feature description methods, the algorithm achieves matching segmentation results of noteworthy precision. The experimental results show that the algorithm has better results compared to the existing mainstream segmentation networks. And it can achieve the accurate matching of segmentation results.

Binocular Visual Measurement Method Based on Feature Matching.

To address the issues of low measurement accuracy and unstable results when using binocular cameras to detect objects with sparse surface textures, weak surface textures, occluded surfaces, low-contrast surfaces, and surfaces with intense lighting variations, a three-dimensional measurement method based on an improved feature matching algorithm is proposed. Initially, features are extracted from the left and right images obtained by the binocular camera. The extracted feature points serve as seed points, and a one-dimensional search space is established accurately based on the disparity continuity and epipolar constraints. The optimal search range and seed point quantity are obtained using the particle swarm optimization algorithm. The zero-mean normalized cross-correlation coefficient is employed as a similarity measure function for region growing. Subsequently, the left and right images are matched based on the grayscale information of the feature regions, and seed point matching is performed within each matching region. Finally, the obtained matching pairs are used to calculate the three-dimensional information of the target object using the triangulation formula. The proposed algorithm significantly enhances matching accuracy while reducing algorithm complexity. Experimental results on the Middlebury dataset show an average relative error of 0.75% and an average measurement time of 0.82 s. The error matching rate of the proposed image matching algorithm is 2.02%, and the PSNR is 34 dB. The algorithm improves the measurement accuracy for objects with sparse or weak textures, demonstrating robustness against brightness variations and noise interference.

Cross-Parallel Attention and Efficient Match Transformer for Aerial Tracking

Visual object tracking is a key technology that is used in unmanned aerial vehicles (UAVs) to achieve autonomous navigation. In recent years, with the rapid development of deep learning, tracking algorithms based on Siamese neural networks have received widespread attention. However, because of complex and diverse tracking scenarios, as well as limited computational resources, most existing tracking algorithms struggle to ensure real-time stable operation while improving tracking performance. Therefore, studying efficient and fast-tracking frameworks, and enhancing the ability of algorithms to respond to complex scenarios has become crucial. Therefore, this paper proposes a cross-parallel attention and efficient match transformer for aerial tracking (SiamEMT). Firstly, we carefully designed the cross-parallel attention mechanism to encode global feature information and to achieve cross-dimensional interaction and feature correlation aggregation via parallel branches, highlighting feature saliency and reducing global redundancy information, as well as improving the tracking algorithm’s ability to distinguish between targets and backgrounds. Meanwhile, we implemented an efficient match transformer to achieve feature matching. This network utilizes parallel, lightweight, multi-head attention mechanisms to pass template information to the search region features, better matching the global similarity between the template and search regions, and improving the algorithm’s ability to perceive target location and feature information. Experiments on multiple drone public benchmark tests verified the accuracy and robustness of the proposed tracker in drone tracking scenarios. In addition, on the embedded artificial intelligence (AI) platform AGX Xavier, our algorithm achieved real-time tracking speed, indicating that our algorithm can be effectively applied to UAV tracking scenarios.

Local feature matching using deep learning: A survey

Local feature matching enjoys wide-ranging applications in the realm of computer vision, encompassing domains such as image retrieval, 3D reconstruction, and object recognition. However, challenges persist in improving the accuracy and robustness of matching due to factors like viewpoint and lighting variations. In recent years, the introduction of deep learning models has sparked widespread exploration into local feature matching techniques. The objective of this endeavor is to furnish a comprehensive overview of local feature matching methods. These methods are categorized into two key segments based on the presence of detectors. The Detector-based category encompasses models inclusive of Detect-then-Describe, Joint Detection and Description, Describe-then-Detect, as well as Graph Based techniques. In contrast, the Detector-free category comprises CNN Based, Transformer Based, and Patch Based methods. Our study extends beyond methodological analysis, incorporating evaluations of prevalent datasets and metrics to facilitate a quantitative comparison of state-of-the-art techniques. The paper also explores the practical application of local feature matching in diverse domains such as Structure from Motion, Remote Sensing Image Registration, and Medical Image Registration, underscoring its versatility and significance across various fields. Ultimately, we endeavor to outline the current challenges faced in this domain and furnish future research directions, thereby serving as a reference for researchers involved in local feature matching and its interconnected domains. A comprehensive list of studies in this survey is available at https://github.com/vignywang/Awesome-Local-Feature-Matching.

Synthesising realistic 2D microstructures of unidirectional fibre-reinforced composites with a generative adversarial network

The microstructure governs the behaviour of unidirectional fibre-reinforced composites. In this study, we developed a Deep Convolutional Generative Adversarial Network (DCGAN) to generate realistic 2D transverse microstructures of such composites. We evaluated the DCGAN-generated microstructures using three different methods: Fréchet inception distance, walking through the latent space, and feature matching. The results from these evaluations confirmed that the generated microstructures are distinct and not simply a replication of the training data. The generated microstructures were then compared to real microstructures, confirming that they match qualitatively and quantitatively with respect to detailed statistical characteristics, including fibre diameters, fibre volume fraction, fibre spatial distribution, and resin-rich pockets. We also found that microstructures created by traditional generators could not capture the real resin-rich pockets. This illustrates the capability and value of DCGAN to generate realistic transverse microstructures and provides insights for modelling methods based on real images.