Feature Matching Research Articles

A reliable unmanned aerial vehicle multi-target tracking system with global motion compensation for monitoring Procapra przewalskii

Procapra przewalskii, which inhabits plateau areas, faces the constant threat of poaching and unpredictable risks that impede its survival. The implementation of a comprehensive, real-time monitoring and tracking system for Procapra przewalskii using artificial intelligence and unmanned aerial vehicle (UAV) technology is crucial to safeguard its existence. Therefore, a UAV multi-object-tracking (MOT) system with global motion compensation (GMC) was proposed in this study. YOLOv7 and Deep SORT were employed for object detection and tracking, respectively. Furthermore, the Kalman filter (KF) in Deep SORT is optimized to enhance the accuracy of object-tracking. Moreover, a novel appearance feature-extraction network (FEN) is introduced to enable more effective multi-scale feature (MSF) extraction. In addition, a GMC module was proposed to align neighboring frames through feature matching. This facilitates the correction of the position of the target in the subsequent frame, mitigating the impact of UAV camera motion on tracking. The results demonstrated the remarkable tracking accuracy of the system. Compared with the Deep SORT model, the proposed system exhibited an increase of 6.4% in MOTA, 2.7% in MOTP, and 7.9% in IDF1. Through a comprehensive evaluation and analysis of real-world tracking scenarios, the system proposed in this study exhibits reliability in complex scenes and holds the potential to significantly enhance the protection of Procapra przewalskii from threats.

DINA: Deformable INteraction Analogy

We introduce deformable interaction analogy (DINA) as a means to generate close interactions between two 3D objects. Given a single demo interaction between an anchor object (e.g. a hand) and a source object (e.g. a mug grasped by the hand), our goal is to generate many analogous 3D interactions between the same anchor object and various new target objects (e.g. a toy airplane), where the anchor object is allowed to be rigid or deformable. To this end, we optimize the pose or shape of the anchor object to adapt it to a new target object to mimic the demo. To facilitate the optimization, we advocate using interaction interface (ITF), defined by a set of points sampled on the anchor object, as a descriptive and robust interaction representation that is amenable to non-rigid deformation. We model similarity between interactions using ITF, while for interaction analogy, we transform the ITF, either rigidly or non-rigidly, to guide the feature matching to the reposing and deformation of the anchor object. Qualitative and quantitative experiments show that our ITF-guided deformable interaction analogy works surprisingly well even with simple distance features compared to variants of state-of-the-art methods that utilize more sophisticated interaction representations and feature learning from large datasets.

Knowledge distillation based on projector integration and classifier sharing

Knowledge distillation can transfer the knowledge from the pre-trained teacher model to the student model, thus effectively accomplishing model compression. Previous studies have carefully crafted knowledge representation, targeting loss function design, and distillation location selection, but there have been few studies on the role of classifiers in distillation. Previous experiences have shown that the final classifier of the model has an essential role in making inferences, so this paper attempts to narrow the gap in performance between models by having the student model directly use the classifier of the teacher model for the final inference, which requires an additional projector to help match features of the student encoder with the teacher's classifier. However, a single projector cannot fully align the features, and integrating multiple projectors may result in better performance. Considering the balance between projector size and performance, through experiments, we obtain the size of projectors for different network combinations and propose a simple method for projector integration. In this way, the student model undergoes feature projection and then uses the classifiers of the teacher model for inference, obtaining a similar performance to the teacher model. Through extensive experiments on the CIFAR-100 and Tiny-ImageNet datasets, we show that our approach applies to various teacher–student frameworks simply and effectively.

Amputation surgery associated with shortened survival in patients with localized extremity bone sarcoma

BackgroundThis study assesses survival rates among patients with localized extremity bone sarcoma who have undergone amputation, pinpointing subpopulations that are disproportionately affected by amputation-related survival disparities. MethodsExamination of data was conducted using the Surveillance, Epidemiology, and End Results (SEER) program, analyzing records of 3765 patients diagnosed with localized extremity bone sarcoma between 2000 and 2019. Of these, 857 received amputations (Amputation cohort), and 2908 underwent limb-sparing surgeries. Propensity score matching, considering demographic and clinical features, was utilized to ensure a fair comparison. ResultsFollowing propensity score matching, the study focused on 1714 cases. The Amputation cohort was observed to have significantly poorer survival rates (Cancer-Specific Survival [CSS]: Hazard Ratio [HR] = 1.28, 95% Confidence Interval [CI]: 1.05–1.55; Overall Survival [OS]: HR = 1.37, 95% CI: 1.15–1.63). Subsequent subgroup analysis indicated that individuals with tumors exceeding 8 cm in size or those located in the lower limbs were notably at a higher risk of shortened survival (for tumors >8 cm – CSS: HR = 1.32, 95% CI: 1.02–1.71; OS: HR = 1.39, 95% CI: 1.09–1.76; for lower limb tumors – CSS: HR = 1.25, 95% CI: 1.01–1.54; OS: HR = 1.33, 95% CI: 1.11–1.61). ConclusionsOur findings demonstrate that patients diagnosed with localized extremity bone sarcoma undergoing amputation exhibit lower survival rates, especially in cases involving tumors of greater size or those situated in the lower limbs. In patient groups where amputation is inevitable, careful follow-up is required after surgical intervention.

FPGA and computer-vision-based atom tracking technology for scanning probe microscopy

Atom tracking technology enhanced with innovative algorithms has been implemented in this study, utilizing a comprehensive suite of controllers and software independently developed domestically. Leveraging an on-board field-programmable gate array (FPGA) with a core frequency of 100 MHz, our system facilitates reading and writing operations across 16 channels, performing discrete incremental proportional-integral-derivative (PID) calculations within 3.4 microseconds. Building upon this foundation, gradient and extremum algorithms are further integrated, incorporating circular and spiral scanning modes with a horizontal movement accuracy of 0.38 pm. This integration enhances the real-time performance and significantly increases the accuracy of atom tracking. Atom tracking achieves an equivalent precision of at least 142 pm on a highly oriented pyrolytic graphite (HOPG) surface under room temperature atmospheric conditions. Through applying computer vision and image processing algorithms, atom tracking can be used when scanning a large area. The techniques primarily consist of two algorithms: the region of interest (ROI)-based feature matching algorithm, which achieves 97.92% accuracy, and the feature description-based matching algorithm, with an impressive 99.99% accuracy. Both implementation approaches have been tested for scanner drift measurements, and these technologies are scalable and applicable in various domains of scanning probe microscopy with broad application prospects in the field of nanoengineering.

Electrically Equivalent Head Tissue Materials for Electroencephalogram Study on Head Surrogates

With the recent advent of smart wearable sensors for monitoring brain activities in real-time, the scopes for using Electroencephalograms (EEGs) and Magnetoencephalography (MEG) in mobile and dynamic environments have become more relevant. However, their application in dynamic and open environments, typical of mobile wearable use, poses challenges. Presently, there is limited clinical data on using EEG/MEG as wearables. To advance these technologies at a time when large-scale clinical trials are not feasible, many researchers have turned to realistic phantom heads to further explore EEG and MEG capabilities. However, to achieve translational results, such phantom heads should have matching geometric features and electrical properties. Here, we have designed and fabricated multilayer chopped carbon fiber–PDMS reinforced composites to represent phantom head tissues. Two types of phantom layers are fabricated, namely seven-layer and four-layer systems with a goal to achieve matching electrical conductivities in each layer. Desired electrical conductivities are obtained by varying the weight fraction of the carbon fibers in PDMS. Then, the prototype system was calibrated and tested with a 32-electrode EEG cap. The test results demonstrated that the phantom effectively generates a variety of scalp potential patterns, achieved through a finite number of internal dipole generators within the phantom sample. This innovative design holds potential as a valuable test platform for assessing wearable EEG technology as well as developing an EEG analysis process.

Learning single and multi-scene camera pose regression with transformer encoders

Contemporary state-of-the-art localization methods perform feature matching against a structured scene model or learn to regress the scene 3D coordinates. The resulting matches between 2D query pixels and 3D scene coordinates are used to estimate the camera pose using PnP and RANSAC, requiring the camera intrinsics for both the query and reference images. An alternative approach is to directly regress the camera pose from the query image. Although less accurate, absolute camera pose regression does not require any additional information at inference time and is typically lightweight and fast. Recently, Transformers were proposed for learning multi-scene camera pose regression, employing encoders to attend to spatially varying deep features while using decoders to embed multiple scene queries at once. In this work, we show that Transformer Encoders can aggregate and extract task-informative latent representations for learning both single- and multi- scene camera pose regression, without Transformer-Decoders. Our approach is shown to reduce the runtime and memory of previous Transformer-based multi-scene solutions, while comparing favorably with contemporary pose regression schemes and achieving state-of-the-art accuracy on multiple indoor and outdoor regression benchmarks. In particular, to the best of our knowledge, our approach is the first absolute regression approach to attain sub-meter average accuracy across outdoor scenes. We make our code publicly available at: https://github.com/yolish/transposenet.

A Vascular Feature Detection and Matching Method Based on Dual-Branch Fusion and Structure Enhancement.

How to obtain internal cavity features and perform image matching is a great challenge for laparoscopic 3D reconstruction. This paper proposes a method for detecting and associating vascular features based on dual-branch weighted fusion vascular structure enhancement. Our proposed method is divided into three stages, including analyzing various types of minimally invasive surgery (MIS) images and designing a universal preprocessing framework to make our method generalized. We propose a Gaussian weighted fusion vascular structure enhancement algorithm using the dual-branch Frangi measure and MFAT (multiscale fractional anisotropic tensor) to address the structural measurement differences and uneven responses between venous vessels and microvessels, providing effective structural information for vascular feature extraction. We extract vascular features through dual-circle detection based on branch point characteristics, and introduce NMS (non-maximum suppression) to reduce feature point redundancy. We also calculate the ZSSD (zero sum of squared differences) and perform feature matching on the neighboring blocks of feature points extracted from the front and back frames. The experimental results show that the proposed method has an average accuracy and repeatability score of 0.7149 and 0.5612 in the Vivo data set, respectively. By evaluating the quantity, repeatability, and accuracy of feature detection, our method has more advantages and robustness than the existing methods.

Multi-view 3D reconstruction based on deep learning: A survey and comparison of methods

An important objective in computer vision is to analyze multiple images and subsequently reconstruct the shape and structure in 3D. Traditional multi-view 3D reconstruction techniques extract and match key features from images with known camera parameters. However, this approach is inefficient and fails to fully exploit the advantages of multi-view information. Advancements in deep learning have revolutionized multi-view 3D reconstruction by enabling end-to-end 3D shape inferencing without the need for sequential feature matching typically found in conventional algorithms. Recent rapid progress in this field necessitates a thorough review of current algorithms and provide insight into method of improving 3D reconstruction performance. This review classifies reconstruction algorithms according to their resultant model, including depth map, voxel, point cloud, mesh, and implicit surface. Additionally, this review encompasses the inclusion of frequently employed network training loss functions for network training, assessment metrics, and the incorporation of 3D datasets. Experimental results are also presented to assess the performance of different algorithms. Finally, the paper concludes with a summary, discussion of challenges, and potential future directions.

Accurate semantic segmentation of small-body craters for navigation

Feature extraction and matching play integral roles in autonomous vision navigation. As a typical morphological feature, craters can be used as navigation landmarks. However, the spin of small bodies introduces substantial variations in illumination and viewing angles within acquired images, thereby posing challenges to feature extraction and matching algorithms. Addressing these challenges, a deep learning-based approach emerges as pivotal in mitigating the limitations of existing navigation algorithms. This paper presents an algorithm designed to segment noteworthy craters and irregular regions on such small bodies, thereby furnishing crucial autonomous navigation data for missions involving uncharted terrains. By seamlessly integrating deep learning techniques with conventional computer vision-based feature description methods, the algorithm achieves matching segmentation results of noteworthy precision. The experimental results show that the algorithm has better results compared to the existing mainstream segmentation networks. And it can achieve the accurate matching of segmentation results.

Binocular Visual Measurement Method Based on Feature Matching.

To address the issues of low measurement accuracy and unstable results when using binocular cameras to detect objects with sparse surface textures, weak surface textures, occluded surfaces, low-contrast surfaces, and surfaces with intense lighting variations, a three-dimensional measurement method based on an improved feature matching algorithm is proposed. Initially, features are extracted from the left and right images obtained by the binocular camera. The extracted feature points serve as seed points, and a one-dimensional search space is established accurately based on the disparity continuity and epipolar constraints. The optimal search range and seed point quantity are obtained using the particle swarm optimization algorithm. The zero-mean normalized cross-correlation coefficient is employed as a similarity measure function for region growing. Subsequently, the left and right images are matched based on the grayscale information of the feature regions, and seed point matching is performed within each matching region. Finally, the obtained matching pairs are used to calculate the three-dimensional information of the target object using the triangulation formula. The proposed algorithm significantly enhances matching accuracy while reducing algorithm complexity. Experimental results on the Middlebury dataset show an average relative error of 0.75% and an average measurement time of 0.82 s. The error matching rate of the proposed image matching algorithm is 2.02%, and the PSNR is 34 dB. The algorithm improves the measurement accuracy for objects with sparse or weak textures, demonstrating robustness against brightness variations and noise interference.

Cross-Parallel Attention and Efficient Match Transformer for Aerial Tracking

Visual object tracking is a key technology that is used in unmanned aerial vehicles (UAVs) to achieve autonomous navigation. In recent years, with the rapid development of deep learning, tracking algorithms based on Siamese neural networks have received widespread attention. However, because of complex and diverse tracking scenarios, as well as limited computational resources, most existing tracking algorithms struggle to ensure real-time stable operation while improving tracking performance. Therefore, studying efficient and fast-tracking frameworks, and enhancing the ability of algorithms to respond to complex scenarios has become crucial. Therefore, this paper proposes a cross-parallel attention and efficient match transformer for aerial tracking (SiamEMT). Firstly, we carefully designed the cross-parallel attention mechanism to encode global feature information and to achieve cross-dimensional interaction and feature correlation aggregation via parallel branches, highlighting feature saliency and reducing global redundancy information, as well as improving the tracking algorithm’s ability to distinguish between targets and backgrounds. Meanwhile, we implemented an efficient match transformer to achieve feature matching. This network utilizes parallel, lightweight, multi-head attention mechanisms to pass template information to the search region features, better matching the global similarity between the template and search regions, and improving the algorithm’s ability to perceive target location and feature information. Experiments on multiple drone public benchmark tests verified the accuracy and robustness of the proposed tracker in drone tracking scenarios. In addition, on the embedded artificial intelligence (AI) platform AGX Xavier, our algorithm achieved real-time tracking speed, indicating that our algorithm can be effectively applied to UAV tracking scenarios.

Local feature matching using deep learning: A survey

Local feature matching enjoys wide-ranging applications in the realm of computer vision, encompassing domains such as image retrieval, 3D reconstruction, and object recognition. However, challenges persist in improving the accuracy and robustness of matching due to factors like viewpoint and lighting variations. In recent years, the introduction of deep learning models has sparked widespread exploration into local feature matching techniques. The objective of this endeavor is to furnish a comprehensive overview of local feature matching methods. These methods are categorized into two key segments based on the presence of detectors. The Detector-based category encompasses models inclusive of Detect-then-Describe, Joint Detection and Description, Describe-then-Detect, as well as Graph Based techniques. In contrast, the Detector-free category comprises CNN Based, Transformer Based, and Patch Based methods. Our study extends beyond methodological analysis, incorporating evaluations of prevalent datasets and metrics to facilitate a quantitative comparison of state-of-the-art techniques. The paper also explores the practical application of local feature matching in diverse domains such as Structure from Motion, Remote Sensing Image Registration, and Medical Image Registration, underscoring its versatility and significance across various fields. Ultimately, we endeavor to outline the current challenges faced in this domain and furnish future research directions, thereby serving as a reference for researchers involved in local feature matching and its interconnected domains. A comprehensive list of studies in this survey is available at https://github.com/vignywang/Awesome-Local-Feature-Matching.

Synthesising realistic 2D microstructures of unidirectional fibre-reinforced composites with a generative adversarial network

The microstructure governs the behaviour of unidirectional fibre-reinforced composites. In this study, we developed a Deep Convolutional Generative Adversarial Network (DCGAN) to generate realistic 2D transverse microstructures of such composites. We evaluated the DCGAN-generated microstructures using three different methods: Fréchet inception distance, walking through the latent space, and feature matching. The results from these evaluations confirmed that the generated microstructures are distinct and not simply a replication of the training data. The generated microstructures were then compared to real microstructures, confirming that they match qualitatively and quantitatively with respect to detailed statistical characteristics, including fibre diameters, fibre volume fraction, fibre spatial distribution, and resin-rich pockets. We also found that microstructures created by traditional generators could not capture the real resin-rich pockets. This illustrates the capability and value of DCGAN to generate realistic transverse microstructures and provides insights for modelling methods based on real images.

Predicting future velocity of mineral flotation froth using STMA-LSTM with sequence images

During the mineral flotation process, the surface froth image contains characteristic information that is closely related to the production index of the process. However, a delay often exists between the state of the flotation tank and the taste calculation results, which hinders real-time process regulation. To address this challenge, a video sequence prediction algorithm is proposed to forecast the future sequence of flotation froth, and a Spatiotemporal Motion-Attentive Long Short-Term Memory (STMA-LSTM) network model is introduced to estimate and predict the future image information of the froth by integrating spatiotemporal information. After obtaining the future froth image, an image velocity measurement algorithm is utilized to achieve the future velocity prediction of the flotation froth. To address the ambiguity of future froth images and the real-time demand of future velocity estimation, a Fast Froth Velocity Estimation (FFVE) algorithm is introduced. This algorithm is suitable for fast estimation of flotation froth velocity due to its low sensitivity to image blurring factors and fast feature matching. The proposed approach implements future motion velocity prediction of froth images, thereby addressing the delay in froth taste prediction and enhancing the overall efficiency of the process control in the flotation field.

A Semantically Aware Multi-View 3D Reconstruction Method for Urban Applications

The task of 3D reconstruction of urban targets holds pivotal importance for various applications, including autonomous driving, digital twin technology, and urban planning and development. The intricate nature of urban landscapes presents substantial challenges in attaining 3D reconstructions with high precision. In this paper, we propose a semantically aware multi-view 3D reconstruction method for urban applications which incorporates semantic information into the technical 3D reconstruction. Our research primarily focuses on two major components: sparse reconstruction and dense reconstruction. For the sparse reconstruction process, we present a semantic consistency-based error filtering approach for feature matching. To address the challenge of errors introduced by the presence of numerous dynamic objects in an urban scene, which affects the Structure-from-Motion (SfM) process, we propose a computation strategy based on dynamic–static separation to effectively eliminate mismatches. For the dense reconstruction process, we present a semantic-based Semi-Global Matching (sSGM) method. This method leverages semantic consistency to assess depth continuity, thereby enhancing the cost function during depth estimation. The improved sSGM method not only significantly enhances the accuracy of reconstructing the edges of the targets but also yields a dense point cloud containing semantic information. Through validation using architectural datasets, the proposed method was found to increase the reconstruction accuracy by 32.79% compared to the original SGM, and by 63.06% compared to the PatchMatch method. Therefore, the proposed reconstruction method holds significant potential in urban applications.

3D seismic Fault Detection via Contrastive-Reconstruction Representation Learning

Fault detection is a critical step in structural modeling and characterizing reservoirs. 3D seismic fault labeling is almost impossible to obtain, while networks trained on synthetic fault data have limited generalization to real data. We use self-supervised representation learning to enable networks to learn seismic field data features during pre-training, improving generalization. However, widely popular ViT/SwinViT-based methods cannot capture low-level fault features well. CNN-based have limited capability in transferring to downstream tasks. Tackle this, we designed the Tiny Self-Attention and extensively embedded it into the HRNet. It merges the advantages of convolutional and self-attention, allowing for in-depth learning of seismic representation information during the representation learning phase, thus achieving better inter-class separation in downstream tasks. We crafted two proxy tasks. The contrastive task aims to minimize the projected feature distance of overlapping areas from two distinct views. To tackle the issue of memory overflow and training suspension caused by the high spatial and temporal complexity of 3D high-resolution feature matching, we introduced sparse distance matching and an adaptive feature aggregation. In reconstruction task, we designed a masking strategy tailored to the unique attributes of 3D seismic data. It process named “Fault Detection via Contrast-Reconstruction Representation Learning” (FaultCRL). Experimentally, we compared the latest fault detection methods, and representation learning techniques like MAE and SwinUNETR, showcasing the notable advantage of FaultCRL in fault detection tasks. The appendix provides all qualitative results from mainstream geophysical journals regarding The Netherlands-F3 data in recent years, indicating that our approach has reached the current state-of-the-art level.

Incremental Template Neighborhood Matching for 3D anomaly detection

The unsupervised 3D industrial anomaly detection is gaining increasing attention in the computer vision community. Most advanced multimodal methods obtain anomaly detection models by one-time training, ignoring the continuous generation of new samples in industrial scenarios. In this paper, we propose Incremental Template Neighborhood Matching (ITNM), a method that enables the model to be incrementally updated by incorporating new samples, simultaneously achieving better anomaly detection and localization performance. We use a weighted feature concatenation to combine the RGB information with the point cloud information to avoid the bias caused by the difference of feature dimension and magnitude. In training stage, Pixel-wise Coreset Selection (PCS) is used to compress the multimodal template set to prevent excessive memory usage. PCS preserves the distribution of nominal sample features as much as possible, while retaining the pixel position information. In inference stage, the query feature is compared to the ones within the pixel’s neighborhood in the template set. The distance between the query feature and the most similar template feature is calculated as the anomaly score. Template Neighborhood Matching (TNM) avoids underestimation of anomaly scores due to misaligned template matching of anomaly features. Our method achieves competitive performance on the MVTec 3D-AD dataset, when a low false positive rate is required. More importantly, the results of the incremental training experiments show that the incremental updating process is effective.

Feature‐enhanced representation with transformers for multi‐view stereo

AbstractMost existing multi‐view stereo (MVS) methods fail to consider global context information in the stage of feature extraction and cost aggregation. As transformers have shown remarkable performance on various vision tasks due to their ability to perceive global contextual information, this paper proposes a transformer‐based feature enhancement network (TF‐MVSNet) to facilitate feature representation learning by combining local features (both 2D and 3D) with long‐range contextual information. To reduce memory consumption of feature matching, the cross‐attention mechanism is leveraged to efficiently construct 3D cost volumes under the epipolar constraint. Additionally, a colour‐guided network is designed to refine depth maps at a coarse stage, hence reducing incorrect depth predictions at a fine stage. Extensive experiments were performed on the DTU dataset and Tanks and Temples (T&T) benchmark and results are reported.

Sea ice drift vector extraction based on feature matching using CS-1 Images

The study emphasizes the crucial role of sea ice drift in Arctic climate research and the protection of human activities. Employing C-SAR/01 (CS-1) imagery as the primary data source, the study assesses the effectiveness of the ORB feature matching algorithm for extracting Arctic Sea ice drift vectors. Additionally, it investigates the spatial distribution variations in the drift vectors extracted from the HH and HV polarization channels. The accuracy of the extracted drift vectors is validated using manually extracted sea ice drift data. Experimental results reveal a significant disparity in the number of drift vectors extracted from CS-1 HV polarization images compared to HH polarization images. The sea ice drift vectors extracted using the Oriented fast and Roasted Bried (ORB) operator demonstrate an average velocity error of less than 0.27 cm/s (0.31 km/d) and an average direction error of less than 5.66° in HV polarization images.