Feature Correspondences Research Articles

Align While Fusion: A Generalized Nonaligned Multiview Multilabel Classification Method.

In the task of multiview multilabel (MVML) classification, each object is described by several heterogeneous view features and annotated with multiple relevant labels. Existing MVML methods usually assume that these heterogeneous features are strictly view-aligned, and they directly conduct cross-view information fusion to train a multilabel prediction model. However, in real-world scenarios, such strict view-aligned requirement can be hardly satisfied due to the recurrent spatiotemporal asynchronism when collecting MVML data, which would cause inaccurate multiview fusion results and degrade the classification performance. To address this issue, we propose a generalized nonaligned MVML (GNAM) classification method, which achieves multiview information fusion while aligning cross-view features and accordingly learns a desired multilabel classifier. Specifically, we first introduce a multiorder matching alignment strategy to achieve cross-view feature alignments, where both first-order feature correspondence and second-order structure correspondence are jointly integrated to guarantee the compactness of the view-alignment results. Afterward, a commonality-and individuality-based multiview fusion structure is formulated on the aligned-view features to excavate the consistencies and complementarities across different views, which leads all relevant multiview semantic labels, especially rare labels, to be characterized more comprehensively. Finally, we embed adaptive global label correlations to multilabel classification model to further enhance its semantic expression integrity and develop an alternative algorithm to optimize the whole model. Extensive experimental results have verified that GNAM is significantly superior to other state-of-the-art methods.

TriCI: Triple Cross-Intra Branch Contrastive Learning for Point Cloud Analysis.

Whereas contrastive learning eliminates the need for labeled data, existing methods may suffer from inadequate features due to the conventional single shared encoder structure and struggle to fully harness the rich spectrum of 3D augmentations. In this paper, we propose TriCI, a self-supervised method that designs a triple-branch contrastive learning architecture. During contrastive pre-training, we generate three augmented versions of each input point cloud sample and pair each augmented sample with the original one, resulting in three unique positive pairs. We subsequently feed the pairs into three distinct encoders, each of which extracts features from its corresponding input positive pair. We design a novel cross-branch contrastive loss and use it along with the intra-branch contrastive loss to jointly train our network. The proposed cross-branch loss effectively aligns the output features from different perspectives for pre-training and facilitates their integration for downstream tasks, particularly in object-level scenarios. The intra-branch loss helps maximize the feature correspondences within positive pairs. Extensive experiments demonstrate the superiority of our TriCI in self-supervised learning, and show its strong ability in enhancing the performance of downstream object classification and part segmentation tasks. Interestingly, our TriCI achieves a 92.9% accuracy for linear SVM evaluation on ModelNet40, exceeding its closest competitor by 1.7% and even exceeding some supervised methods.

Multimodel fore-/background alignment for seam-based parallax-tolerant image stitching

Image stitching with large parallax is a challenging computer vision problem. Although existing seam-based approaches were proposed to achieve pleasing results, issues like object dislocation, disappearance, and duplication can still occur. In this paper, to alleviate these problems, we propose a novel seam-based parallax-tolerant image stitching method, which relies on accurately aligning background and foreground regions using multiple warping models. To estimate various spatially smooth models based on feature correspondences from depth-varying objects, we introduce an iterative algorithm that selects inliers and solves the mesh warping model by assigning weights to data. Additionally, we construct matching confidences of foreground pixels based on selecting and grouping unaligned feature pairs, thus penalizing the duplication of seam cuts. To further improve alignment, we refine the models by minimizing pixel-level errors. We then choose the best seam among multiple candidate alignment and seam finding solutions. Finally, we re-estimate the warping model by sampling and weighting points near the seam to achieve a natural-looking stitching result. Experimental results on real-world images demonstrate the effectiveness and superiority of our proposed method over other state-of-the-arts.

SMILE: Siamese Multi-scale Interactive-representation LEarning for Hierarchical Diffeomorphic Deformable image registration

Deformable medical image registration plays an important role in many clinical applications. It aims to find a dense deformation field to establish point-wise correspondences between a pair of fixed and moving images. Recently, unsupervised deep learning-based registration methods have drawn more and more attention because of fast inference at testing stage. Despite remarkable progress, existing deep learning-based methods suffer from several limitations including: (a) they often overlook the explicit modeling of feature correspondences due to limited receptive fields; (b) the performance on image pairs with large spatial displacements is still limited since the dense deformation field is regressed from features learned by local convolutions; and (c) desirable properties, including topology-preservation and the invertibility of transformation, are often ignored. To address above limitations, we propose a novel Convolutional Neural Network (CNN) consisting of a Siamese Multi-scale Interactive-representation LEarning (SMILE) encoder and a Hierarchical Diffeomorphic Deformation (HDD) decoder. Specifically, the SMILE encoder aims for effective feature representation learning and spatial correspondence establishing while the HDD decoder seeks to regress the dense deformation field in a coarse-to-fine manner. We additionally propose a novel Local Invertible Loss (LIL) to encourage topology-preservation and local invertibility of the regressed transformation while keeping high registration accuracy. Extensive experiments conducted on two publicly available brain image datasets demonstrate the superiority of our method over the state-of-the-art (SOTA) approaches. Specifically, on the Neurite-OASIS dataset, our method achieved an average DSC of 0.815 and an average ASSD of 0.633 mm.

G-Aligner: a graph-based feature alignment method for untargeted LC–MS-based metabolomics

BackgroundLiquid chromatography–mass spectrometry is widely used in untargeted metabolomics for composition profiling. In multi-run analysis scenarios, features of each run are aligned into consensus features by feature alignment algorithms to observe the intensity variations across runs. However, most of the existing feature alignment methods focus more on accurate retention time correction, while underestimating the importance of feature matching. None of the existing methods can comprehensively consider feature correspondences among all runs and achieve optimal matching.ResultsTo comprehensively analyze feature correspondences among runs, we propose G-Aligner, a graph-based feature alignment method for untargeted LC–MS data. In the feature matching stage, G-Aligner treats features and potential correspondences as nodes and edges in a multipartite graph, considers the multi-run feature matching problem an unbalanced multidimensional assignment problem, and provides three combinatorial optimization algorithms to find optimal matching solutions. In comparison with the feature alignment methods in OpenMS, MZmine2 and XCMS on three public metabolomics benchmark datasets, G-Aligner achieved the best feature alignment performance on all the three datasets with up to 9.8% and 26.6% increase in accurately aligned features and analytes, and helped all comparison software obtain more accurate results on their self-extracted features by integrating G-Aligner to their analysis workflow. G-Aligner is open-source and freely available at https://github.com/CSi-Studio/G-Aligner under a permissive license. Benchmark datasets, manual annotation results, evaluation methods and results are available at https://doi.org/10.5281/zenodo.8313034ConclusionsIn this study, we proposed G-Aligner to improve feature matching accuracy for untargeted metabolomics LC–MS data. G-Aligner comprehensively considered potential feature correspondences between all runs, converting the feature matching problem as a multidimensional assignment problem (MAP). In evaluations on three public metabolomics benchmark datasets, G-Aligner achieved the highest alignment accuracy on manual annotated and popular software extracted features, proving the effectiveness and robustness of the algorithm.

AA-LMM: Robust Accuracy-Aware Linear Mixture Model for Remote Sensing Image Registration

Remote sensing image registration has been widely applied in military and civilian fields, such as target recognition, visual navigation and change detection. The dynamic changes in the sensing environment and sensors bring differences to feature point detection in amount and quality, which is still a common and intractable challenge for feature-based registration approaches. With such multiple perturbations, the extracted feature points representing the same physical location in space may have different location accuracy. Most existing matching methods focus on recovering the optimal feature correspondences while they ignore the diversities of different points in position, which easily brings the model into a bad local extrema, especially when existing with the outliers and noises. In this paper, we present a novel accuracy-aware registration model for remote sensing. A soft weighting is designed for each sample to preferentially select more reliable sample points. To better estimate the transformation between input images, an optimal sparse approximation is applied to approach the transformation by multiple iterations, which effectively reduces the computation complexity and also improves the accuracy of approximation. Experimental results show that the proposed method outperforms the state-of-the-art approaches in both matching accuracy and correct matches.

Abstract 14804: Automated Semantic Segmentation and Stenosis Detection of Coronary Arteries on Invasive Coronary Angiograms Using Hypergraph Matching and Information Fusion

Background: Invasive coronary angiography (ICA) remains the gold standard for diagnosing CAD. Our objective was to develop a machine learning method for coronary semantic segmentation and stenosis detection in ICA. Method: We first extract coronary artery branches using ICAs and convert the vascular tree into ICA graphs. Then, we construct the hyper association graph from two individual ICA graphs. The vertex in the hyper association graph encodes the correspondence between two artery branches, and each hyperedge represents the correspondence between multiple artery branches. Hypergraph matching is used to find feature correspondence by considering higher-order structural information, and the artery semantic labeling task is converted into a vertex classification. Ultimately, by learning the semantic mapping of artery branches between ICAs, the unlabeled arteries are categorized by labeled arteries so that semantic segmentation is achieved. After performing the semantic labeling, a deterministic network performs stenosis detection. More specifically, a convolution long-short-term network is employed to extract imaging features, and an LSTM is employed to extract the radius features along the artery. The late fusion technology is employed to fuse the feature representation, and a softmax classifier is employed to detect the stenosis. Results: 310 ICAs from 179 subjects were collected and utilized for data analysis. Our graph-matching model achieved an average accuracy of 0.86 for coronary artery semantic labeling. The deterministic network achieved an accuracy of 0.80 for coronary artery stenosis detection. Conclusion: We have developed and validated a hypergraph-based graph-matching algorithm for coronary semantic labeling and a fusion network for artery stenosis detection. This has important implications for integrating AI software in ICA, potentially resulting in automated report generation or as an assistant for clinical decision-making.

Multi-modal person re-identification based on transformer relational regularization

For robust multi-modal person re-identification (re-ID) models, it is crucial to effectively utilize the complementary information and constraint relationships among different modalities. However, current multi-modal methods often overlook the correlation between modalities at the feature fusion stage. To address this issue, we propose a novel multimodal person re-ID method called Transformer Relation Regularization (TRR). Firstly, we introduce an adaptive collaborative matching module that facilitates the exchange of useful information by mining feature correspondences between modalities. This module allows for the integration of complementary information, enhancing the re-ID performance. Secondly, we propose an enhanced embedded module that corrects general information using discriminative information within each modality. By leveraging this approach, we improve the model’s stability in challenging multi-modal environments. Lastly, we propose an adaptive triple loss to enhance sample utilization efficiency and mitigate the problem of inconsistent representation among multimodal samples. This loss function optimizes the model’s ability to distinguish between different individuals, leading to improved re-ID accuracy. Experimental results on several challenging visible-infrared person re-ID benchmark datasets demonstrate that our proposed TRR method achieves optimal performance. Additionally, extensive ablation studies validate the effective contribution of each component to the overall model. In summary, our proposed TRR method effectively leverages complementary information, addresses the correlation between modalities, and improves the re-ID performance in multi-modal scenarios. The results obtained from various benchmark datasets and the comprehensive analysis support the efficacy of our approach.

Surgical navigation for guidewire placement from intraoperative fluoroscopy in orthopaedic surgery.

Objective. Surgical guidewires are commonly used in placing fixation implants to stabilize fractures. Accurate positioning of these instruments is challenged by difficulties in 3D reckoning from 2D fluoroscopy. This work aims to enhance the accuracy and reduce exposure times by providing 3D navigation for guidewire placement from as little as two fluoroscopic images.Approach. Our approach combines machine learning-based segmentation with the geometric model of the imager to determine the 3D poses of guidewires. Instrument tips are encoded as individual keypoints, and the segmentation masks are processed to estimate the trajectory. Correspondence between detections in multiple views is established using the pre-calibrated system geometry, and the corresponding features are backprojected to obtain the 3D pose. Guidewire 3D directions were computed using both an analytical and an optimization-based method. The complete approach was evaluated in cadaveric specimens with respect to potential confounding effects from the imaging geometry and radiographic scene clutter due to other instruments.Main results. The detection network identified the guidewire tips within 2.2 mm and guidewire directions within 1.1°, in 2D detector coordinates. Feature correspondence rejected false detections, particularly in images with other instruments, to achieve 83% precision and 90% recall. Estimating the 3D direction via numerical optimization showed added robustness to guidewires aligned with the gantry rotation plane. Guidewire tips and directions were localized in 3D world coordinates with a median accuracy of 1.8 mm and 2.7°, respectively.Significance. The paper reports a new method for automatic 2D detection and 3D localization of guidewires from pairs of fluoroscopic images. Localized guidewires can be virtually overlaid on the patient's pre-operative 3D scan during the intervention. Accurate pose determination for multiple guidewires from two images offers to reduce radiation dose by minimizing the need for repeated imaging and provides quantitative feedback prior to implant placement.

SAKS: Sampling Adaptive Kernels From Subspace for Point Cloud Graph Convolution

Convolution on 3D point clouds has been extensively explored in geometric deep learning, but it is far from perfect. Convolution operations on point clouds with the fixed kernel indistinguishably capture correspondences between feature pairs, thereby raising an inherent drawback of limited distinctive feature learning. This paper proposes a novel approach to Sampling Adaptive Kernels from Subspace (SAKS) for graph convolution. It adaptively constructs convolution kernels for different feature correspondences according to the unique coordinate representations under the learned subspace. Associating the subspace design with the deep network is a novel concept, providing different viewpoints on feature learning. Specifically, incomplete orthogonal bases are learned at each convolution layer to span a linear subspace in an elaborately designed manner. Subsequently, adaptive kernels are sampled from the learned subspace via unique coordinates parameterized by feature pairs. Unlike existing adaptive convolution methods in a bruteforce manner, the low-rank property of the subspace reduces the computational complexity of this method. Moreover, we theoretically prove that the proposed SAKS derives the principal components of the kernel distribution, which is similar to principal component analysis under some prior assumptions. Extensive experimental results on point cloud classification and segmentation tasks show that SAKS outperforms state-of-the-arts on various benchmark datasets.

Molecular Joint Representation Learning via Multi-modal Information of SMILES and Graphs.

In recent years, artificial intelligence has played an important role on accelerating the whole process of drug discovery. Various of molecular representation schemes of different modals (e.g. textual sequence or graph) are developed. By digitally encoding them, different chemical information can be learned through corresponding network structures. Molecular graphs and Simplified Molecular Input Line Entry System (SMILES) are popular means for molecular representation learning in current. Previous works have done attempts by combining both of them to solve the problem of specific information loss in single-modal representation on various tasks. To further fusing such multi-modal imformation, the correspondence between learned chemical feature from different representation should be considered. To realize this, we propose a novel framework of molecular joint representation learning via Multi-Modal information of SMILES and molecular Graphs, called MMSG. We improve the self-attention mechanism by introducing bond-level graph representation as attention bias in Transformer to reinforce feature correspondence between multi-modal information. We further propose a Bidirectional Message Communication Graph Neural Network (BMC GNN) to strengthen the information flow aggregated from graphs for further combination. Numerous experiments on public property prediction datasets have demonstrated the effectiveness of our model.

Automatic identification of sites prone to topographic seismic amplification effects by the current seismic codes

Current seismic codes provide proxies to estimate seismic amplification effects expected in correspondence of some morphological features. To make possible any empirical validation of these proxies, these features must be univocally identified on the basis of an automatic procedure. To this purpose, based on geomorphological considerations, a GIS-based numerical approach has been developed. The results of a morphometric analysis allowed the correct identification and mapping of the landforms of concern, at a detail corresponding to the resolution of the available digital elevation model (DEM). Some case-studies are provided to show the feasibility of the proposed approach.

Pyramid Attention Network for Image Restoration

Self-similarity refers to the image prior widely used in image restoration algorithms that small but similar patterns tend to occur at different locations and scales. However, recent advanced deep convolutional neural network-based methods for image restoration do not take full advantage of self-similarities by relying on self-attention neural modules that only process information at the same scale. To solve this problem, we present a novel Pyramid Attention module for image restoration, which captures long-range feature correspondences from a multi-scale feature pyramid. Inspired by the fact that corruptions, such as noise or compression artifacts, drop drastically at coarser image scales, our attention module is designed to be able to borrow clean signals from their “clean” correspondences at the coarser levels. The proposed pyramid attention module is a generic building block that can be flexibly integrated into various neural architectures. Its effectiveness is validated through extensive experiments on multiple image restoration tasks: image denoising, demosaicing, compression artifact reduction, and super resolution. Without any bells and whistles, our PANet (pyramid attention module with simple network backbones) can produce state-of-the-art results with superior accuracy and visual quality. Our code is available at https://github.com/SHI-Labs/Pyramid-Attention-Networks

AGConv: Adaptive Graph Convolution on 3D Point Clouds.

Convolution on 3D point clouds is widely researched yet far from perfect in geometric deep learning. The traditional wisdom of convolution characterises feature correspondences indistinguishably among 3D points, arising an intrinsic limitation of poor distinctive feature learning. In this article, we propose Adaptive Graph Convolution (AGConv) for wide applications of point cloud analysis. AGConv generates adaptive kernels for points according to their dynamically learned features. Compared with the solution of using fixed/isotropic kernels, AGConv improves the flexibility of point cloud convolutions, effectively and precisely capturing the diverse relations between points from different semantic parts. Unlike the popular attentional weight schemes, AGConv implements the adaptiveness inside the convolution operation instead of simply assigning different weights to the neighboring points. Extensive evaluations clearly show that our method outperforms state-of-the-arts of point cloud classification and segmentation on various benchmark datasets. Meanwhile, AGConv can flexibly serve more point cloud analysis approaches to boost their performance. To validate its flexibility and effectiveness, we explore AGConv-based paradigms of completion, denoising, upsampling, registration and circle extraction, which are comparable or even superior to their competitors.

Self Correspondence Distillation for End-to-End Weakly-Supervised Semantic Segmentation

Efficiently training accurate deep models for weakly supervised semantic segmentation (WSSS) with image-level labels is challenging and important. Recently, end-to-end WSSS methods have become the focus of research due to their high training efficiency. However, current methods suffer from insufficient extraction of comprehensive semantic information, resulting in low-quality pseudo-labels and sub-optimal solutions for end-to-end WSSS. To this end, we propose a simple and novel Self Correspondence Distillation (SCD) method to refine pseudo-labels without introducing external supervision. Our SCD enables the network to utilize feature correspondence derived from itself as a distillation target, which can enhance the network's feature learning process by complementing semantic information. In addition, to further improve the segmentation accuracy, we design a Variation-aware Refine Module to enhance the local consistency of pseudo-labels by computing pixel-level variation. Finally, we present an efficient end-to-end Transformer-based framework (TSCD) via SCD and Variation-aware Refine Module for the accurate WSSS task. Extensive experiments on the PASCAL VOC 2012 and MS COCO 2014 datasets demonstrate that our method significantly outperforms other state-of-the-art methods. Our code is available at https://github.com/Rongtao-Xu/RepresentationLearning/tree/main/SCD-AAAI2023.

Automatic landmark detection and registration of brain cortical surfaces via quasi-conformal geometry and convolutional neural networks

In medical imaging, surface registration is extensively used for performing systematic comparisons between anatomical structures, with a prime example being the highly convoluted brain cortical surfaces. To obtain a meaningful registration, a common approach is to identify prominent features on the surfaces and establish a low-distortion mapping between them with the feature correspondence encoded as landmark constraints. Prior registration works have primarily focused on using manually labeled landmarks and solving highly nonlinear optimization problems, which are time-consuming and hence hinder practical applications. In this work, we propose a novel framework for the automatic landmark detection and registration of brain cortical surfaces using quasi-conformal geometry and convolutional neural networks. We first develop a landmark detection network (LD-Net) that allows for the automatic extraction of landmark curves given two prescribed starting and ending points based on the surface geometry. We then utilize the detected landmarks and quasi-conformal theory for achieving the surface registration. Specifically, we develop a coefficient prediction network (CP-Net) for predicting the Beltrami coefficients associated with the desired landmark-based registration and a mapping network called the disk Beltrami solver network (DBS-Net) for generating quasi-conformal mappings from the predicted Beltrami coefficients, with the bijectivity guaranteed by quasi-conformal theory. Experimental results are presented to demonstrate the effectiveness of our proposed framework. Altogether, our work paves a new way for surface-based morphometry and medical shape analysis.

A Novel Biologically Inspired Structural Model for Feature Correspondence

A Novel Biologically Inspired Structural Model for Feature Correspondence

LC-HRMS based metabolomics workflow: an alternative strategy for metabolite identification in the anti-doping field.

The proposed metabolomic workflow, based on coupling high-resolution mass spectrometry with computational tools, can be an alternative strategy for metabolite detection and identification. This approach allows the extension of the investigation field to chemically different compounds, maximizing the information obtainable from the data and minimizing the time and resources required. Urine samples were collected from 5 healthy volunteers before and after oral administration of 3β-hydroxyandrost-5-ene-7,17-dione as a model compound and defining three excretion time intervals. Raw data were acquired in both positive and negative ionization modes using an Agilent Technologies 1290 Infinity II series HPLC coupled to a 6545 Accurate-Mass Quadrupole Time-of-Flight. They were then processed to align peak retention times with the same accurate mass, and the resulting data matrix was subjected to multivariate analysis. Multivariate analysis (PCA and PLS-DA models) demonstrated high similarity between samples belonging to the same collection time interval and clear discrimination between different excretion intervals. The blank and long excretion groups were distinguished suggesting the presence of long excretion markers, which are of remarkable interest in anti-doping analyses. The correspondence of some significant features with metabolites reported in the literature confirmed the rationale and usefulness of the proposed metabolomic approach. The presented study proposes a metabolomics workflow for the early detection and characterization of drug metabolites by untargeted urinary analysis to reduce the range of substances still excluded from routine screening. Its application has detected minor steroid metabolites, as well as unexpected endogenous alterations, proving to be an alternative strategy that can allow gathering a more complete range of information in the antidoping field.

A Cross-View Image Matching Method with Feature Enhancement

Most cross-view image matching algorithms focus on designing network structures with excellent performance, ignoring the content information of the image. At the same time, there are non-fixed targets such as cars, ships, and pedestrians in ground perspective images and aerial perspective images. Differences in perspective, direction, and scale cause serious interference with the cross-view matching process. This paper proposes a cross-view image matching method with feature enhancement, which first transforms the empty image to generate a transformation image aligned with the ground–aerial image domain to establish a preliminary geometric correspondence between the ground-space image. Then, the rich feature information of the deep network and the edge information of the cross-convolution layer are used to establish the feature correspondence between the ground-space images. The feature fusion module enhances the tolerance of the network model to scale differences, improving the interference problem of transient non-fixed targets on the matching performance in the images. Finally, the maximum pooling and feature aggregation strategies are adopted to aggregate local features with obvious distinguishability into global features to complete the accurate matching between ground images. The experimental results show that the proposed method has good advance and high accuracy on CVUSA, which is commonly used in public datasets, reaching 92.23%, 98.47%, and 99.74% on the top 1, top 5 and top 10 indicators, respectively, outperforming the original method in the dataset with a limited field of view and image center, better completing the cross-perspective image matching task.

Аналіз стану дослідження проблеми визначення параметрів відносного положення об'єктів орбітального сервісу

Recently considerable attention has been paid to the problem of estimating the pose of an on-orbit service object. Determining the pose at a close distance still remains an open line of research, especially for non-cooperative objects (targets) of on-orbit service. The goal of this work is to overview the state of the art in the problem of determining the relative motion parameters of on-orbit service objects with emphasis on close proximity operations with non-cooperative and unknown targets. The method employed is the analysis of publications devoted to this problem over the last decade. The analysis showed the following. Determining the pose of a non-cooperative orbital object using video systems is a classical approach due to the advantages of light weight and low power consumption. Video camera based pose estimation algorithms usually require prior knowledge of the target features. The main methods of pose estimation still involve approaches based on the recognition and correspondence of image features for consecutive frames or with a target model. Another major approach to pose determination is lidar navigation, where the recognition and correspondence of features based on lidar-derived target surface point clouds are also common methods. Recently, a trend has emerged towards the development of non-feature methods for target pose determination, including unknown targets. The three-dimensional nature of lidar point cloud data is favorable for target pose estimation without any target model. As to the applicability of target pose estimation methods to an unknown target, the implementation of the obvious approach based on constructing a three-dimensional model of the target by processing a series of its images prior to estimating its spatial motion takes a lot of time, which is critical in close proximity operations. The trend in target pose estimation is the development of methods for simultaneous estimation of the pose and shape of an unknown object. In general, the case of an unknown object has not yet been fully investigated.