Patch Features Research Articles

QDETRv: Query-Guided DETR for One-Shot Object Localization in Videos

In this work, we study one-shot video object localization problem that aims to localize instances of unseen objects in the target video using a single query image of the object. Toward addressing this challenging problem, we extend a popular and successful object detection method, namely DETR (Detection Transformer), and introduce a novel approach –query-guided detection transformer for videos (QDETRv). A distinctive feature of QDETRv is its capacity to exploit information from the query image and spatio-temporal context of the target video, which significantly aids in precisely pinpointing the desired object in the video. We incorporate cross-attention mechanisms that capture temporal relationships across adjacent frames to handle the dynamic context in videos effectively. Further, to ensure strong initialization for QDETRv, we also introduce a novel unsupervised pretraining technique tailored to videos. This involves training our model on synthetic object trajectories with an analogous objective as the query-guided localization task. During this pretraining phase, we incorporate recurrent object queries and loss functions that encourage accurate patch feature reconstruction. These additions enable better temporal understanding and robust representation learning. Our experiments show that the proposed model significantly outperforms the competitive baselines on two public benchmarks, VidOR and ImageNet-VidVRD, extended for one-shot open-set localization tasks.

Semantic-Aware Autoregressive Image Modeling for Visual Representation Learning

The development of autoregressive modeling (AM) in computer vision lags behind natural language processing (NLP) in self-supervised pre-training. This is mainly caused by the challenge that images are not sequential signals and lack a natural order when applying autoregressive modeling. In this study, inspired by human beings’ way of grasping an image, i.e., focusing on the main object first, we present a semantic-aware autoregressive image modeling (SemAIM) method to tackle this challenge. The key insight of SemAIM is to autoregressively model images from the semantic patches to the less semantic patches. To this end, we first calculate a semantic-aware permutation of patches according to their feature similarities and then perform the autoregression procedure based on the permutation. In addition, considering that the raw pixels of patches are low-level signals and are not ideal prediction targets for learning high-level semantic representation, we also explore utilizing the patch features as the prediction targets. Extensive experiments are conducted on a broad range of downstream tasks, including image classification, object detection, and instance/semantic segmentation, to evaluate the performance of SemAIM. The results demonstrate SemAIM achieves state-of-the-art performance compared with other self-supervised methods. Specifically, with ViT-B, SemAIM achieves 84.1% top-1 accuracy for fine-tuning on ImageNet, 51.3% AP and 45.4% AP for object detection and instance segmentation on COCO, which outperforms the vanilla MAE by 0.5%, 1.0%, and 0.5%, respectively. Code is available at https://github.com/skyoux/SemAIM.

Semantic-Oriented Feature Coupling Transformer for Vehicle Re-Identification in Intelligent Transportation System

More robust intelligent transportation systems including autonomous driving systems are in full flourish with the revolution of deep learning and the 6G wireless communication network. Vehicle Re-Identification, an indispensable branch of the intelligent transportation system, aims to retrieve specific vehicles captured from non-overlapping cameras. However, this is fundamentally challenging with the substantial inter-class similarity and substantial intra-class divergence. Embedding semantic information into vehicle re-identification task has gained ample interest, but the performance needs to be further improved. This work proposes a semantic-oriented feature coupling transformer (SOFCT) for vehicle re-identification as a solution. Specifically, the knowledge-based transformer is first embedded to model images with discriminative attributes. Second, original patches are divided into five semantic groups via semantics-patches coupling, and the feature extractions for different semantics are performed in the semantic feature extraction (SFE) transformer. Third, patch features are weighted via semantics-patches coupling in the patch feature weighting (PFW) transformer, the weighted feature is fed into subsequent encoders to excavate information. Finally, two groups of learnable semantics are embedded to automatically learn semantic features in the learnable semantic extraction (LSE) transformer. Experiments demonstrate that the proposed SOFCT method surpasses other state-of-the-arts with the mAP/Rank-1 of 80.7%/96.6%, 89.8%/84.5%, 86.4%/80.9%, and 84.3%/78.7% on VeRi776 and VehicleID.

Self-supervised context-aware correlation filter for robust landmark tracking in liver ultrasound sequences.

Respiratory motion-induced displacement of internal organs poses a significant challenge in image-guided radiation therapy, particularly affecting liver landmark tracking accuracy. Addressing this concern, we propose a self-supervised method for robust landmark tracking in long liver ultrasound sequences. Our approach leverages a Siamese-based context-aware correlation filter network, trained by using the consistency loss between forward tracking and back verification. By effectively utilizing both labeled and unlabeled liver ultrasound images, our model, Siam-CCF , mitigates the impact of speckle noise and artifacts on ultrasonic image tracking by a context-aware correlation filter. Additionally, a fusion strategy for template patch feature helps the tracker to obtain rich appearance information around the point-landmark. Siam-CCF achieves a mean tracking error of 0.79±0.83 mm at a frame rate of 118.6 fps, exhibiting a superior speed-accuracy trade-off on the public MICCAI 2015 Challenge on Liver Ultrasound Tracking (CLUST2015) 2D dataset. This performance won the 5th place on the CLUST2015 2D point-landmark tracking task. Extensive experiments validate the effectiveness of our proposed approach, establishing it as one of the top-performing techniques on the CLUST2015 online leaderboard at the time of this submission.

Model-Agnostic Binary Patch Grouping for Bone Marrow Whole Slide Image Representation

Histopathology is the reference standard for pathology diagnosis, and has evolved with the digitization of glass slides, i.e. whole slide images (WSIs). Trained histopathologists can help to diagnose disease by examining WSIs visually, but this process is time-consuming and prone to variability. To address these issues, AI models are being developed to create slide-level representations of WSIs, summarizing the entire slide as a single vector. This enables various computational pathology applications, including inter-slide search, multi-modal training, and slide-level classification. Achieving expressive and robust slide-level representations hinges on patch feature extraction and aggregation steps. We propose integrating an additional Binary Patch Grouping (BPG) step, a plugin that can be integrated into various slide-level representation pipelines to enhance the quality of slide-level representation in bone marrow histopathology. BPG excludes patches with less clinical relevance through minimal interaction with the pathologist: a one-time human intervention for the entire process. We further investigated domain-general versus domain-specific feature extraction models based on convolution and attention and examined two different feature aggregation methods, with and without BPG, showing BPG’s generalizability. We show that BPG boosts the performance of WSI retrieval (mAP@10) by 4% and improves WSI classification (weighted-F1) by 5% relative to not using BPG. Additionally, we found that the pipeline with BPG, domain-general large models and parameterized pooling produced the best-quality slide-level representations.

Patterns of Alluviation in Mixed Bedrock‐Alluvial Channels: 1. Numerical Model

AbstractMixed bedrock‐alluvial rivers can exhibit partial alluvial cover, which plays an important role in controlling bedrock erosion rates and landscape evolution. However, numerical morphodynamic models are generally unable to predict quasi‐steady persistent patches of depositional alluvial features under conditions where the sediment supply is less than the sediment transport capacity of the channel. Hence, we present a new two‐dimensional depth‐averaged morphodynamic model that can be applied to both fully alluvial and mixed bedrock‐alluvial channels, and we use the model to gain insight into the mechanisms responsible for the development of sediment patches and patterns of bedrock alluviation. The model computes hydrodynamics, sediment transport, and bed evolution using a roughness partitioning that accounts for differential roughness of sediment and bedrock, roughness due to sediment transport, and form drag. The model successfully replicates observations of bar development and migration from a fully alluvial flume experiment, and it models persistent sediment patches observed in a mixed bedrock‐alluvial flume experiment. Numerical experiments in which the form drag, sediment transport roughness, and bedform stress correction were neglected did not successfully reproduce the observed persistent sediment cover in the mixed bedrock‐alluvial case, suggesting that accounting for these different roughness components is critical to successfully model sediment dynamics in bedrock channels.

Multi-Source Patch Feature Fusion With Neighborhood Flash Attention Transformer for Pixel-Level Vehicle and Road Recognition in Hyperspectral Image

Multi-Source Patch Feature Fusion With Neighborhood Flash Attention Transformer for Pixel-Level Vehicle and Road Recognition in Hyperspectral Image

Unleash the Power of State Space Model for Whole Slide Image with Local Aware Scanning and Importance Resampling.

Whole slide image (WSI) analysis is gaining prominence within the medical imaging field. However, previous methods often fall short of efficiently processing entire WSIs due to their gigapixel size. Inspired by recent developments in state space models, this paper introduces a new Pathology Mamba (PAM) for more accurate and robust WSI analysis. PAM includes three carefully designed components to tackle the challenges of enormous image size, the utilization of local and hierarchical information, and the mismatch between the feature distributions of training and testing during WSI analysis. Specifically, we design a Bi-directional Mamba Encoder to process the extensive patches present in WSIs effectively and efficiently, which can handle large-scale pathological images while achieving high performance and accuracy. To further harness the local information and inherent hierarchical structure of WSI, we introduce a novel Local-aware Scanning module, which employs a local-aware mechanism alongside hierarchical scanning to adeptly capture both the local information and the overarching structure within WSIs. Moreover, to alleviate the patch feature distribution misalignment between training and testing, we propose a Test-time Importance Resampling module to conduct testing patch resampling to ensure consistency of feature distribution between the training and testing phases, and thus enhance model prediction. Extensive evaluation on nine WSI datasets with cancer subtyping and survival prediction tasks demonstrates that PAM outperforms current state-of-the-art methods and also its enhanced capability in modeling discriminative areas within WSIs. The source code is available at https://github.com/HKU-MedAI/PAM.

Multimodal Proxy-Free Face Anti-Spoofing Exploiting Local Patch Features

Multimodal Proxy-Free Face Anti-Spoofing Exploiting Local Patch Features

Voting-based patch sequence autoregression network for adaptive point cloud completion

Point cloud completion aims to estimate the whole shapes of objects from their partial scans, and one of the main obstacles that prevents current methods from being applied in real-world scenarios is the variety of structural losses in real-scanned objects, which can hardly be fully included and reflected by the training samples. In this paper, we introduce Patch Sequence Autoregression Network (PSA-Net), a learning-based method that can be trained without the partial point clouds in dataset and is inherently adaptable to input scans with different levels of shape incompleteness: It makes restoring the unseen parts of objects be equivalent to predicting the missing tokens in local patch embedding sequences, and such prediction can start from any initial states. Specifically, we first introduce a Sequential Patch AutoEncoder that reconstructs complete point clouds from quantized patch feature sequences. Second, we establish a Mixed Patch Autoregression pipeline that can flexibly infer the whole sequence from any number of known tokens at any positions. Third, we propose a Voting-Based Mapping module that makes input points softly vote for their possible related tokens in sequences based on their local areas, which transforms partial point clouds to masked sequences in test. Quantitative and qualitative evaluations on two synthetic and four real-world datasets illustrate the competitive performances of our network when comparing with existing approaches.

Curriculum Consistency Learning and Multi-Scale Contrastive Constraint in Semi-Supervised Medical Image Segmentation.

Data scarcity poses a significant challenge in medical image segmentation, thereby highlighting the importance of leveraging sparse annotation data. In addressing this issue, semi-supervised learning has emerged as an effective approach for training neural networks using limited labeled data. In this study, we introduced a curriculum consistency constraint within the context of semi-supervised medical image segmentation, thus drawing inspiration from the human learning process. By dynamically comparing patch features with full image features, we enhanced the network's ability to learn. Unlike existing methods, our approach adapted the patch size to simulate the human curriculum process, thereby progressing from easy to hard tasks. This adjustment guided the model toward improved convergence optima and generalization. Furthermore, we employed multi-scale contrast learning to enhance the representation of features. Our method capitalizes on the features extracted from multiple layers to explore additional semantic information and point-wise representations. To evaluate the effectiveness of our proposed approach, we conducted experiments on the Kvasir-SEG polyp dataset and the ISIC 2018 skin lesion dataset. The experimental results demonstrated that our method surpassed state-of-the-art semi-supervised methods by achieving a 9.2% increase in the mean intersection over union (mIoU) for the Kvasir-SEG dataset. This improvement substantiated the efficacy of our proposed curriculum consistency constraint and multi-scale contrastive loss.

Hyperspectral image band selection method combining the variation degree of salient features

ABSTRACTBand selection (BS) work for hyperspectral remote sensing images (HRSIs) has attracted increasing attention from researchers. In the HRSI, the salient features of each band always include the more important discriminative features. However, most existing BS methods always have difficulty in simultaneously performing local-global consistency analysis on salient features in bands of HRSIs. In addition, these methods also ignore the consideration of the diversity of band information to some extent when selecting a band subset. To tackle these problems, a novel unsupervised hyperspectral BS method combining the variation degree of salient features (UBSF) is proposed in this article. UBSF first promotes the singular value decomposition method to analyse the salient features in bands and develops a new low-dimensional local-global salient feature representation (NLR) method for discriminative feature representations of the bands. The NLR method combines the variation degree of the features of the significant representative patches of each band in their corresponding significant feature subspaces to achieve a local-global consistency analysis of salient features of each band of HRSIs. Next, UBSF proposes a new two-stage strategy (TSS) to preserve the diversity of BS results as much as possible. In TSS, we propose a novel similarity criterion, which can make the highly similar band subsets with near information amount in the overall bands adaptively compressed according to the obtained similarity results of bands further. Then, the band subset with the most information is selected as the final BS results in the obtained reduced-dimensional HRSI. Finally, the validity of the UBSF method is verified based on three measured HRSI datasets. The experimental results show that the proposed UBSF is superior to other state-of-the-art BS methods.

MG-Trans: Multi-Scale Graph Transformer With Information Bottleneck for Whole Slide Image Classification.

Multiple instance learning (MIL)-based methods have become the mainstream for processing the megapixel-sized whole slide image (WSI) with pyramid structure in the field of digital pathology. The current MIL-based methods usually crop a large number of patches from WSI at the highest magnification, resulting in a lot of redundancy in the input and feature space. Moreover, the spatial relations between patches can not be sufficiently modeled, which may weaken the model's discriminative ability on fine-grained features. To solve the above limitations, we propose a Multi-scale Graph Transformer (MG-Trans) with information bottleneck for whole slide image classification. MG-Trans is composed of three modules: patch anchoring module (PAM), dynamic structure information learning module (SILM), and multi-scale information bottleneck module (MIBM). Specifically, PAM utilizes the class attention map generated from the multi-head self-attention of vision Transformer to identify and sample the informative patches. SILM explicitly introduces the local tissue structure information into the Transformer block to sufficiently model the spatial relations between patches. MIBM effectively fuses the multi-scale patch features by utilizing the principle of information bottleneck to generate a robust and compact bag-level representation. Besides, we also propose a semantic consistency loss to stabilize the training of the whole model. Extensive studies on three subtyping datasets and seven gene mutation detection datasets demonstrate the superiority of MG-Trans.

Enhancing Anomaly Detection Models for Industrial Applications through SVM-Based False Positive Classification

Unsupervised anomaly detection models are crucial for the efficiency of industrial applications. However, frequent false alarms hinder the widespread adoption of unsupervised anomaly detection, especially in fault detection tasks. To this end, our research delves into the dependence of false alarms on the baseline anomaly detector by analyzing the high-response regions in anomaly maps. We introduce an SVM-based false positive classifier as a post-processing module, which identifies false alarms from positive predictions at the object level. Moreover, we devise a sample synthesis strategy that generates synthetic false positives from the trained baseline detector while producing synthetic defect patch features from fuzzy domain knowledge. Following comprehensive evaluations, we showcase substantial performance enhancements in two advanced out-of-distribution anomaly detection models, Cflow and Fastflow, across image and pixel-level anomaly detection performance metrics. Substantive improvements are observed in two distinct industrial applications, with notable instances of elevating the image-level F1-score from 46.15% to 78.26% in optimal scenarios and boosting pixel-level AUROC from 72.36% to 94.74%.

An Automatic and Robust Visual SLAM Method for Intra-Abdominal Environment Reconstruction

Three-dimensional (3D) surface reconstruction is used to solve the problem of the narrow field of view in laparoscopy. It can provide surgeons or computer-assisted surgery systems with real-time complete internal abdominal anatomy. However, rapid changes in image depth, less texture, and specular reflection pose a challenge for the reconstruction. It is difficult to stably complete the reconstruction process using feature-based simultaneous localization and mapping (SLAM) method. This paper proposes a robust laparoscopic 3D surface reconstruction method using SLAM, which can automatically select appropriate parameters for stereo matching and robustly find matching point pairs for laparoscope motion estimation. The changing trend of disparity maps is used to predict stereo matching parameters to improve the quality of the disparity map. Feature patch extraction and tracking are selected to replace feature point extraction and matching in motion estimation, which reduces its failure and interruption in feature-based SLAM. The proposed feature patch matching method is suitable for parallel computing, which can improve its computing speed. Evaluation results on public in vivo and ex vivo porcine abdominal video data show the efficiency and robustness of our 3D surface reconstruction approach.

PatchAugNet: Patch feature augmentation-based heterogeneous point cloud place recognition in large-scale street scenes

Point Cloud Place Recognition (PCPR) in street scenes is an essential task in the fields of autonomous driving, robot navigation, and urban map updating. However, the domain gap between heterogeneous point clouds and the difficulty of feature characterization in large-scale complex street scenes pose significant challenges for existing PCPR methods. Most PCPR methods only take into account point clouds collected by the same platforms and sensors, thus they are with poor domain transferability. In this paper, we propose PatchAugNet, which utilizes patch feature augmentation and adaptive pyramid feature aggregation to achieve better performance and generalizability for Heterogeneous Point Cloud-based Place Recognition (HPCPR) tasks. Firstly, multi-scale local features are extracted by the pyramid feature extraction module. Secondly, local features are enhanced by the patch feature augmentation module to overcome the domain gap problem and achieve better feature representation as well as network generalizability. Finally, a global feature is generated using an adaptive pyramid feature aggregation module, which automatically adjusts and balances the proportion of intra-scale and inter-scale features according to the scene content. To evaluate the performance of PatchAugNet, a large-scale heterogeneous point cloud dataset consisting of high-precision Mobile Laser Scanning (MLS) point clouds and helmet-mounted Portable Laser Scanning (PLS) point clouds is collected. The dataset covers various street scenes with a length of over 20 km. The comprehensive experimental results indicate that PatchAugNet achieves State-Of-The-Art (SOTA) performance with 83.43 % recall@top1% and 60.34 % recall@top1 on unseen large-scale street scenes, outperforming existing SOTA PCPR methods by + 9.57 recall@top1% and + 15.50 recall@top1, while exhibiting better generalizability. For source code and detailed experimental results, please refer to: https://github.com/WHU-USI3DV/PatchAugNet.

EERCA-ViT: Enhanced Effective Region and Context-Aware Vision Transformers for image sentiment analysis

Different parts of an image have a strong or weak guiding effect on emotions. The key to emotion recognition of images is to fully exploit the regions associated with emotions. Therefore, this paper proposes a visual sentiment classification model with two branches based on visual transformer, termed as Enhanced Effective Region and Context-Aware Vision Transformers (EERCA-ViT). This model includes a primary branch and an auxiliary branch. The primary branch simulates interdependencies between patches by squeezing and stimulating patches (P-SE), thereby highlighting effective region features in the global feature. The auxiliary branch removes feature patches that have been tagged by the primary branch through the context-aware module (CAM), forcing the network to discover different sentiment discriminative regions. At the same time, a two-part loss function is constructed to improve the robustness of the model. Finally, extensive experiments on six benchmark datasets show that the proposed method outperforms the state-of-the-art image sentiment analysis methods. Furthermore, the effectiveness of the different modules of the framework (P-SE and CAM) has been well demonstrated through extensive comparative experiments.

Automated identification of copy-move forgery using Hessian and patch feature extraction techniques.

Copy-move forgeries are a conceptual sort of image retouching that involves duplicating a portion of an image and moving it to a different position within the original image, whether by modifying the duplicated part or just moving it altogether. To identify cloned portions that have been reproduced into the same digitized image, the suggested hybrid features in this article combine using the Hessian and Raw patch features on gray-level images. Using a suggested model that combines patch features built on key points detected by the Hessian detector on gray-level image, localization of duplicated and pasted portions of the manipulated image were found. After using the combined features in the matching step, the parallelism condition was applied together with the random sample consensus method to eliminate mismatches. Two databases, GRIP and the image manipulation dataset (IMD), were used for the detection and characterization, and the empirical findings show that the approach was successful in achieving an F1 score 100% for the GRIP database. Additionally, with the IMD database, it produced a 92.13% F1 score. The proposed method was also shown to be effective by obtaining high F1 scores in images where noise, JPEG compression, and scaling attacks were applied to make forgery detection difficult.

X-LDA: An interpretable and knowledge-informed heterogeneous graph learning framework for LncRNA-disease association prediction

The identification of disease-related long noncoding RNAs (lncRNAs) is beneficial to unravel the intricacies of gene expression regulation and epigenetic signatures. Computational methods provide a cost-effective means to explore lncRNA-disease associations (LDAs). However, these methods often lack interpretability, leaving their predictions less convincing to biological and medical researchers. We propose an interpretable and knowledge-informed heterogeneous graph learning framework based on graph patch convolution and integrated gradients to predict LDAs and provides intuitive explanations for its predictions, called X-LDA. The heterogeneous graph is the foundation of the predictions of LDAs, we construct the knowledge-informed heterogeneous graph including LDAs drawn from biological experiments, lncRNA similarities rooted in gene sequences, disease similarities constructed based on disease categorizations. To integrate diverse biological premises and facilitate interpretability, we define nine distinct graph patch types, which encapsulate essential topological relationships within lncRNA-disease node pairs. X-LDA is designed to employ parameter sharing and multi-convolution kernels to grasp common and multiple perspectives of the graph patches, respectively. This approach culminates in the fusion of various semantic information into context embeddings. These post-hoc explanations hinge on graph patch features and integrated gradients, shedding light on the underlying factors driving predictions. Cross validation experiment on the dataset curated from databases and literatures demonstrates that the superior performance of X-LDA in comparison to nine state-of-the-art methods of three categories. X-LDA achieves a larger average area under the receiver operating curve 0.9891 (by at least 6.68%), and a larger average area under the precision–recall curve 0.7907 (by at least 23.2%) than competitive methods. The results of our well-designed ablation and interpretability experiments and Kyoto Encyclopedia of Genes and Genomes (KEGG) enrichment analysis demonstrate X-LDA’s robustness, learnability, predictability, and interpretability. The applicability of X-LDA is also demonstrated through a case study involving the investigation of associated lncRNAs in prostate cancer, colorectal cancer, and breast cancer.

High-order relational generative adversarial network for video super-resolution

Video super-resolution can reconstruct a sequence of high-resolution frames with temporally consistent contents from their corresponding low-resolution sequences. The key challenge for this task is how to effectively utilize both inter-frame temporal relations and intra-frame spatial relations. The existing methods for super-resolving the videos commonly estimate optical flows to align the features of multiple frames based on temporal correlations. However, motion estimation is often error-prone and hence largely hinders the recovery of plausible details. Moreover, high-order contextual dependencies in the feature space are rarely exploited for further enhancing the spatio-temporal information fusion. To this end, we propose a novel generative adversarial network to super-resolve low-resolution videos, which makes full use of patch embeddings and is effective in exploring high-order spatio-temporal relations of the feature patches. Specifically, a motion-aware relation module is designed to handle the alignment between neighboring frames and reference ones. Depending on a patch-matching strategy for adaptive selection of multiple most similar patches, the cross-scale graph is constructed to reliably aggregate these patches using a feature pyramid. Based on the structure of multi-scale graph, a context-aware relation module is developed to capture high-order dependencies among resulting warped patches for better leveraging long-range complementary contexts. To further enhance reconstruction ability, the temporal position information of video sequences is also encoded into this module. Dual discriminators with cycle consistent constraints are adopted to provide more informative feedback to the generator while maintaining the global coherence. Extensive experiments have demonstrated the effectiveness of the proposed method in terms of quantitative and qualitative evaluation metrics.