Class Prototypes Research Articles

Advancing Fine-Grained Few-Shot Object Detection on Remote Sensing Images with Decoupled Self-Distillation and Progressive Prototype Calibration

In data-scarcity scenarios, few-shot object detection (FSOD) methods exhibit a notable advantage in alleviating the over-fitting problem. Currently, research on FSOD in the field of remote sensing is advancing rapidly and FSOD methods based on the fine-tuning paradigm have initially displayed their excellent performance. However, existing fine-tuning methods often encounter classification confusion issues. This is potentially because of the shortage of explicit modeling for transferable common knowledge and the biased class distribution, especially for fine-grained targets with higher inter-class similarity and intra-class variance. In view of this, we first propose a decoupled self-distillation (DSD) method to construct class prototypes in two decoupled feature spaces and measure inter-class correlations as soft labels or aggregation weights. To ensure a robust set of class prototypes during the self-distillation process, we devise a feature filtering module (FFM) to preselect high-quality class representative features. Furthermore, we introduce a progressive prototype calibration module (PPCM) with two steps, compensating the base prototypes with the prior base distribution and then calibrating the novel prototypes with adjacent calibrated base prototypes. Experiments on MAR20 and customized SHIP20 datasets have demonstrated the superior performance of our method compared to other existing advanced FSOD methods, simultaneously confirming the effectiveness of all proposed components.

Few-Shot Object Detection for SAR Images via Context-Aware and Robust Gaussian Flow Representation

In recent decades, few-shot object detection in SAR imagery has gained prominence as a major research focus. The unique imaging mechanism of SAR causes the model to suffer from foreground–background imbalance and inaccurate extraction of class prototypes for novel class instances. Therefore, we propose an innovative few-shot object detection algorithm for SAR images via context-aware and robust Gaussian flow representation. First, we design the Context-Aware Enhancement module to address the foreground–context imbalance problem by refining representative support features into fine-grained prototypes, which are deeply fused with query features based on the prototype matching paradigm. Second, we devise the Manifold Class Distribution Estimation module to address the difficulty of class distribution estimation and the fluctuation of class centers of the sparse novel class. Furthermore, we formulate the Category-Balanced Difference Aggregation module to model the relationship between the base class and the novel class, addressing the sensitivity of the model to the variance of the novel class instances. Finally, we design the Cosine Decoupling Module so that the aggregated features are executed only on the classification branch without affecting the precise localization of the target. Experiments based on SAR-AIRcraft-1.0 and the self-constructed MSAR-AIR dataset indicate that the fine-grained detection and identification performance of the novel class of airplanes can reach 32.90% and 55.26%, respectively, in the 10-shot and 50-shot cases. In addition, our method enables cross-domain detection for different scenarios and sample types and exhibits excellent generalization performance in data-sparse scenarios.

Optimization of image compression using artificial neural networks

The object of research is artificial neural networks of adaptive resonance theory (ART). ART neural networks are classified by matching input data to one of the existing classes, provided that the data is sufficiently similar to the Class prototypes. Continuous and discrete adaptive resonance theory networks ART-1 and ART-2 work effectively in recognition systems, especially in conditions of high uncertainty, when it is necessary to identify a large number of different images. The main problem that was solved in this study was to optimize the process of image compression using artificial neural networks, because image compression is widely used in many scientific and technical fields and becomes especially relevant when transmitting over narrow-band communication channels. A way to overcome these difficulties may be to select basic data for reconstruction from an open data set (Modified National Institute of Standards and Technology) – Fashion-MNIST. There are still unresolved issues related to the fact that lossy compression algorithms with increasing compression ratio usually generate artifacts that are clearly visible to the human eye. A compression algorithm based on neural networks is described, which establishes a correspondence between the input and output spaces consisting of elements of the codebook and neurons. The proposed method uses a different approach (First Order), rather than a simple difference coding scheme (zero order), where the new code is calculated by subtracting the previous encoded block. The peak signal-to-noise ratio of PSNR and the root-mean-square error (MSE) of these algorithms is 24.7 DB with a compression ratio of 25.22. The main area of practical use of the results obtained is improved image compression for processing large – volume video and photo materials without significant loss of quality

Multi-Modal Prototypes for Few-Shot Object Detection in Remote Sensing Images

Few-shot object detection has attracted extensive attention due to the abomination of time-consuming or even impractical large-scale data labeling. Current studies attempted to employ prototype-matching approaches for object detection, constructing class prototypes from textual or visual features. However, single visual prototypes exhibit limited generalization in few-shot scenarios, while single textual prototypes lack the spatial details of remote sensing targets. Therefore, to achieve the best of both worlds, we propose a prototype aggregating module to integrate textual and visual prototypes, leveraging both semantics from textual prototypes and spatial details from visual prototypes. In addition, the transferability of multi-modal few-shot detectors from natural scenarios to remote sensing scenarios remains unexplored, and previous training strategies for FSOD do not adequately consider the characteristics of text encoders. To address the issue, we have conducted extensive ablation studies on different feature extractors of the detector and propose an efficient two-stage training strategy, which takes the characteristics of the text feature extractor into account. Experiments on two common few-shot detection benchmarks demonstrate the effectiveness of our proposed method. In four widely used data splits of DIOR, our method significantly outperforms previous state-of-the-art methods by at most 8.7%.

Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation.

Few-Shot Instance Segmentation (FSIS) requires detecting and segmenting novel classes with limited support examples. Existing methods based on Region Proposal Networks (RPNs) face two issues: 1) overfitting suppresses novel class objects and 2) dual-branch models require complex spatial correlation strategies to prevent spatial information loss when generating class prototypes. We introduce a unified framework, Reference Twice (RefT), to exploit the relationship between support and query features for FSIS and related tasks. Our three main contributions are: 1) a novel transformer-based baseline that avoids overfitting, offering a new direction for FSIS; 2) demonstrating that support object queries encode key factors after base training, allowing query features to be enhanced twice at both feature and query levels using simple cross-attention, thus avoiding complex spatial correlation interaction; and 3) introducing a class-enhanced base knowledge distillation loss to address the issue of DETR-like models struggling with incremental settings due to the input projection layer, enabling easy extension to incremental FSIS. Extensive experimental evaluations on the COCO dataset under three FSIS settings demonstrate that our method performs favorably against existing approaches across different shots, e.g., +8.2/+9.4 performance gain over state-of-the-art methods with 10/30-shots.

Unsupervised prototype self-calibration based on hybrid attention contrastive learning for enhanced few-shot action recognition

The collection and annotation of large-scale video data pose significant challenges, prompting the exploration of few-shot models to recognize unseen actions with limited training samples. However, most existing models rely on complex temporal alignment strategies, neglecting the necessity of constructing accurate class prototypes in action recognition. To address this limitation, we propose an unsupervised prototype self-calibration based on hybrid attention contrastive learning (UPSHC), aiming to refine prototypes through various strategies. Our network comprises three innovative components: a hybrid attention network (HAN), a dual adaptive contrastive learning mechanism (DACL), and an unsupervised prototype self-calibration mechanism (UPSC). The HAN integrates self-attention and cross-attention mechanisms to highlight important spatiotemporal regions, reducing intra-class variations and enhancing inter-class variations. The DACL mechanism constructs prototype-centered and query-centered contrastive losses to deeply explore the similarities between prototypes and samples, improving the model’s sensitivity to subtle inter-class differences. Additionally, DACL employs adaptive margins to dynamically adjust the distances between positive and negative samples, mitigating the unreliability of fixed margins. The UPSC module utilizes unlabeled query samples in an unsupervised manner to optimize target prototypes without requiring additional training. Experimental results on three benchmark datasets demonstrate that UPSHC achieves state-of-the-art performance, highlighting the superiority of our prototype enhancement approach in few-shot action recognition.

Prototypical class-wise test-time adaptation

Test-time adaptation (TTA) refines pre-trained models during deployment, enabling them to effectively manage new, previously unseen data. However, existing TTA methods focus mainly on global domain alignment, which reduces domain-level gaps but often leads to suboptimal performance. This is because they fail to explicitly consider class-wise alignment, resulting in errors when reliable pseudo-labels are unavailable and source domain samples are inaccessible. In this study, we propose a prototypical class-wise test-time adaptation method, which consists of class-wise prototype adaptation and reliable pseudo-labeling. A main challenge in this approach is the lack of direct access to source domain samples. We leverage the class-specific knowledge contained in the weights of the pre-trained model. To construct class prototypes from the unlabeled target domain, we further introduce a methodology to enhance the reliability of pseudo labels. Our method is adaptable to various models and has been extensively validated, consistently outperforming baselines across multiple benchmark datasets.

MvWECM: Multi-view Weighted Evidential [formula omitted]-Means clustering

Traditional multi-view clustering algorithms, designed to produce hard or fuzzy partitions, often neglect the inherent ambiguity and uncertainty in the cluster assignment of objects. This oversight may lead to performance degradation. To address these issues, this paper introduces a novel multi-view clustering method, termed MvWECM, capable of generating credal partitions within the framework of belief functions. The objective function of MvWECM is introduced considering the uncertainty in the cluster structure included in the multi-view dataset. We take into account inter-view conflict to effectively leverage coherent information across different views. Moreover, the effectiveness is heightened through the incorporation of adaptive view weights, which are customized to modulate their smoothness in accordance with their entropy. The optimization method to get the optimal credal membership and class prototypes is derived. The view wights can be also provided as a by-product. Experimental results on several real-word datasets demonstrate the effectiveness and superiority of MvWECM by comparing with some state-of-the-art methods.

Multivariate prototype representation for domain-generalized incremental learning

Deep learning models often suffer from catastrophic forgetting when fine-tuned with samples of new classes. This issue becomes even more challenging when there is a domain shift between training and testing data. In this paper, we address the critical yet less explored Domain-Generalized Class-Incremental Learning (DGCIL) task. We propose a DGCIL approach designed to memorize old classes, adapt to new classes, and reliably classify objects from unseen domains. Specifically, our loss formulation maintains classification boundaries while suppressing domain-specific information for each class. Without storing old exemplars, we employ knowledge distillation and estimate the drift of old class prototypes as incremental training progresses. Our prototype representations are based on multivariate Normal distributions, with means and covariances continually adapted to reflect evolving model features, providing effective representations for old classes. We then sample pseudo-features for these old classes from the adapted Normal distributions using Cholesky decomposition. Unlike previous pseudo-feature sampling strategies that rely solely on average mean prototypes, our method captures richer semantic variations. Experiments on several benchmarks demonstrate the superior performance of our method compared to the state of the art.

Unbiased Feature Learning with Causal Intervention for Visible-Infrared Person Re-Identification

Visible-infrared person re-identification (VI-ReID) aims to match individuals across different modalities. Existing methods can learn class-separable features but still struggle with modality gaps within class due to the modality-specific information, which is discriminative in one modality but not present in another (e.g., a black striped shirt). The presence of the interfering information creates a spurious correlation with the class label, which hinders alignment across modalities. To this end, we propose an Unbiased feature learning method based on Causal inTervention for VI-ReID from three aspects. Firstly, through the proposed structural causal graph, we demonstrate that modality-specific information acts as a confounder that restricts the intra-class feature alignment. Secondly, we propose a causal intervention method to remove the confounder using an effective approximation of backdoor adjustment, which involves adjusting the spurious correlation between features and labels. Thirdly, we incorporate the proposed approximation method into the basic VI-ReID model. Specifically, the confounder can be removed by adjusting the extracted features with a set of weighted pre-trained class prototypes from different modalities, where the weight is adapted based on the features. Extensive experiments on the SYSU-MM01 and RegDB datasets demonstrate that our method outperforms state-of-the-art methods. Code is available at https://github.com/NJUPT-MCC/UCT .

A Pseudo-Labeling Multi-Screening-Based Semi-Supervised Learning Method for Few-Shot Fault Diagnosis.

In few-shot fault diagnosis tasks in which the effective label samples are scarce, the existing semi-supervised learning (SSL)-based methods have obtained impressive results. However, in industry, some low-quality label samples are hidden in the collected dataset, which can cause a serious shift in model training and lead to the performance of SSL-based method degradation. To address this issue, the latest prototypical network-based SSL techniques are studied. However, most prototypical network-based scenarios consider that each sample has the same contribution to the class prototype, which ignores the impact of individual differences. This article proposes a new SSL method based on pseudo-labeling multi-screening for few-shot bearing fault diagnosis. In the proposed work, a pseudo-labeling multi-screening strategy is explored to accurately screen the pseudo-labeling for improving the generalization ability of the prototypical network. In addition, the AdaBoost adaptation-based weighted technique is employed to obtain accurate class prototypes by clustering multiple samples, improving the performance that deteriorated by low-quality samples. Specifically, the squeeze and excitation block technique is used to enhance the useful feature information and suppress non-useful feature information for extracting accuracy features. Finally, three well-known bearing datasets are selected to verify the effectiveness of the proposed method. The experiments illustrated that our method can receive better performance than that of the state-of-the-art methods.

Word vector embedding and self-supplementing network for Generalized Few-shot Semantic Segmentation

Under the condition of sufficient base class samples and a few novel class samples, Generalized Few-shot Semantic Segmentation (GFSS) classifies each pixel in the query image as base class, novel class, or background. A standard GFSS approach involves two training stages: base class learning and novel class updating. However, inter-class interference and information loss which contribute to the poor performance of GFSS, have not been synthetical considered. To address the problem, we propose an Embedded-Self-Supplementing Network (ESSNet), i.e., semantic word embedding and query set self-supplementing information to enhance segmentation accuracy. Specifically, the semantic word embedding module employs distance information between word vectors to assist the model in learning the distance between class prototypes. In order to transform the semantic word vector prototypes from the semantic space to the visual embedding space, we designed a triplet loss function to supervise the word vector embedding module, where the word vector prototype serves as an anchor and positive-negative samples are collected among the general features of the support image. To compensate for the information loss caused by using prototypes to represent classes, we propose a self-supplementing module to mine the information contained in the query image. Specifically, this module first makes a preliminary prediction on the query image, then selects high-confidence area to form pseudo labels, and finally uses pseudo labels to extract query prototypes to supplement the missing information. Extensive experiments on PASCAL-5i and COCO-20i show that ESSNet has superior performance and outperforms state-of-the-art methods in all settings.

Debiased hybrid contrastive learning with hard negative mining for unsupervised person re-identification

The goal of unsupervised person re-identification is to retrieve a specific person across several non-overlapping cameras without the aid of manual labeling information. In recent times, contrastive learning has found extensive application in undertaking the complexities of unsupervised person Re-ID. Nevertheless, prevailing approaches often ignore the bias in negative proxy sampling and the significance of hard negatives in contrastive learning. These limitations have constrained the performance of existing methods. To solve these issues, we introduce a Debiased Hybrid Contrastive Learning with Hard Negative Mining (DHCL-HNM) approach. Particularly, the proposed approach employs an instance-level memory bank to save the class prototypes for all training images. In each training epoch, the memory bank undergoes clustering, dividing the dataset into un-clustered outliers and clustered images with pseudo labels. Then, the debiasing of negative proxies and the hard negative mining are integrated into a hybrid contrastive learning process to enhance the intra-class similarity and the instance discrimination. The debiasing operation is realized during the sampling of negative proxies to reduce the negative effects of false negatives. In the meantime, the hard negative mining can guide the Re-ID model to concentrate on the hard negatives by reweighting negative proxies based on their similarities to the anchor sample. The efficiency of the proposed method in the realm of unsupervised person Re-ID is demonstrated through comprehensive experiment outcomes conducted on several datasets.

Prototype-Based Semantic Segmentation.

Deep learning based semantic segmentation solutions have yielded compelling results over the preceding decade. They encompass diverse network architectures (FCN based or attention based), along with various mask decoding schemes (parametric softmax based or pixel-query based). Despite the divergence, they can be grouped within a unified framework by interpreting the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, we reveal inherent limitations within the parametric segmentation regime, and accordingly develop a nonparametric alternative based on non-learnable prototypes. In contrast to previous approaches that entail the learning of a single weight/query vector per class in a fully parametric manner, our approach represents each class as a set of non-learnable prototypes, relying solely upon the mean features of training pixels within that class. The pixel-wise prediction is thus achieved by nonparametric nearest prototype retrieving. This allows our model to directly shape the pixel embedding space by optimizing the arrangement between embedded pixels and anchored prototypes. It is able to accommodate an arbitrary number of classes with a constant number of learnable parameters. Through empirical evaluation with FCN based and Transformer based segmentation models (i.e., HRNet, Swin, SegFormer, Mask2Former) and backbones (i.e., ResNet, HRNet, Swin, MiT), our nonparametric framework shows superior performance on standard segmentation datasets (i.e., ADE20 K, Cityscapes, COCO-Stuff), as well as in large-vocabulary semantic segmentation scenarios. We expect that this study will provoke a rethink of the current de facto semantic segmentation model design.

Adaptive prototype few-shot image classification method based on feature pyramid.

Few-shot learning aims to enable machines to recognize unseen novel classes using limited samples akin to human capabilities. Metric learning is a crucial approach to addressing this challenge, with its performance primarily dependent on the effectiveness of feature extraction and prototype computation. This article introduces an Adaptive Prototype few-shot image classification method based on Feature Pyramid (APFP). APFP employs a novel feature extraction method called FResNet, which builds upon the ResNet architecture and leverages a feature pyramid structure to retain finer details. In the 5-shot scenario, traditional methods for computing average prototypes exhibit limitations due to the typically diverse and uneven distribution of samples, where simple means may inadequately reflect such diversity. To address this issue, APFP proposes an Adaptive Prototype method (AP) that dynamically computes class prototypes of the support set based on the similarity between support set samples and query samples. Experimental results demonstrate that APFP achieves 67.98% and 85.32% accuracy in the 5-way 1-shot and 5-way 5-shot scenarios on the MiniImageNet dataset, respectively, and 84.02% and 94.44% accuracy on the CUB dataset. These results indicate that the proposed APFP method addresses the few-shot learning problem.

Weighted Contrastive Prototype Network for Few-Shot Hyperspectral Image Classification with Noisy Labels

Few-shot hyperspectral image classification aims to develop the ability of classifying image pixels by using relatively few labeled pixels per class. However, due to the inaccuracy of the localization system and the bias of the ground survey, the potential noisy labels in the training data pose a very significant challenge to few-shot hyperspectral image classification. To solve this problem, this paper proposes a weighted contrastive prototype network (WCPN) for few-shot hyperspectral image classification with noisy labels. WCPN first utilizes a similarity metric to generate the weights of the samples from the same classes, and applies them to calibrate the class prototypes of support and query sets. Then the weighted prototype network will minimize the distance between features and prototypes to train the network. WCPN also incorporates a weighted contrastive regularization function that uses the sample weights as gates to filter the fake positive samples whose labels are incorrect to further improve the discriminative power of the prototypes. We conduct experiments on multiple hyperspectral image datasets with artificially generated noisy labels, and the results show that the WCPN has excellent performance that can sufficiently mitigate the impact of noisy labels.

Cost-Sensitive Weighted Contrastive Learning Based on Graph Convolutional Networks for Imbalanced Alzheimer's Disease Staging.

Identifying the progression stages of Alzheimer's disease (AD) can be considered as an imbalanced multi-class classification problem in machine learning. It is challenging due to the class imbalance issue and the heterogeneity of the disease. Recently, graph convolutional networks (GCNs) have been successfully applied in AD classification. However, these works did not handle the class imbalance issue in classification. Besides, they ignore the heterogeneity of the disease. To this end, we propose a novel cost-sensitive weighted contrastive learning method based on graph convolutional networks (CSWCL-GCNs) for imbalanced AD staging using resting-state functional magnetic resonance imaging (rs-fMRI). The proposed method is developed on a multi-view graph constructed by the functional connectivity (FC) and high-order functional connectivity (HOFC) features of the subjects. A novel cost-sensitive weighted contrastive learning procedure is proposed to capture discriminative information from the minority classes, encouraging the samples in the minority class to provide adequate supervision. Considering the heterogeneity of the disease, the weights of the negative pairs are introduced into contrastive learning and they are computed based on the distance to class prototypes, which are automatically learned from the training data. Meanwhile, the cost-sensitive mechanism is further introduced into contrastive learning to handle the class imbalance issue. The proposed CSWCL-GCN is evaluated on 720 subjects (including 184 NCs, 40 SMC patients, 208 EMCI patients, 172 LMCI patients and 116 AD patients) from the ADNI (Alzheimer's Disease Neuroimaging Initiative). Experimental results show that the proposed CSWCL-GCN outperforms state-of-the-art methods on the ADNI database.

Contrastive prototype network with prototype augmentation for few-shot classification

In recent years, metric-based meta-learning methods have received widespread attention because of their effectiveness in solving few-shot classification problems. However, the scarcity of data frequently results in suboptimal embeddings, causing a discrepancy between anticipated class prototypes and those derived from the support set. These problems severely limit the generalizability of such methods, necessitating further development of Few-Shot Learning (FSL). In this study, we propose the Contrastive Prototype Network (CPN) consisting of three components: (1) Contrastive learning proposed as an auxiliary path to reduce the distance between homogeneous samples and amplify the differences between heterogeneous samples, thereby enhancing the effectiveness and quality of embeddings; (2) A pseudo-prototype strategy proposed to address the bias in prototypes, whereby the pseudo prototypes generated using query set samples are integrated with the initial prototypes to obtain more representative prototypes; (3) A new data augmentation technique, mixupPatch, introduced to alleviate the issue of insufficient data samples, whereby enhanced images are generated by blending the images and labels from different samples, to increase the number of samples. Extensive experiments and ablation studies conducted on five datasets demonstrated that CPN achieves robust results against recent solutions.

OCIE: Augmenting model interpretability via Deconfounded Explanation-Guided Learning

Deep neural networks (DNNs) often encounter significant challenges related to opacity, inherent biases, and shortcut learning, which undermine their practical reliability. In this study, we address these issues by constructing a causal graph to model the unbiased learning process of DNNs. This model reveals that recurrent background information in training samples acts as a confounder, leading to spurious correlations between model inputs and outputs, causing biased predictions. To mitigate these problems and promote unbiased feature learning, we propose the Object-guided Consistency Interpretation Enhancement (OCIE) methodology. OCIE enhances DNN interpretability by integrating explicit objects and explanations into the model’s learning process. Initially, OCIE employs a graph-based algorithm to identify explicit objects within self-supervised vision transformer-learned features. Subsequently, it constructs class prototypes to eliminate invalid detected objects. Finally, OCIE aligns explanations with explicit objects, directing the model’s attention towards the most distinctive classification features rather than irrelevant backgrounds. Extensive experiments on different image classification datasets, including general (ImageNet), fine-grained (Stanford Cars and CUB-200), and medical (HAM) datasets, using two prevailing network architectures, demonstrate that OCIE significantly enhances explanation consistency across all datasets. Furthermore, OCIE proves particularly advantageous for fine-grained classification, especially in few-shot scenarios, by improving both interpretability and classification performance. Additionally, our findings highlight the impact of centralized explanations on the sufficiency of model decisions, suggesting that focusing explanations on explicit objects improves the reliability of DNN predictions. Our code is available at: https://github.com/DLAIResearch/OCIE.

Partial label learning via identifying outlier features

Partial label learning (PLL) is a classification problem where each training instance is ambiguously annotated with a set of candidate labels, among which only one is the ground truth label. The topological structure in the feature space is frequently used by existing PLL approaches to clarify if a candidate label is the ground truth label for a training example. However, these techniques frequently experience cumulative errors brought on by the error-prone labeling confidence estimate throughout the topological structure because of the redundant and noisy features occurring naturally in the feature space. In this paper, we propose a novel approach through Identifying Outlier Features (PL-IOF) to address this challenge. The feature space is decomposed into class prototype features and outlier features, where the former captures the discriminative features of each class prototype, and the latter captures the outlier features in each training instance. A unified framework is specifically suggested to simultaneously optimize class prototypes and outlier features, as well as to estimate labeling confidences over incomplete label training instances. This framework ensures the high quality of the extracted class prototypes by recognizing the outliers. Experiments on both synthetic and real-world datasets demonstrate the out-performance of PL-IOF over the state-of-the-art.