Class Prototypes Research Articles

MvWECM: Multi-view Weighted Evidential [formula omitted]-Means clustering

Traditional multi-view clustering algorithms, designed to produce hard or fuzzy partitions, often neglect the inherent ambiguity and uncertainty in the cluster assignment of objects. This oversight may lead to performance degradation. To address these issues, this paper introduces a novel multi-view clustering method, termed MvWECM, capable of generating credal partitions within the framework of belief functions. The objective function of MvWECM is introduced considering the uncertainty in the cluster structure included in the multi-view dataset. We take into account inter-view conflict to effectively leverage coherent information across different views. Moreover, the effectiveness is heightened through the incorporation of adaptive view weights, which are customized to modulate their smoothness in accordance with their entropy. The optimization method to get the optimal credal membership and class prototypes is derived. The view wights can be also provided as a by-product. Experimental results on several real-word datasets demonstrate the effectiveness and superiority of MvWECM by comparing with some state-of-the-art methods.

Unbiased Feature Learning with Causal Intervention for Visible-Infrared Person Re-Identification

Visible-infrared person re-identification (VI-ReID) aims to match individuals across different modalities. Existing methods can learn class-separable features but still struggle with modality gaps within class due to the modality-specific information, which is discriminative in one modality but not present in another (e.g., a black striped shirt). The presence of the interfering information creates a spurious correlation with the class label, which hinders alignment across modalities. To this end, we propose an Unbiased feature learning method based on Causal inTervention for VI-ReID from three aspects. Firstly, through the proposed structural causal graph, we demonstrate that modality-specific information acts as a confounder that restricts the intra-class feature alignment. Secondly, we propose a causal intervention method to remove the confounder using an effective approximation of backdoor adjustment, which involves adjusting the spurious correlation between features and labels. Thirdly, we incorporate the proposed approximation method into the basic VI-ReID model. Specifically, the confounder can be removed by adjusting the extracted features with a set of weighted pre-trained class prototypes from different modalities, where the weight is adapted based on the features. Extensive experiments on the SYSU-MM01 and RegDB datasets demonstrate that our method outperforms state-of-the-art methods. Code is available at https://github.com/NJUPT-MCC/UCT .

A Pseudo-Labeling Multi-Screening-Based Semi-Supervised Learning Method for Few-Shot Fault Diagnosis.

In few-shot fault diagnosis tasks in which the effective label samples are scarce, the existing semi-supervised learning (SSL)-based methods have obtained impressive results. However, in industry, some low-quality label samples are hidden in the collected dataset, which can cause a serious shift in model training and lead to the performance of SSL-based method degradation. To address this issue, the latest prototypical network-based SSL techniques are studied. However, most prototypical network-based scenarios consider that each sample has the same contribution to the class prototype, which ignores the impact of individual differences. This article proposes a new SSL method based on pseudo-labeling multi-screening for few-shot bearing fault diagnosis. In the proposed work, a pseudo-labeling multi-screening strategy is explored to accurately screen the pseudo-labeling for improving the generalization ability of the prototypical network. In addition, the AdaBoost adaptation-based weighted technique is employed to obtain accurate class prototypes by clustering multiple samples, improving the performance that deteriorated by low-quality samples. Specifically, the squeeze and excitation block technique is used to enhance the useful feature information and suppress non-useful feature information for extracting accuracy features. Finally, three well-known bearing datasets are selected to verify the effectiveness of the proposed method. The experiments illustrated that our method can receive better performance than that of the state-of-the-art methods.

Debiased hybrid contrastive learning with hard negative mining for unsupervised person re-identification

The goal of unsupervised person re-identification is to retrieve a specific person across several non-overlapping cameras without the aid of manual labeling information. In recent times, contrastive learning has found extensive application in undertaking the complexities of unsupervised person Re-ID. Nevertheless, prevailing approaches often ignore the bias in negative proxy sampling and the significance of hard negatives in contrastive learning. These limitations have constrained the performance of existing methods. To solve these issues, we introduce a Debiased Hybrid Contrastive Learning with Hard Negative Mining (DHCL-HNM) approach. Particularly, the proposed approach employs an instance-level memory bank to save the class prototypes for all training images. In each training epoch, the memory bank undergoes clustering, dividing the dataset into un-clustered outliers and clustered images with pseudo labels. Then, the debiasing of negative proxies and the hard negative mining are integrated into a hybrid contrastive learning process to enhance the intra-class similarity and the instance discrimination. The debiasing operation is realized during the sampling of negative proxies to reduce the negative effects of false negatives. In the meantime, the hard negative mining can guide the Re-ID model to concentrate on the hard negatives by reweighting negative proxies based on their similarities to the anchor sample. The efficiency of the proposed method in the realm of unsupervised person Re-ID is demonstrated through comprehensive experiment outcomes conducted on several datasets.

Multivariate prototype representation for domain-generalized incremental learning

Deep learning models often suffer from catastrophic forgetting when fine-tuned with samples of new classes. This issue becomes even more challenging when there is a domain shift between training and testing data. In this paper, we address the critical yet less explored Domain-Generalized Class-Incremental Learning (DGCIL) task. We propose a DGCIL approach designed to memorize old classes, adapt to new classes, and reliably classify objects from unseen domains. Specifically, our loss formulation maintains classification boundaries while suppressing domain-specific information for each class. Without storing old exemplars, we employ knowledge distillation and estimate the drift of old class prototypes as incremental training progresses. Our prototype representations are based on multivariate Normal distributions, with means and covariances continually adapted to reflect evolving model features, providing effective representations for old classes. We then sample pseudo-features for these old classes from the adapted Normal distributions using Cholesky decomposition. Unlike previous pseudo-feature sampling strategies that rely solely on average mean prototypes, our method captures richer semantic variations. Experiments on several benchmarks demonstrate the superior performance of our method compared to the state of the art.

Word vector embedding and self-supplementing network for Generalized Few-shot Semantic Segmentation

Under the condition of sufficient base class samples and a few novel class samples, Generalized Few-shot Semantic Segmentation (GFSS) classifies each pixel in the query image as base class, novel class, or background. A standard GFSS approach involves two training stages: base class learning and novel class updating. However, inter-class interference and information loss which contribute to the poor performance of GFSS, have not been synthetical considered. To address the problem, we propose an Embedded-Self-Supplementing Network (ESSNet), i.e., semantic word embedding and query set self-supplementing information to enhance segmentation accuracy. Specifically, the semantic word embedding module employs distance information between word vectors to assist the model in learning the distance between class prototypes. In order to transform the semantic word vector prototypes from the semantic space to the visual embedding space, we designed a triplet loss function to supervise the word vector embedding module, where the word vector prototype serves as an anchor and positive-negative samples are collected among the general features of the support image. To compensate for the information loss caused by using prototypes to represent classes, we propose a self-supplementing module to mine the information contained in the query image. Specifically, this module first makes a preliminary prediction on the query image, then selects high-confidence area to form pseudo labels, and finally uses pseudo labels to extract query prototypes to supplement the missing information. Extensive experiments on PASCAL-5i and COCO-20i show that ESSNet has superior performance and outperforms state-of-the-art methods in all settings.

Prototype-Based Semantic Segmentation.

Deep learning based semantic segmentation solutions have yielded compelling results over the preceding decade. They encompass diverse network architectures (FCN based or attention based), along with various mask decoding schemes (parametric softmax based or pixel-query based). Despite the divergence, they can be grouped within a unified framework by interpreting the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, we reveal inherent limitations within the parametric segmentation regime, and accordingly develop a nonparametric alternative based on non-learnable prototypes. In contrast to previous approaches that entail the learning of a single weight/query vector per class in a fully parametric manner, our approach represents each class as a set of non-learnable prototypes, relying solely upon the mean features of training pixels within that class. The pixel-wise prediction is thus achieved by nonparametric nearest prototype retrieving. This allows our model to directly shape the pixel embedding space by optimizing the arrangement between embedded pixels and anchored prototypes. It is able to accommodate an arbitrary number of classes with a constant number of learnable parameters. Through empirical evaluation with FCN based and Transformer based segmentation models (i.e., HRNet, Swin, SegFormer, Mask2Former) and backbones (i.e., ResNet, HRNet, Swin, MiT), our nonparametric framework shows superior performance on standard segmentation datasets (i.e., ADE20 K, Cityscapes, COCO-Stuff), as well as in large-vocabulary semantic segmentation scenarios. We expect that this study will provoke a rethink of the current de facto semantic segmentation model design.

Adaptive prototype few-shot image classification method based on feature pyramid

Few-shot learning aims to enable machines to recognize unseen novel classes using limited samples akin to human capabilities. Metric learning is a crucial approach to addressing this challenge, with its performance primarily dependent on the effectiveness of feature extraction and prototype computation. This article introduces an Adaptive Prototype few-shot image classification method based on Feature Pyramid (APFP). APFP employs a novel feature extraction method called FResNet, which builds upon the ResNet architecture and leverages a feature pyramid structure to retain finer details. In the 5-shot scenario, traditional methods for computing average prototypes exhibit limitations due to the typically diverse and uneven distribution of samples, where simple means may inadequately reflect such diversity. To address this issue, APFP proposes an Adaptive Prototype method (AP) that dynamically computes class prototypes of the support set based on the similarity between support set samples and query samples. Experimental results demonstrate that APFP achieves 67.98% and 85.32% accuracy in the 5-way 1-shot and 5-way 5-shot scenarios on the MiniImageNet dataset, respectively, and 84.02% and 94.44% accuracy on the CUB dataset. These results indicate that the proposed APFP method addresses the few-shot learning problem.

Weighted Contrastive Prototype Network for Few-Shot Hyperspectral Image Classification with Noisy Labels

Few-shot hyperspectral image classification aims to develop the ability of classifying image pixels by using relatively few labeled pixels per class. However, due to the inaccuracy of the localization system and the bias of the ground survey, the potential noisy labels in the training data pose a very significant challenge to few-shot hyperspectral image classification. To solve this problem, this paper proposes a weighted contrastive prototype network (WCPN) for few-shot hyperspectral image classification with noisy labels. WCPN first utilizes a similarity metric to generate the weights of the samples from the same classes, and applies them to calibrate the class prototypes of support and query sets. Then the weighted prototype network will minimize the distance between features and prototypes to train the network. WCPN also incorporates a weighted contrastive regularization function that uses the sample weights as gates to filter the fake positive samples whose labels are incorrect to further improve the discriminative power of the prototypes. We conduct experiments on multiple hyperspectral image datasets with artificially generated noisy labels, and the results show that the WCPN has excellent performance that can sufficiently mitigate the impact of noisy labels.

Cost-Sensitive Weighted Contrastive Learning Based on Graph Convolutional Networks for Imbalanced Alzheimer's Disease Staging.

Identifying the progression stages of Alzheimer's disease (AD) can be considered as an imbalanced multi-class classification problem in machine learning. It is challenging due to the class imbalance issue and the heterogeneity of the disease. Recently, graph convolutional networks (GCNs) have been successfully applied in AD classification. However, these works did not handle the class imbalance issue in classification. Besides, they ignore the heterogeneity of the disease. To this end, we propose a novel cost-sensitive weighted contrastive learning method based on graph convolutional networks (CSWCL-GCNs) for imbalanced AD staging using resting-state functional magnetic resonance imaging (rs-fMRI). The proposed method is developed on a multi-view graph constructed by the functional connectivity (FC) and high-order functional connectivity (HOFC) features of the subjects. A novel cost-sensitive weighted contrastive learning procedure is proposed to capture discriminative information from the minority classes, encouraging the samples in the minority class to provide adequate supervision. Considering the heterogeneity of the disease, the weights of the negative pairs are introduced into contrastive learning and they are computed based on the distance to class prototypes, which are automatically learned from the training data. Meanwhile, the cost-sensitive mechanism is further introduced into contrastive learning to handle the class imbalance issue. The proposed CSWCL-GCN is evaluated on 720 subjects (including 184 NCs, 40 SMC patients, 208 EMCI patients, 172 LMCI patients and 116 AD patients) from the ADNI (Alzheimer's Disease Neuroimaging Initiative). Experimental results show that the proposed CSWCL-GCN outperforms state-of-the-art methods on the ADNI database.

Contrastive prototype network with prototype augmentation for few-shot classification

In recent years, metric-based meta-learning methods have received widespread attention because of their effectiveness in solving few-shot classification problems. However, the scarcity of data frequently results in suboptimal embeddings, causing a discrepancy between anticipated class prototypes and those derived from the support set. These problems severely limit the generalizability of such methods, necessitating further development of Few-Shot Learning (FSL). In this study, we propose the Contrastive Prototype Network (CPN) consisting of three components: (1) Contrastive learning proposed as an auxiliary path to reduce the distance between homogeneous samples and amplify the differences between heterogeneous samples, thereby enhancing the effectiveness and quality of embeddings; (2) A pseudo-prototype strategy proposed to address the bias in prototypes, whereby the pseudo prototypes generated using query set samples are integrated with the initial prototypes to obtain more representative prototypes; (3) A new data augmentation technique, mixupPatch, introduced to alleviate the issue of insufficient data samples, whereby enhanced images are generated by blending the images and labels from different samples, to increase the number of samples. Extensive experiments and ablation studies conducted on five datasets demonstrated that CPN achieves robust results against recent solutions.

OCIE: Augmenting model interpretability via Deconfounded Explanation-Guided Learning

Deep neural networks (DNNs) often encounter significant challenges related to opacity, inherent biases, and shortcut learning, which undermine their practical reliability. In this study, we address these issues by constructing a causal graph to model the unbiased learning process of DNNs. This model reveals that recurrent background information in training samples acts as a confounder, leading to spurious correlations between model inputs and outputs, causing biased predictions. To mitigate these problems and promote unbiased feature learning, we propose the Object-guided Consistency Interpretation Enhancement (OCIE) methodology. OCIE enhances DNN interpretability by integrating explicit objects and explanations into the model’s learning process. Initially, OCIE employs a graph-based algorithm to identify explicit objects within self-supervised vision transformer-learned features. Subsequently, it constructs class prototypes to eliminate invalid detected objects. Finally, OCIE aligns explanations with explicit objects, directing the model’s attention towards the most distinctive classification features rather than irrelevant backgrounds. Extensive experiments on different image classification datasets, including general (ImageNet), fine-grained (Stanford Cars and CUB-200), and medical (HAM) datasets, using two prevailing network architectures, demonstrate that OCIE significantly enhances explanation consistency across all datasets. Furthermore, OCIE proves particularly advantageous for fine-grained classification, especially in few-shot scenarios, by improving both interpretability and classification performance. Additionally, our findings highlight the impact of centralized explanations on the sufficiency of model decisions, suggesting that focusing explanations on explicit objects improves the reliability of DNN predictions. Our code is available at: https://github.com/DLAIResearch/OCIE.

Partial label learning via identifying outlier features

Partial label learning (PLL) is a classification problem where each training instance is ambiguously annotated with a set of candidate labels, among which only one is the ground truth label. The topological structure in the feature space is frequently used by existing PLL approaches to clarify if a candidate label is the ground truth label for a training example. However, these techniques frequently experience cumulative errors brought on by the error-prone labeling confidence estimate throughout the topological structure because of the redundant and noisy features occurring naturally in the feature space. In this paper, we propose a novel approach through Identifying Outlier Features (PL-IOF) to address this challenge. The feature space is decomposed into class prototype features and outlier features, where the former captures the discriminative features of each class prototype, and the latter captures the outlier features in each training instance. A unified framework is specifically suggested to simultaneously optimize class prototypes and outlier features, as well as to estimate labeling confidences over incomplete label training instances. This framework ensures the high quality of the extracted class prototypes by recognizing the outliers. Experiments on both synthetic and real-world datasets demonstrate the out-performance of PL-IOF over the state-of-the-art.

A prototype-assisted clustered federated learning for big data security and privacy preservation

In the rapidly expanding field of IoT, data production has reached an unprecedented scale, providing valuable insights that accelerate decision-making processes. However, ensuring the privacy and security of this massive amount of data poses significant challenges. In this paper, we propose using clustered federated learning (CFL) as a solution to ensure both the security and privacy of big data by uploading model weights while keeping the data stored locally. Nevertheless, there are practical challenges in applying CFL to big data: (1) the participating FL clients are unlikely to have identical data distributions; (2) insufficient attention is given to the similarity between different clusters; and (3) CFL tends to ignore the class imbalance problem (i.e., long-tailed), which hinders its application in big data and affects the quality of target tasks. To address these issues and enable widespread CFL deployment in big data applications, this paper proposes a prototype-assisted clustered federated learning framework (MDSPFL). It relaxes the assumption of unique data distribution for each client, allowing the client’s local dataset to follow multiple source distributions considering classification class imbalance, thereby aligning with clients in a big data environment. Specifically, MDSPFL employs the proximal update mechanism to handle workload surges caused by mixed distribution and unavailability of similarity between cluster models. Additionally, MDSPFL introduces a class-balanced local training mechanism to resolve the long-tailed problem, which utilizes contrastive learning and class prototypes to enforce a uniform distribution of all classes in the feature space. We conduct extensive experiments on different datasets (EMNIST, Cifar10, Cifar100), and the experimental results demonstrate the effectiveness of our proposed MDSPFL in big data scenarios with imbalance and mixed-distribution clients.

Reshaping Bioacoustics Event Detection: Leveraging Few-Shot Learning (FSL) with Transductive Inference and Data Augmentation.

Bioacoustic event detection is a demanding endeavor involving recognizing and classifying the sounds animals make in their natural habitats. Traditional supervised learning requires a large amount of labeled data, which are hard to come by in bioacoustics. This paper presents a few-shot learning (FSL) method incorporating transductive inference and data augmentation to address the issues of too few labeled events and small volumes of recordings. Here, transductive inference iteratively alters class prototypes and feature extractors to seize essential patterns, whereas data augmentation applies SpecAugment on Mel spectrogram features to augment training data. The proposed approach is evaluated by using the Detecting and Classifying Acoustic Scenes and Events (DCASE) 2022 and 2021 datasets. Extensive experimental results demonstrate that all components of the proposed method achieve significant F-score improvements of 27% and 10%, for the DCASE-2022 and DCASE-2021 datasets, respectively, compared to recent advanced approaches. Moreover, our method is helpful in FSL tasks because it effectively adapts to sounds from various animal species, recordings, and durations.

Unsupervised and semi-supervised domain adaptation networks considering both global knowledge and prototype-based local class information for Motor Imagery Classification

The non-stationarity of EEG signals results in variability across sessions, impeding model building and data sharing. In this paper, we propose a domain adaptation method called GPL, which simultaneously considers global knowledge and prototype-based local class information to enhance the classification accuracy of motor imagery signals. Depending on the amount of labeled data available in the target domain, the method is implemented in both unsupervised and semi-supervised versions. Specifically, at the global level, we employ the maximum mean difference (MMD) loss to globally constrain the feature space, achieving comprehensive alignment. In the context of class-level operations, we propose two memory banks designed to accommodate class prototypes in each domain and constrain feature embeddings by applying two prototype-based contrastive losses. The source contrastive loss is used to organize source features spatially based on categories, thereby reconciling inter-class and intra-class relationships, while the interactive contrastive loss is employed to facilitate cross-domain information interaction. Simultaneously, in unsupervised scenarios, to mitigate the adverse effects of excessive pseudo-labels, we introduce an entropy-aware strategy that dynamically evaluates the confidence level of target data and personalized constraints on the participation of interactive contrastive loss. To validate our approach, extensive experiments were conducted on a highly regarded public EEG dataset, namely Dataset IIa of the BCI Competition IV, as well as a large-scale EEG dataset called GigaDB. The experiments yielded average classification accuracies of 86.03% and 84.22% respectively. These results demonstrate that our method is an effective EEG decoding model, conducive to advancing the development of motor imagery brain–computer interfaces. The architecture proposed in this study and the code for data partitioning can be found at https://github.com/zhangdx21/GPL.

RDProtoFusion: Refined discriminative prototype-based multi-task fusion for cross-domain few-shot learning

Few-shot learning and meta-learning have been widely studied for their ability to reduce the burden of data annotation. However, real-world applications often involve training and target tasks from different source and target domains, leading to poor generalization of existing methods. To address this issue, we propose a novel method called RDProtoFusion, which leverages refined discriminative class prototypes to achieve task augmentation and bridge the domain shift. Firstly, we refine naive prototypes with query samples to avoid computation bias caused by the limited support size. Secondly, we perform multi-task fusion through class-representative prototypes, resulting in flexible task-augmentation with low complexity that effectively alleviates overfitting. Thirdly, we propose a prototype contrastive loss to obtain robust and accurate prototypes, by enhancing the discrimination between class prototypes. The proposed prototype refinement, fusion and contrast can improve generalization of the model effectively. We conduct extensive experiments on multiple cross-domain few-shot learning benchmarks, demonstrating that RDProtoFusion achieves state-of-the-art performance. Overall, our proposed method shows great potential for addressing the challenge of domain shift in few-shot learning, offering a promising avenue for real-world applications.

Towards interactive AI-authoring with prototypical few-shot classifiers in histopathology

A vast multitude of tasks in histopathology could potentially benefit from the support of artificial intelligence (AI). Many examples have been shown in the literature and first commercial products with FDA or CE-IVDR clearance are available. However, two key challenges remain: (1) a scarcity of thoroughly annotated images, respectively the laboriousness of this task, and (2) the creation of robust models that can cope with the data heterogeneity in the field (domain generalization). In this work, we investigate how the combination of prototypical few-shot classification models and data augmentation can address both of these challenges. Based on annotated data sets that include multiple centers, multiple scanners, and two tumor entities, we examine the robustness and the adaptability of few-shot classifiers in multiple scenarios. We demonstrate that data from one scanner and one site are sufficient to train robust few-shot classification models by applying domain-specific data augmentation. The models achieved classification performance of around 90% on a multiscanner and multicenter database, which is on par with the accuracy achieved on the primary single-center single-scanner data. Various convolutional neural network (CNN) architectures can be used for feature extraction in the few-shot model. A comparison of nine state-of-the-art architectures yielded that EfficientNet B0 provides the best trade-off between accuracy and inference time. The classification of prototypical few-shot models directly relies on class prototypes derived from example images of each class. Therefore, we investigated the influence of prototypes originating from images from different scanners and evaluated their performance also on the multiscanner database. Again, our few-shot model showed a stable performance with an average absolute deviation in accuracy compared to the primary prototypes of 1.8% points. Finally, we examined the adaptability to a new tumor entity: classification of tissue sections containing urothelial carcinoma into normal, tumor, and necrotic regions. Only three annotations per subclass (e.g., muscle and adipose tissue are subclasses of normal tissue) were provided to adapt the few-shot model, which obtained an overall accuracy of 93.6%. These results demonstrate that prototypical few-shot classification is an ideal technology for realizing an interactive AI authoring system as it only requires few annotations and can be adapted to new tasks without involving retraining of the underlying feature extraction CNN, which would in turn require a selection of hyper-parameters based on data science expert knowledge. Similarly, it can be regarded as a guided annotation system. To this end, we realized a workflow and user interface that targets non-technical users.

Global prototype distillation for heterogeneous federated learning.

Federated learning is a distributed machine learning paradigm where the goal is to collaboratively train a high quality global model while private training data remains local over distributed clients. However, heterogenous data distribution over clients is severely challenging for federated learning system, which severely damage the quality of model. In order to address this challenge, we propose global prototype distillation (FedGPD) for heterogenous federated learning to improve performance of global model. The intuition is to use global class prototypes as knowledge to instruct local training on client side. Eventually, local objectives will be consistent with the global optima so that FedGPD learns an improved global model. Experiments show that FedGPD outperforms previous state-of-art methods by 0.22% ~1.28% in terms of average accuracy on representative benchmark datasets.

Focus on informative graphs! Semi-supervised active learning for graph-level classification

Graph-level classification is a critical problem in social analysis and bioinformatics. Since annotated labels are typically costly, we intend to study this challenging task in semi-supervised scenarios with limited budgets. Inspired by the fact that active learning is capable of interactively querying an oracle to annotate a small number of informative examples in the unlabeled dataset, we develop a novel Semi-supervised active learning framework termed GraphSpa for graph-level classification. To make the most of labeling budgets, we propose an effective unlabeled data selection strategy that takes both local similarity and global semantic structure into account. Specifically, we first construct an adaptive queue with labeled samples and select informative samples that have a low degree of similarity to the queue using the Min-Max principle from the local view. Further, we introduce class prototypes and select samples with a large predictive loss discrepancy from the global view. To harness the full potential of unlabeled data, we develop a semi-supervised active learning framework on the basis of our fusion selection strategy coupled with graph contrastive learning during active learning. The effectiveness of our GraphSpa is validated against state-of-the-art methods through experimental results on diverse real-world benchmark datasets.