Prototype-based Classification Research Articles

Frozen is better than learning: A new design of prototype-based classifier for semantic segmentation

Semantic segmentation models comprise an encoder to extract features and a classifier for prediction. However, the learning of the classifier suffers from the ambiguity which is caused by two factors: (1) the weights of a classifier for similar categories may have positive similarities lowing the performance for similar categories, named correlation ambiguity, and (2) the classifier is prone to predict the category with a larger ℓ2 norm and vice versa, termed prior ambiguity. To comedy the issues, we propose Category-Basis Prototype (CBP), frozen and mutually orthogonalized prototypes with equalℓ2norm. Orthogonalization prevents the prototypes from being similar to each other and the equality decouples the prediction from the ℓ2 norm. To better shape the feature space, we propose Online Centroid Contrastive Loss (OCCL) equipped with centroid and category-level losses. Experiments show that our method yields compelling results over two widely applied benchmarks indicating the effectiveness of our methods.

Introspective GAN: Learning to grow a GAN for incremental generation and classification

Lifelong learning, the ability to continually learn new concepts throughout our life, is a hallmark of human intelligence. Generally, humans learn a new concept by knowing what it looks like and what makes it different from the others, which are correlated. Those two ways can be characterized by generation and classification in machine learning respectively. In this paper, we carefully design a dynamically growing GAN called Introspective GAN (IntroGAN) that can perform incremental generation and classification simultaneously with the guidance of prototypes, inspired by their roles of efficient information organization in human visual learning and excellent performance in other fields like zero-shot/few-shot/incremental learning. Specifically, we incorporate prototype-based classification which is robust to feature change in incremental learning and GAN as a generative memory to alleviate forgetting into a unified end-to-end framework. A comprehensive benchmark on the joint incremental generation and classification task is proposed and our method demonstrates promising results. Additionally, we conduct comprehensive analyses over the properties of IntroGAN and verify that generation and classification can be mutually beneficial in incremental scenarios, which is an inspiring area to be further exploited. The code is available at https://github.com/TonyPod/IntroGAN.

An Interpretable and Accurate Deep-Learning Diagnosis Framework Modeled With Fully and Semi-Supervised Reciprocal Learning.

The deployment of automated deep-learning classifiers in clinical practice has the potential to streamline the diagnosis process and improve the diagnosis accuracy, but the acceptance of those classifiers relies on both their accuracy and interpretability. In general, accurate deep-learning classifiers provide little model interpretability, while interpretable models do not have competitive classification accuracy. In this paper, we introduce a new deep-learning diagnosis framework, called InterNRL, that is designed to be highly accurate and interpretable. InterNRL consists of a student-teacher framework, where the student model is an interpretable prototype-based classifier (ProtoPNet) and the teacher is an accurate global image classifier (GlobalNet). The two classifiers are mutually optimised with a novel reciprocal learning paradigm in which the student ProtoPNet learns from optimal pseudo labels produced by the teacher GlobalNet, while GlobalNet learns from ProtoPNet's classification performance and pseudo labels. This reciprocal learning paradigm enables InterNRL to be flexibly optimised under both fully- and semi-supervised learning scenarios, reaching state-of-the-art classification performance in both scenarios for the tasks of breast cancer and retinal disease diagnosis. Moreover, relying on weakly-labelled training images, InterNRL also achieves superior breast cancer localisation and brain tumour segmentation results than other competing methods.

Semi-supervised remote sensing image scene classification with prototype-based consistency

Deep learning significantly improves the accuracy of remote sensing image scene classification, benefiting from the large-scale datasets. However, annotating the remote sensing images is time-consuming and even tough for experts. Deep neural networks trained using a few labeled samples usually generalize less to new unseen images. In this paper, we propose a semi-supervised approach for remote sensing image scene classification based on the prototype-based consistency, by exploring massive unlabeled images. To this end, we, first, propose a feature enhancement module to extract discriminative features. This is achieved by focusing the model on the foreground areas. Then, the prototype-based classifier is introduced to the framework, which is used to acquire consistent feature representations. We conduct a series of experiments on NWPU-RESISC45 and Aerial Image Dataset (AID). Our method improves the State-Of-The-Art (SOTA) method on NWPU-RESISC45 from 92.03% to 93.08% and on AID from 94.25% to 95.24% in terms of accuracy.

Classification of DNA Sequence Based on a Non-gradient Algorithm: Pseudoinverse Learners.

This chapter proposes a prototype-based classification approach for analyzing DNA barcodes that uses a spectral representation of DNA sequences and a non-gradient neural network. Biological sequences can be viewed as data components with higher non-fixed dimensions, which correspond to the length of the sequences. Through computational procedures such as one-hot encoding, numerical encoding plays an important role in DNA sequence evaluation (OHE). However, the OHE method has some disadvantages: (1) It does not add any details that could result in an additional predictive variable, and (2) if the variable has many classes, OHE significantly expands the feature space. To address these shortcomings, this chapter proposes a computationally efficient framework for classifying DNA sequences of living organisms in the image domain. A multilayer perceptron trained by a pseudoinverse learning autoencoder (PILAE) algorithm is used in the proposed strategy. The learning control parameters and the number of hidden layers do not have to be specified during the PILAE training process. As a result, the PILAE classifier outperforms other deep neural network (DNN) strategies such as the VGG-16 and Xception models.

Autoencoder-Enhanced Regularized Prototypical Network for New Energy Vehicle battery fault detection

As the ownership of new energy vehicles (NEVs) is experiencing a sustained growth, the safety of NEVs has become increasingly prominent, with power battery faults emerging as the primary cause of fire accidents in NEVs. Successful detection of incipient faults can not only improve the safety and reliability but also provide optimal maintenance instructions for NEV batteries. Since NEVs operate under normal conditions most of the time, the occurrence of system faults leading to faulty samples is extremely rare, thus exhibiting the characteristics of a long-tail distribution. To resolve this problem, this paper proposes an Autoencoder-Enhanced Regularized Prototypical Network (ARPN) to overcome the limitations of class imbalance and uneven sample distribution in the data. The model consists of three key components. First, an autoencoder with an encoder and decoder structure is introduced to map the data to a low-dimensional latent space and back, effectively extracting key structural information from the original data and make full use of the limited number of faulty training samples. Secondly, a multi-layer regularized feature embedding strategy is designed to reduce the complexity of the embedding space, which makes the prototype vector get easier to be separated and distinguished and plays an important role in the proposed ARPN. Finally, a prototype-based classification method is adopted to achieve accurate classification by calculating the cosine similarity between the representation of query set samples in the embedding space and the prototype vectors of each class. Experiments on several battery datasets, obtained from the operational data of NEVs, demonstrate the effectiveness of the proposed method.

Taste Bud-Inspired Single-Drop Multitaste Sensing for Comprehensive Flavor Analysis with Deep Learning Algorithms.

The electronic tongue (E-tongue) system has emerged as a significant innovation, aiming to replicate the complexity of human taste perception. In spite of the advancements in E-tongue technologies, two primary challenges remain to be addressed. First, evaluating the actual taste is complex due to interactions between taste and substances, such as synergistic and suppressive effects. Second, ensuring reliable outcomes in dynamic conditions, particularly when faced with high deviation error data, presents a significant challenge. The present study introduces a bioinspired artificial E-tongue system that mimics the gustatory system by integrating multiple arrays of taste sensors to emulate taste buds in the human tongue and incorporating a customized deep-learning algorithm for taste interpretation. The developed E-tongue system is capable of detecting four distinct tastes in a single drop of dietary compounds, such as saltiness, sourness, astringency, and sweetness, demonstrating notable reversibility and selectivity. The taste profiles of six different wines are obtained by the E-tongue system and demonstrated similarities in taste trends between the E-tongue system and user reviews from online, although some disparities still exist. To mitigate these disparities, a prototype-based classifier with soft voting is devised and implemented for the artificial E-tongue system. The artificial E-tongue system achieved a high classification accuracy of ∼95% in distinguishing among six different wines and ∼90% accuracy even in an environment where more than 1/3 of the data contained errors. Moreover, by harnessing the capabilities of deep learning technology, a recommendation system was demonstrated to enhance the user experience.

Lymphoma Recognition in Histology Image of Gastric Mucosal Biopsy with Prototype Learning.

Lymphomas are a group of malignant tumors developed from lymphocytes, which may occur in many organs. Therefore, accurately distinguishing lymphoma from solid tumors is of great clinical significance. Due to the strong ability of graph structure to capture the topology of the micro-environment of cells, graph convolutional networks (GCNs) have been widely used in pathological image processing. Nevertheless, the softmax classification layer of the graph convolutional models cannot drive learned representations compact enough to distinguish some types of lymphomas and solid tumors with strong morphological analogies on H&E-stained images. To alleviate this problem, a prototype learning based model is proposed, namely graph convolutional prototype network (GCPNet). Specifically, the method follows the patch-to-slide architecture first to perform patch-level classification and obtain image-level results by fusing patch-level predictions. The classification model is assembled with a graph convolutional feature extractor and prototype-based classification layer to build more robust feature representations for classification. For model training, a dynamic prototype loss is proposed to give the model different optimization priorities at different stages of training. Besides, a prototype reassignment operation is designed to prevent the model from getting stuck in local minima during optimization. Experiments are conducted on a dataset of 183 Whole slide images (WSI) of gastric mucosa biopsy. The proposed method achieved superior performance than existing methods.Clinical relevance- The work proposed a new deep learning framework tailored to lymphoma recognition on pathological image of gastric mucosal biopsy to differentiate lymphoma, adenocarcinoma and inflammation.

A Framework for Frugal Supervised Learning with Incremental Neural Networks

This study proposes an implementation of an incremental neural network (INN) that was initially designed for affective computing tasks. INNs are a family of machine learning algorithms that combine prototype-based classifiers with neural networks. They achieve state-of-the-art performance with less data than traditional approaches. In this research, we conduct an in-depth review of INN mechanisms and present a research-grade framework that enables the use of INNs on arbitrary data. We evaluated our implementation on two different datasets, including the AVEC2014 Challenge, which involved predicting depressive state from auditive and visual modalities. Our results are encouraging, demonstrating the potential of INNs in situations where approaches have to be explainable or when data are scarce.

An efficient digital pulse processing approach for identification of BGO and LSO scintillator crystals

Identification and discrimination between BGO and LSO scintillators is a fundamental target for handling parallax error within positron emission tomography (PET) applications by depth of interaction. An approach is built for discrimination and identification of BGO and LSO scintillator crystals. This approach is tested using a simulated BGO and LSO pulses. A Matlab Simulink model is implemented for simulation and creation of BGO and LSO scintillation pulses. The simulated pulses depend on 22Na radiation source. The suggested approach has two different algorithms. The first algorithm uses both 1D-Walsh ordered fast Walsh-Hadamard transform (1WFWHT) and fast Chebyshev transform (FCHT) for extracting the features of crystal pulses. The optimum features are selected from 1WFWHT and FCHT using one of three optimization techniques that are binary dragonfly optimization (BDA), binary atom search optimization (BASO) and binary Harris Hawk optimization (BHHO). These optimized features are trained and tested using one of three based classifiers. These classifiers are Naive Bayes classifier (NBC), hierarchical prototype-based (HP) classifier and adaptive neuro-fuzzy inference system (ANFIS) classification. The ANFIS classifier achieves the best accuracy with all optimum (BASO, BDF and BHH) FCHT features. However, the NB classifier introduces the highest accuracy with all optimum (BASO, BDF and BHH) FWHT 1D features. The second algorithm uses the conventional neural network (CNN) for extracting the pulse features. Then, the deep neural network (DNN) is applied for training and testing of the captured pulses. The suggested algorithms are verified and compared in respect of statistical measures. The compared results confirm that the best accuracy and identification rate is accomplished using DNN algorithm. Besides, the DNN has better results compared to conventional classification techniques with optimum feature selection techniques in respect of time consumption. The proposed approach aids in the realisation of overcoming parallax inaccuracy in PET.

Prototype Learning for Automatic Check-Out

The basic goal of Automatic Check-Out (ACO) task is to accurately predict the categories and quantities of products selected by customers in the check-out images. However, there is a significant domain gap between the single-product exemplars as training data and the check-out images as testing data. To mitigate the domain gap, we propose a novel method termed as Prototype Learning for Automatic Check-Out (PLACO). In PLACO, prototype learning is designed to reach the goal in two ways. Specifically, in the prototype-based classifier learning module, to fully exploit the invariance of category prototypes, the prototypes obtained from the single-product exemplars are employed to generate classifiers for classifying the proposals of check-out image. On the other side, in prototype alignment module, prototypes for both the single-product exemplar and check-out image domains are entered simultaneously to ensure intra-category compactness and inter-category sparsity. Moreover, to further improve the performance of PLACO, we develop a discriminative re-ranking module to both adjust the predicted scores of product proposals for bringing more discriminative ability in classifier learning and provide a reasonable sorting possibility by considering the fine-grained nature. Experiments are conducted on the large-scale RPC dataset for evaluations. Our PLACO obtains the optimal results in both traditional ACO task setting and incremental task setting.

Open Self-Supervised Features for Remote-Sensing Image Scene Classification Using Very Few Samples

Big models, large datasets, and self-supervised learning (SSL) have recently gained substantial research interest due to their potential to alleviate our reliance on annotations. Considering the current high generalization ability of self-supervised models in literature, we explore in the letter how helpful SSL can be for a crucial task in remote sensing (RS), image scene classification, when forced to rely on only a few labeled samples. We proposed a simple prototype-based classification procedure without training and fine-tuning, which uses open self-supervised features from the contrastive language-image pre-training (CLIP). We test our method by exploiting ready-to-use open features on four diversified benchmark datasets, including red-green-blue (RGB) and multispectral (MS) images. Highly competitive accuracy has been obtained compared to work with similar settings, i.e., based on an exceedingly small number of labels. To the best of our knowledge, our model is the first to achieve such high accuracy in austere label conditions. We further analyze our approach from different perspectives, including its advantages and limitations, reasons for its astonishing performance, potential applications, and future improvements.

Fuzzy prototype selection-based classifiers for imbalanced data. Case study

• Two classification methods based on prototype selection are presented. • These methods are based on fuzzy similarity relations as a more flexible alternative. • Proposed methods demonstrate good performance on imbalanced datasets. • These methods allow instance reduction in the learning process. • Prototypes help explain and interpret events. Imbalanced data are popular in the machine learning community due to their likelihood of appearing in real-world application areas and the problems they present for classical classifiers. The goal of this work is to extend the capabilities of prototype-based classifiers using fuzzy similarity relations and to make them sensitive to class-imbalanced data classification. This paper proposes two new fuzzy logic-based prototype selection classifiers for imbalanced datasets, Imb-SPBASIR-Fuzzy_V1 (FPS-v1) and Imb-SPBASIR-Fuzzy_V2 (FPS-v1), and shows a comparative study of them with state-of-the-art methods on public datasets from the UCI machine learning repository. The results on the selected datasets suggest that fuzzy logic-based prototype selection classifiers perform well and efficiently, indicating that it is a viable alternative. The fuzzy relationships provided by this approach allow better results than the state-of-the-art models. Further analysis showed that the proposed fuzzy-based prototypes methods permit obtaining more accurate to deal with the correct prophylaxis, timely diagnosis and treatment of postoperative mediastinitis.

Model and Training Method of the Resilient Image Classifier Considering Faults, Concept Drift, and Adversarial Attacks

Modern trainable image recognition models are vulnerable to different types of perturbations; hence, the development of resilient intelligent algorithms for safety-critical applications remains a relevant concern to reduce the impact of perturbation on model performance. This paper proposes a model and training method for a resilient image classifier capable of efficiently functioning despite various faults, adversarial attacks, and concept drifts. The proposed model has a multi-section structure with a hierarchy of optimized class prototypes and hyperspherical class boundaries, which provides adaptive computation, perturbation absorption, and graceful degradation. The proposed training method entails the application of a complex loss function assembled from its constituent parts in a particular way depending on the result of perturbation detection and the presence of new labeled and unlabeled data. The training method implements principles of self-knowledge distillation, the compactness maximization of class distribution and the interclass gap, the compression of feature representations, and consistency regularization. Consistency regularization makes it possible to utilize both labeled and unlabeled data to obtain a robust model and implement continuous adaptation. Experiments are performed on the publicly available CIFAR-10 and CIFAR-100 datasets using model backbones based on modules ResBlocks from the ResNet50 architecture and Swin transformer blocks. It is experimentally proven that the proposed prototype-based classifier head is characterized by a higher level of robustness and adaptability in comparison with the dense layer-based classifier head. It is also shown that multi-section structure and self-knowledge distillation feature conserve resources when processing simple samples under normal conditions and increase computational costs to improve the reliability of decisions when exposed to perturbations.

Learning to Capture the Query Distribution for Few-Shot Learning

In the Few-Shot Learning (FSL), much of the related efforts only rely on the few available labeled samples (support set) building approach. However, the challenge is that the support set is easy-to-be-biased, so that they cannot be competent prototypes and are hard to represent the class distribution, leading to performance bottlenecks. In this paper, we propose to solve this obstacle by capturing the distribution of the unlabeled samples (query set). We propose two sampling methods: DeepSearch (<inline-formula> <tex-math notation="LaTeX">$\cal DS$ </tex-math></inline-formula>) and WideSearch (<inline-formula> <tex-math notation="LaTeX">$\cal WS$ </tex-math></inline-formula>). Both approaches are simple to implement and have no trainable parameters. They search the query samples near to the support set in different manners. Afterward, the statistic information is calculated, and we generate the latent samples according to it. The generated latent set is promising. First, it brings the query set distribution information to the classifier, which significantly improves the performance of the cross-entropy-based classifier. Second, it helps the support set become the better prototypes, which boosts the performance of the prototype-based classifier. Third, we find few latent samples are enough to boost the performance. Abundant experiments prove the proposed method achieves state-of-the-art performance on the few-shot tasks. Finally, rich ablation studies explain the compelling details of our approach.

Prototype-based classifier learning for long-tailed visual recognition

In this paper, we tackle the long-tailed visual recognition problem from the categorical prototype perspective by proposing a prototype-based classifier learning (PCL) method. Specifically, thanks to the generalization ability and robustness, categorical prototypes reveal their advantages of representing the category semantics. Coupled with their class-balance characteristic, categorical prototypes also show potential for handling data imbalance. In our PCL, we propose to generate the categorical classifiers based on the prototypes by performing a learnable mapping function. To further alleviate the impact of imbalance on classifier generation, two kinds of classifier calibration approaches are designed from both prototype-level and example-level aspects. Extensive experiments on five benchmark datasets, including the large-scale iNaturalist, Places-LT, and ImageNet-LT, justify that the proposed PCL can outperform state-of-the-arts. Furthermore, validation experiments can demonstrate the effectiveness of tailored designs in PCL for long-tailed problems.

Self-organizing Divisive Hierarchical Voronoi Tessellation-based classifier

In this paper, a novel approach to the self-organization of hierarchical prototype-based classifiers from data is proposed. The approach recursively partitions the data at multiple levels of granularity into shape-free clusters of different sizes, resembling Voronoi tessellation, and naturally aggregates the resulting cluster medoids into a multi-layered prototype-based structure according to their descriptive abilities. Different from conventional classification models, it is nonparametric and entirely data-driven, and the learned model can offer a high-level of transparency and interpretability thanks to the underlying prototype-based nature. The system identification process underpinning the approach is driven by the aim of separating data samples of different classes into nonoverlapping multi-granular clusters. Its associated decision-making process follows the “nearest prototype” principle and hence, the rationales of the subsequent decisions made can be explicitly explained. Experimental studies based on popular benchmark classification problems, as well as on a practical application to remote sensing image classification, demonstrate the efficacy of the proposed approach.

A Self-Training Hierarchical Prototype-based Ensemble Framework for Remote Sensing Scene Classification

Remote sensing scene classification plays a critical role in a wide range of real-world applications. Technically, however, scene classification is an extremely challenging task due to the huge complexity in remotely sensed scenes, and the difficulty in acquiring labelled data for model training such as supervised deep learning. To tackle these issues, a novel semi-supervised ensemble framework is proposed here using the self-training hierarchical prototype-based classifier as the base learner for chunk-by-chunk prediction. The framework has the ability to build a powerful ensemble model from both labelled and unlabelled images with minimum supervision. Different feature descriptors are employed in the proposed ensemble framework to offer multiple independent views of images. Thus, the diversity of base learners is guaranteed for ensemble classification. To further increase the overall accuracy, a novel cross-checking strategy was introduced to enable the base learners to exchange pseudo-labelling information during the self-training process, and maximize the correctness of pseudo-labels assigned to unlabelled images. Extensive numerical experiments on popular benchmark remote sensing scenes demonstrated the effectiveness of the proposed ensemble framework, especially where the number of labelled images available is limited. For example, the classification accuracy achieved on the OPTIMAL-31, PatternNet and RSI-CB256 datasets was up to 99.91%, 98. 67% and 99.07% with only 40% of the image sets used as labelled training images, surpassing or at least on par with mainstream benchmark approaches trained with double the number of labelled images.

Efficient computation of counterfactual explanations and counterfactual metrics of prototype-based classifiers

The increasing use of machine learning in practice and legal regulations like EU’s GDPR cause the necessity to be able to explain the prediction and behavior of machine learning models. A prominent example of particularly intuitive explanations of AI models in the context of decision making are counterfactual explanations. Yet, it is still an open research problem how to efficiently compute counterfactual explanations for many models. In this contribution, we investigate how to efficiently compute counterfactual explanations for an important class of models, prototype-based classifiers such as learning vector quantization models. In particular, we derive specific convex and non-convex programs depending on the used metric. Typically counterfactual explanations deliver a feedback in terms of changes of the input features which lead to a different output – one application scenario is the link of these required changes to actionable items to change the desired outcome. Yet, rather than minimum changes of the input, it is interesting to address minimum changes of the model itself, which are required to lead to a different result. rather than a change of its inputs. We phrase this question as a counterfactual of the model prescription rather than the data points. We focus on distance-based classifiers (in particular learning vector quantization models), where model changes correspond to changes of metric parameters, and we develop efficient optimization techniques to generate such counterfactual metric changes depending on the chosen model.

AI-Based Multi Sensor Fusion for Smart Decision Making: A Bi-Functional System for Single Sensor Evaluation in a Classification Task.

Sensor fusion has gained a great deal of attention in recent years. It is used as an application tool in many different fields, especially the semiconductor, automotive, and medical industries. However, this field of research, regardless of the field of application, still presents different challenges concerning the choice of the sensors to be combined and the fusion architecture to be developed. To decrease application costs and engineering efforts, it is very important to analyze the sensors’ data beforehand once the application target is defined. This pre-analysis is a basic step to establish a working environment with fewer misclassification cases and high safety. One promising approach to do so is to analyze the system using deep neural networks. The disadvantages of this approach are mainly the required huge storage capacity, the big training effort, and that these networks are difficult to interpret. In this paper, we focus on developing a smart and interpretable bi-functional artificial intelligence (AI) system, which has to discriminate the combined data regarding predefined classes. Furthermore, the system can evaluate the single source signals used in the classification task. The evaluation here covers each sensor contribution and robustness. More precisely, we train a smart and interpretable prototype-based neural network, which learns automatically to weight the influence of the sensors for the classification decision. Moreover, the prototype-based classifier is equipped with a reject option to measure classification certainty. To validate our approach’s efficiency, we refer to different industrial sensor fusion applications.