Prototype Learning Research Articles

FoPro: Few-Shot Guided Robust Webly-Supervised Prototypical Learning

Recently, webly supervised learning (WSL) has been studied to leverage numerous and accessible data from the Internet. Most existing methods focus on learning noise-robust models from web images while neglecting the performance drop caused by the differences between web domain and real-world domain. However, only by tackling the performance gap above can we fully exploit the practical value of web datasets. To this end, we propose a Few-shot guided Prototypical (FoPro) representation learning method, which only needs a few labeled examples from reality and can significantly improve the performance in the real-world domain. Specifically, we initialize each class center with few-shot real-world data as the ``realistic" prototype. Then, the intra-class distance between web instances and ``realistic" prototypes is narrowed by contrastive learning. Finally, we measure image-prototype distance with a learnable metric. Prototypes are polished by adjacent high-quality web images and involved in removing distant out-of-distribution samples. In experiments, FoPro is trained on web datasets with a few real-world examples guided and evaluated on real-world datasets. Our method achieves the state-of-the-art performance on three fine-grained datasets and two large-scale datasets. Compared with existing WSL methods under the same few-shot settings, FoPro still excels in real-world generalization. Code is available at https://github.com/yuleiqin/fopro.

On Error and Compression Rates for Prototype Rules

We study the close interplay between error and compression in the non-parametric multiclass classification setting in terms of prototype learning rules. We focus in particular on a recently proposed compression-based learning rule termed OptiNet. Beyond its computational merits, this rule has been recently shown to be universally consistent in any metric instance space that admits a universally consistent rule---the first learning algorithm known to enjoy this property. However, its error and compression rates have been left open. Here we derive such rates in the case where instances reside in Euclidean space under commonly posed smoothness and tail conditions on the data distribution. We first show that OptiNet achieves non-trivial compression rates while enjoying near minimax-optimal error rates. We then proceed to study a novel general compression scheme for further compressing prototype rules that locally adapts to the noise level without sacrificing accuracy. Applying it to OptiNet, we show that under a geometric margin condition further gain in the compression rate is achieved. Experimental results comparing the performance of the various methods are presented.

Head-Free Lightweight Semantic Segmentation with Linear Transformer

Existing semantic segmentation works have been mainly focused on designing effective decoders; however, the computational load introduced by the overall structure has long been ignored, which hinders their applications on resource-constrained hardwares. In this paper, we propose a head-free lightweight architecture specifically for semantic segmentation, named Adaptive Frequency Transformer (AFFormer). AFFormer adopts a parallel architecture to leverage prototype representations as specific learnable local descriptions which replaces the decoder and preserves the rich image semantics on high-resolution features. Although removing the decoder compresses most of the computation, the accuracy of the parallel structure is still limited by low computational resources. Therefore, we employ heterogeneous operators (CNN and vision Transformer) for pixel embedding and prototype representations to further save computational costs. Moreover, it is very difficult to linearize the complexity of the vision Transformer from the perspective of spatial domain. Due to the fact that semantic segmentation is very sensitive to frequency information, we construct a lightweight prototype learning block with adaptive frequency filter of complexity O(n) to replace standard self attention with O(n^2). Extensive experiments on widely adopted datasets demonstrate that AFFormer achieves superior accuracy while retaining only 3M parameters. On the ADE20K dataset, AFFormer achieves 41.8 mIoU and 4.6 GFLOPs, which is 4.4 mIoU higher than Segformer, with 45% less GFLOPs. On the Cityscapes dataset, AFFormer achieves 78.7 mIoU and 34.4 GFLOPs, which is 2.5 mIoU higher than Segformer with 72.5% less GFLOPs. Code is available at https://github.com/dongbo811/AFFormer.

Generalized Category Discovery with Decoupled Prototypical Network

Generalized Category Discovery (GCD) aims to recognize both known and novel categories from a set of unlabeled data, based on another dataset labeled with only known categories. Without considering differences between known and novel categories, current methods learn about them in a coupled manner, which can hurt model's generalization and discriminative ability. Furthermore, the coupled training approach prevents these models transferring category-specific knowledge explicitly from labeled data to unlabeled data, which can lose high-level semantic information and impair model performance. To mitigate above limitations, we present a novel model called Decoupled Prototypical Network (DPN). By formulating a bipartite matching problem for category prototypes, DPN can not only decouple known and novel categories to achieve different training targets effectively, but also align known categories in labeled and unlabeled data to transfer category-specific knowledge explicitly and capture high-level semantics. Furthermore, DPN can learn more discriminative features for both known and novel categories through our proposed Semantic-aware Prototypical Learning (SPL). Besides capturing meaningful semantic information, SPL can also alleviate the noise of hard pseudo labels through semantic-weighted soft assignment. Extensive experiments show that DPN outperforms state-of-the-art models by a large margin on all evaluation metrics across multiple benchmark datasets. Code and data are available at https://github.com/Lackel/DPN.

Doodle to Object: Practical Zero-Shot Sketch-Based 3D Shape Retrieval

Zero-shot (ZS) sketch-based three-dimensional (3D) shape retrieval (SBSR) is challenging due to the abstraction of sketches, cross-domain discrepancies between two-dimensional sketches and 3D shapes, and ZS-driven semantic knowledge transference from seen to unseen categories. Extant SBSR datasets suffer from lack of data, and no current SBSR methods consider ZS scenarios. In this paper, we contribute a new Doodle2Object (D2O) dataset consisting of 8,992 3D shapes and over 7M sketches spanning 50 categories. Then, we propose a novel prototype contrastive learning (PCL) method that effectively extracts features from different domains and adapts them to unseen categories. Specifically, our PCL method combines the ideas of contrastive and cluster-based prototype learning, and several randomly selected prototypes of different classes are assigned to each sample. By comparing these prototypes, a given sample can be moved closer to the same semantic class of samples while moving away from negative ones. Extensive experiments on two common SBSR benchmarks and our D2O dataset demonstrate the efficacy of the proposed PCL method for ZS-SBSR. Resource is available at https://github.com/yigohw/doodle2object.

ContraFeat: Contrasting Deep Features for Semantic Discovery

StyleGAN has shown strong potential for disentangled semantic control, thanks to its special design of multi-layer intermediate latent variables. However, existing semantic discovery methods on StyleGAN rely on manual selection of modified latent layers to obtain satisfactory manipulation results, which is tedious and demanding. In this paper, we propose a model that automates this process and achieves state-of-the-art semantic discovery performance. The model consists of an attention-equipped navigator module and losses contrasting deep-feature changes. We propose two model variants, with one contrasting samples in a binary manner, and another one contrasting samples with learned prototype variation patterns. The proposed losses are computed with pretrained deep features, based on our assumption that the features implicitly possess the desired semantic variation structure including consistency and orthogonality. Additionally, we design two metrics to quantitatively evaluate the performance of semantic discovery methods on FFHQ dataset, and also show that disentangled representations can be derived via a simple training process. Experimentally, we show that our models achieve state-of-the-art semantic discovery results without relying on layer-wise manual selection, and these discovered semantics can be used to manipulate real-world images.

POEM: A prototype cross and emphasis network for few-shot semantic segmentation

To tackle the demand for large labeled datasets and poor unseen class generalization for segmentation tasks, few-shot segmentation (FSS) has received widespread attention, which utilizes prototypical learning to achieve significant progress. However, the prototypes generated by averaging the global object information still suffer from the appearance diversity within the target class. Motivated by the fact that humans only need part attributes rather than all attributions to recognize objects from a particular class, we propose an effective prototype cross and emphasis segmentation network (POEM) to provide more precise guidance in the feature matching process. Firstly, we present a regional crossed referencing module to improve the prototypical learning strategy, which mines the relevant attributes of the target objects in both support and query images and customizes the prototypes for the query images. Secondly, we develop a foreground-emphasized reasoning module to reduce the negative influence caused by complex background clutter, in which the foreground and background attributions are regarded as fundamental information and auxiliary information, respectively. Extensive experiments on two standard datasets have demonstrated the superiority of POEM over the state-of-the-art methods.

An Aggregated Convolutional Transformer Based on Slices and Channels for Multivariate Time Series Classification

Convolutional neural network has achieved remarkable success, and has excellent local feature extraction ability. Similarly, Transformer has been developed markedly in recent years, achieving excellent representation capabilities in terms of global features, which has aroused heated discussions. In terms of multivariate time series classification, most previous networks had convolution and long and short-term memory structures. This paper innovatively proposes a combination of Transformer-encoder and convolutional structures, which we refer to as the Multivariate time series classification Convolutional Transformer Network (MCTNet). The different advantages of convolutional neural network and self-attention are used to capture potential deep information in multivariate time series more accurately. The Transformer is considered to be data-hungry, and combined with the induction bias of the convolutional neural network to solve this problem, early features are extracted through the convolutional layers, and the both squeeze and excitation convolution encoder (BC-Encoder) structure is proposed. Attentional prototype learning is also used to mitigate the limited label problem. Moreover, a new network design that focuses on slices and channels is proposed, moving beyond the concept that the use of Transformer will require many parameters. Experimental results from 26 datasets of the well-known multivariate time series archive UEA show that the performance of our model is better than that of most state-of-the-art models.

A Review on Age and Gender Recognition Using Various Datasets and Deep Learning Models

Abstract: Age and Gender are two facial potentials play a principalrole in the society. An instinctive age and gender appreciation have a massive numeral of real-world uses that comprises a consumer facility, the priority voting system, medical diagnosis, the human computer communication. Deep learning methods are usually used in utmost researches and attained to progress the performance too. Relating dissimilar deep education models and evaluating the enhancement in exactness leads to complementary studies. The chief aim of this paper is to achieve detailed examination of age and gender credentials through numerous datasets and deep learning prototypes. In this paper, the growth made by, deep learning prototypes are highlighting the assistances addressed; the prototypes and dataset used are estimating the acclaimed method with the concerns attained.

A Novel Method for Recognizing Space Radiation Sources Based on Multi-Scale Residual Prototype Learning Network

As a basic task and key link of space situational awareness, space target recognition has become crucial in threat analysis, communication reconnaissance and electronic countermeasures. Using the fingerprint features carried by the electromagnetic signal to recognize is an effective method. Because traditional radiation source recognition technologies are difficult to obtain satisfactory expert features, automatic feature extraction methods based on deep learning have become popular. Although many deep learning schemes have been proposed, most of them are only used to solve the inter-class separable problem and ignore the intra-class compactness. In addition, the openness of the real space may invalidate the existing closed-set recognition methods. In order to solve the above problems, inspired by the application of prototype learning in image recognition, we propose a novel method for recognizing space radiation sources based on a multi-scale residual prototype learning network (MSRPLNet). The method can be used for both the closed- and open-set recognition of space radiation sources. Furthermore, we also design a joint decision algorithm for an open-set recognition task to identify unknown radiation sources. To verify the effectiveness and reliability of the proposed method, we built a set of satellite signal observation and receiving systems in a real external environment and collected eight Iridium signals. The experimental results show that the accuracy of our proposed method can reach 98.34% and 91.04% for the closed- and open-set recognition of eight Iridium targets, respectively. Compared to similar research works, our method has obvious advantages.

Few-shot learning for seismic facies segmentation via prototype learning

The mapping of seismic facies from seismic data is considered a multiclass image semantic segmentation problem. Despite the signification progress made by the deep learning methods in seismic prospecting, the dense prediction problem of seismic facies requires large amounts of annotated seismic facies data, which often are unavailable. These valuable labels are only helpful in one model and field due to geologic heterogeneity. To overcome these challenges, we have developed a few-shot seismic facies segmentation model. Few-shot learning has been designed to learn to perform with very few labels and we design reconstructing masked traces as a pretext task for self-supervised learning to obtain a good feature extractor. By these, this model can use all seismic data from different fields, which is different from image data as the texture-based data. With two different seismic data in turn as a meta-training set and a meta-testing set, our model works well in one- and five-shot settings, which means only one label and five labels, respectively.

Estimating explainable Alzheimer’s disease likelihood map via clinically-guided prototype learning

Identifying Alzheimer’s disease (AD) involves a deliberate diagnostic process owing to its innate traits of irreversibility with subtle and gradual progression. These characteristics make AD biomarker identification from structural brain imaging (e.g., structural MRI) scans quite challenging. Using clinically-guided prototype learning, we propose a novel deep-learning approach through eXplainable AD Likelihood Map Estimation (XADLiME) for AD progression modeling over 3D sMRIs. Specifically, we establish a set of topologically-aware prototypes onto the clusters of latent clinical features, uncovering an AD spectrum manifold. Considering this pseudo map as an enriched reference, we employ an estimating network to approximate the AD likelihood map over a 3D sMRI scan. Additionally, we promote the explainability of such a likelihood map by revealing a comprehensible overview from clinical and morphological perspectives. During the inference, this estimated likelihood map served as a substitute for unseen sMRI scans for effectively conducting the downstream task while providing thorough explainable states.

Open-Set Specific Emitter Identification Based on Prototypical Networks and Extreme Value Theory

Much research has focused on classification within a closed set of emitters, while emitters outside this closed set are misclassified. This paper proposes an open-set recognition model based on prototypical networks and extreme value theory to solve the problem of specific emitter identification in open-set scenes and further improve the recognition accuracy and robustness. Firstly, a one-dimensional convolutional neural network was designed for recognizing I/Q signals, and a squeeze-and-excitation block with an attention mechanism was added to the network to increase the weights of the feature channels with high efficiency. Meanwhile, the recognition was improved by group convolution and channel shuffle. Then, the network was trained with the joint loss function based on prototype learning to complete the separation of intra-class signals and the aggregation of inter-class signals in the feature space. After the training, the Weibull model was fitted for pre-defined classes by incorporating the extreme value theory. Finally, the classification results were obtained according to the known classes and the Weibull model, effectively completing the open-set recognition. The simulation results showed that the proposed model had a higher recognition performance and robustness compared with other classical models for signals collected from five ZigBee and ten USRP 310 devices.

Query-Guided Prototype Learning with Decoder Alignment and Dynamic Fusion in Few-Shot Segmentation

Few-shot segmentation aims to segment objects belonging to a specific class under the guidance of a few annotated examples. Most existing approaches follow the prototype learning paradigm and generate category prototypes by squeezing masked feature maps extracted from images in the support set. These support prototypes may lead to inaccurate predictions when directly compared with features extracted from the query set due to the considerable distribution discrepancy between support and query features. We propose a query-guided prototype learning architecture to address this problem from two aspects: (i) We propose a cross-alignment loss for training the segmentation decoder. This loss function will help the decoder improve its robustness against the distribution discrepancy between support and query features. (ii) We build a dynamic fusion module to strengthen the original support prototype with another prototype extracted from query features. Experiments show that our method achieves promising results compared to previous prototype learning methods on PASCAL-5 i and COCO-20 i datasets.

Human generalization of internal representations through prototype learning with goal-directed attention.

The world is overabundant with feature-rich information obscuring the latent causes of experience. How do people approximate the complexities of the external world with simplified internal representations that generalize to novel examples or situations? Theories suggest that internal representations could be determined by decision boundaries that discriminate between alternatives, or by distance measurements against prototypes and individual exemplars. Each provide advantages and drawbacks for generalization. We therefore developed theoretical models that leverage both discriminative and distance components to form internal representations via action-reward feedback. We then developed three latent-state learning tasks to test how humans use goal-oriented discrimination attention and prototypes/exemplar representations. The majority of participants attended to both goal-relevant discriminative features and the covariance of features within a prototype. A minority of participants relied only on the discriminative feature. Behaviour of all participants could be captured by parameterizing a model combining prototype representations with goal-oriented discriminative attention.

Modern Synergetic Neural Network for Synthetic Aperture Radar Target Recognition.

Feature extraction is an important process for the automatic recognition of synthetic aperture radar targets, but the rising complexity of the recognition network means that the features are abstractly implied in the network parameters and the performances are difficult to attribute. We propose the modern synergetic neural network (MSNN), which transforms the feature extraction process into the prototype self-learning process by the deep fusion of an autoencoder (AE) and a synergetic neural network. We prove that nonlinear AEs (e.g., stacked and convolutional AE) with ReLU activation functions reach the global minimum when their weights can be divided into tuples of M-P inverses. Therefore, MSNN can use the AE training process as a novel and effective nonlinear prototypes self-learning module. In addition, MSNN improves learning efficiency and performance stability by making the codes spontaneously converge to one-hots with the dynamics of Synergetics instead of loss function manipulation. Experiments on the MSTAR dataset show that MSNN achieves state-of-the-art recognition accuracy. The feature visualization results show that the excellent performance of MSNN stems from the prototype learning to capture features that are not covered in the dataset. These representative prototypes ensure the accurate recognition of new samples.

Enhancing Instrumental Competence in Translator Training in a Higher Education Context: A Task-Based Approach

This study aims to design a task-based e-Course for teaching computer-aided translation technology in a higher education context. It also investigates the impact of the e-Course on enhancing the instrumental competence of trainees, as reflected from their use of translation products. This study employs mixed qualitative and quantitative methods using descriptive statistics, pre-post paired sample tests, and an analysis of the translation process. Results of the paired-sample tests showed a noticeable increase in the scores of trainees. The difference in means between the pre and post-tests is statistically significant at P < 0.05. The descriptive statistics of errors have also shown that translation errors decreased dramatically after completing the e-Course. Analysis of the translation process indicated that trainees developed a noticeable mastery over translation competence when employing various CAT tools in the translation process. The study recommends that translator training in a higher education context in this age of globalization and localisation should familiarise trainees with the technical aspects of the industry and align with the industry’s needs and specifications. In this sense, this study culminates in the design of a learning prototype for the teaching of various desktop and cloud-based computer-aided translation tools and not only those used by big companies in a blended, hybrid or fully online environment.

Online learning of positive and negative prototypes with explanations based on kernel expansion

Online learning of positive and negative prototypes with explanations based on kernel expansion

Open-Set Object Detection Using Classification-Free Object Proposal and Instance-Level Contrastive Learning

Detecting both known and unknown objects is a fundamental skill for robot manipulation in unstructured environments. Open-set object detection (OSOD) is a promising direction to handle the problem consisting of two subtasks: objects and background separation, and open-set object classification. In this letter, we present Openset RCNN to address the challenging OSOD. To disambiguate unknown objects and background in the first subtask, we propose to use classification-free region proposal network (CF-RPN) which estimates the objectness score of each region purely using cues from object's location and shape preventing overfitting to the training categories. To identify unknown objects in the second subtask, we propose to represent them using the complementary region of known categories in a latent space which is accomplished by a prototype learning network (PLN). PLN performs instance-level contrastive learning to encode proposals to a latent space and builds a compact region centering with a prototype for each known category. Further, we note that the detection performance of unknown objects can not be unbiasedly evaluated on the situation that commonly used object detection datasets are not fully annotated. Thus, a new benchmark is introduced by reorganizing GraspNet-1billion, a robotic grasp pose detection dataset with complete annotation. Extensive experiments demonstrate the merits of our method. We finally show that our Openset RCNN can endow the robot with an open-set perception ability to support robotic rearrangement tasks in cluttered environments.

Better Class Feature Representation for Few-Shot Object Detection: Feature Aggregation and Feature Space Redistribution

Few-shot object detection is a valuable task and is yet to achieve great progress. Most research studies generate class prototypes by support samples to guide the detection. However, there still exists some challenges with this type of approach. On the one hand, the performance is highly sensitive to the quality of class prototype encoding. But limited training data impede the procession of class prototype learning. On the other hand, similar class prototypes distribute densely in the feature space, which easily leads to misjudgment. In this paper, we propose a better class feature representation (BFR) method to obtain more representative and more discriminative class prototypes with a small size of samples. First, BFR obtains the support set feature vector with richer semantic information via different feature aggregation methods. In addition, the distance metric function is used to identify the outliers and reduce the noisy interference with dynamic weights adjustment. Finally, BFR constrains the distribution of feature prototypes through distance loss, so that different prototypes are far away from each other in the feature space to reduce misjudgment. Extensive experiments on public benchmark datasets show that our method achieves superior performance with the main components bringing an 11.3% performance improvement.