Prototype Representation Research Articles

Prototypical Concept Representation

Concepts are building blocks of human thinking. For machines, concept understanding has also been increasingly important, which makes concept representation a fundamental problem in artificial intelligence. While many concepts have their instances, the massive amount of information carried by instances has long been ignored in current concept representation, which limits the usage of these concepts in applications. In this paper, inspired by prototype theory in cognitive science, we propose prototypical concept representation for machines, which represents each concept with a distributed prototype derived from representations of its instances. For prototypical representation learning, we further introduce a novel model named Prototypical Siamese Network (PSN). PSN is trained under the supervision of <small>isA</small> determination, one of the most important concept-related applications. Results of extensive experiments demonstrate that, our method achieves state-of-the-art performance, thus validating the effectiveness of prototypical concept representation.

A Multiscale Spatial–Spectral Prototypical Network for Hyperspectral Image Few-Shot Classification

Due to the complex environment of hyperspectral image (HSI) gathering area, it is difficult to obtain a large number of labeled samples for HSI. Therefore, how to effectively achieve the HSI few-shot classification is a hot spot of current research. Prototypical network (PN) is one of the most classical few-shot learning algorithms, which has been widely employed for few-shot image classification and few-shot object detection. However, existing PN-based algorithms for HSI only utilize the single-scale spatial-spectral feature extracted from the last layer, ignoring the semantic information with different scales contained in the other layers. To solve this problem, a novel multi-scale spatial-spectral prototypical network (MSSPN) is proposed in this letter. The contribution of this letter is threefold. Firstly, a multi-scale spatial-spectral feature extraction algorithm based on ladder structure is proposed to effectively achieve the integration of spatial-spectral features with different scales. Secondly, with the theory of ladder-structure-based extraction algorithm, we design a multi-scale spatial-spectral prototype representation, which is suggested to be more robust and effective in the multi-scale spatial-spectral metric space. Finally, our proposed MSSPN has the advantage of expandability, and can be easily applied for the other PN-based few-shot learning methods. The experimental results on HSI few-shot classification indicate that our proposed MSSPN algorithm can achieve higher accuracy than the representative HSI classifiers and the existing PN-based algorithms.

Few-Shot Segmentation of Remote Sensing Images Using Deep Metric Learning

Current convolutional neural network (CNN)-based methods for remote sensing image segmentation require a large number of densely annotated images for model training and have limited generalization abilities for unseen object categories. In this letter, we propose a novel few-shot learning-based method for the semantic segmentation of remote sensing images. Our method can perform semantic labeling for unseen object categories with only a few annotated samples. More specifically, our model starts by using a deep CNN to extract high-level semantic features. The prototype representation of each class is then generated by using a masked average pooling on the feature embeddings of the support images with ground truth masks. Finally, our model performs semantic labeling over the query images by matching the feature embedding of each pixel to its nearest prototypes in the embedding space. Our model is optimized with a nonparametric metric learning-based loss function to maximize the intra-class similarity of learned prototypes while minimizing the inter-class similarity. Experiments on International Society for Photogrammetry and Remote Sensing (ISPRS) 2-D semantic labeling dataset demonstrate satisfying in-domain and cross-domain transferring abilities of our model.

Hyperspectral Image Few-Shot Classification Network Based on the Earth Mover’s Distance

Deep learning has achieved promising performance in hyperspectral image (HSI) classification. Training deep models usually requires labeling massive HSIs, which however is prohibitively time-consuming and expensive. To fill in the gap, this paper proposes a novel meta-learning method for HSI few-shot classification that conducts HSI classification with a few labeled samples. Specifically, we introduce the Earth Mover&#x2019;s Distance (EMD) as a metric. The designed EMD metric learning module aims to calculate the similarity of paired embedding features by decomposing embedding features into a set of local representations. The EMD metric aims to find the optimal matching flows between local representations that have the minimum matching cost. Furthermore, we attempt to learn class prototype representation for each hyperspectral class using the EMD metric. The proposed network effectively learns general knowledge from base HSIs and transfers such knowledge to the classification of novel HSIs. We conduct HSI few-shot classification by training on three base HSIs and classification on three novel HSIs. Extensive experimental results on three novel HSI datasets demonstrate that the proposed model outperforms existing state-of-the-art HSI methods, including two HSI few-shot methods.

Which representations of their gender group affect men's orientation towards care? the case of parental leave-taking intentions.

Men are currently underrepresented in traditionally female care-oriented (communal) engagement such as taking parental leave, whereas they are overrepresented in traditionally male (agentic) engagement such as breadwinning or leadership. We examined to what extent different prototypical representations of men affect men’s self-reported parental leave-taking intentions and more generally the future they can imagine for themselves with regard to work and care roles (i.e., their possible selves). We expected prototypes of men that combine the two basic stereotype dimensions of agency and communion to increase men’s communal intentions. In two experiments (N1 = 132, N2 = 233), we presented male participants with contrived newspaper articles that described the ideal man of today with varying degrees of agency and communion (between-subjects design with four conditions; combined agentic and communal vs. agentic vs. communal vs. control condition). Results of Experiment 1 were in line with the main hypothesis that especially presenting a combination of agency and communion increases men’s expectations for communal engagement: As compared to a control condition, men expected more to engage in caretaking in the future, reported higher parental leave-taking intentions, and tended to expect taking longer parental leave. Experiment 2 only partially replicated these findings, namely for parental leave-taking intentions. Both experiments additionally provided initial evidence for a contrast effect in that an exclusive focus on agency also increased men’s self-reported parental leave-taking intentions compared to the control condition. Yet, exclusively emphasizing communion in prototypes of men did not affect men’s communal intentions, which were high to begin with. We further did not find evidence for preregistered mechanisms. We discuss conditions and explanations for the emergence of these mixed effects as well as implications for the communication of gendered norms and barriers to men’s communal engagement more broadly.

Introspective Projection: Prototypical Representations of Policing in the Service of Rule of Law

Mass protests that have taken place over the past decade in various Western democracies have called into question the role of police in society, as officers have employed measures to contain rallies protesting for or against various issues. A number of these protests have resorted to violent means, resisting the police or protesting directly against their role and methods. The present study sought to investigate the prototypical representations of the police that lay citizens use to forge or desist identification with police officers. Social identification enables citizens to consider the police as ingroup members, facilitating respect for their authority. Conversely, identifying the police as outgroup precipitates resistance. The study involved 41 in-depth interviews carried out with citizens of Malta between May and June 2020. Thematic Networks Analysis revealed various points of consensus as well as a number of controversial themes. In particular, respondents demonstrated sceptical attitudes regarding policing on the beat for fear of overfamiliarity, rooted in introspective attributions projected at the police as merely human. Moreover, respondents expressed support for technological innovations that overcome natural psychological tendencies. The findings of this study suggest that seeking increasing trust in the police may be a red herring for policymakers. Rather, efforts should be directed at developing inter-objective systems, (e.g. body-cams), that overcome individual psychological propensities.

Recognition of Stress Activation by Unobtrusive Multi Sensing Setup

It is recognized that stress conditions play an important role in the definition of individual wellness and represent a major risk factor for most non-communicable diseases. Most studies focus on the evaluation of response to maximal stress conditions while a few of them reports results about the detection/monitoring of response to mild stimulations. In this study, we investigate the capability of some physiological signs and indicators (including Heart Rate, Heart Rate Variability, Respiratory Rate, Galvanic Skin Response) to recognize stress in response to moderate cognitive activation in daily life settings. To achieve this goal, we built up an unobtrusive platform to collect signals from healthy volunteers (10 subjects) undergoing cognitive activation via Stroop Color Word Test. We integrated our dataset with data from the Stress Recognition in the Automobile Drivers dataset. Following data harmonization, signal recordings in both datasets were split into five-minute blocks and a set of 12 features was extracted from each block. A feature selection was implemented by two complementary approaches: Sequential Forward Feature Selection (SFFS) and Auto-Encoder (AE) neural networks. Finally, we explored the use of Self-Organizing Map (SOM) to provide a flexible representation of an individual status. From the initial feature set we have determined, by SFFS analysis, that 2 of them (median Respiratory Rate and number peaks in Galvanic Skin Response signals) can discriminate activation statuses from resting ones. In addition, AE experiments also support that two features can suffice for recognition. Finally, we showed that SOM can provide a comprehensive but compact description of activation statuses allowing a fine prototypical representation of individual status.

Self-Supervised Prototype Representation Learning for Event-Based Corporate Profiling

Event-based corporate profiling aims to assess the evolving operational status of the corresponding corporate from its event sequence. Existing studies on corporate profiling have partially addressed the problem via (i) case-by-case empirical analysis by leveraging traditional financial methods, or (ii) the automatic profile inference by reformulating the problem into a supervised learning task. However, both approaches heavily rely on domain knowledge and are labor-intensive. More importantly, the task-specific nature of both approaches prevents the obtained corporate profiles from being applied to diversified downstream applications. To this end, in this paper, we propose a Self-Supervised Prototype Representation Learning (SePaL) framework for dynamic corporate profiling. By exploiting the topological information of an event graph and exploring self-supervised learning techniques, SePaL can obtain unified corporate representations that are robust to event noises and can be easily fine-tuned to benefit various down-stream applications with only a few annotated data. Specifically, we first infer the initial cluster distribution of noise-resistant event prototypes based on latent representations of events. Then, we construct four permutation-invariant self-supervision signals to guide the representation learning of the event prototype. In terms of applications, we exploit the learned time-evolving corporate representations for both stock price spike prediction and corporate default risk evaluation. Experimental results on two real-world corporate event datasets demonstrate the effectiveness of SePaL for these two applications.

Aerial scene understanding in the wild: Multi-scene recognition via prototype-based memory networks

Aerial scene recognition is a fundamental visual task and has attracted an increasing research interest in the last few years. Most of current researches mainly deploy efforts to categorize an aerial image into one scene-level label, while in real-world scenarios, there often exist multiple scenes in a single image. Therefore, in this paper, we propose to take a step forward to a more practical and challenging task, namely multi-scene recognition in single images. Moreover, we note that manually yielding annotations for such a task is extraordinarily time- and labor-consuming. To address this, we propose a prototype-based memory network to recognize multiple scenes in a single image by leveraging massive well-annotated single-scene images. The proposed network consists of three key components: 1) a prototype learning module, 2) a prototype-inhabiting external memory, and 3) a multi-head attention-based memory retrieval module. To be more specific, we first learn the prototype representation of each aerial scene from single-scene aerial image datasets and store it in an external memory. Afterwards, a multi-head attention-based memory retrieval module is devised to retrieve scene prototypes relevant to query multi-scene images for final predictions. Notably, only a limited number of annotated multi-scene images are needed in the training phase. To facilitate the progress of aerial scene recognition, we produce a new multi-scene aerial image (MAI) dataset. Experimental results on variant dataset configurations demonstrate the effectiveness of our network. Our dataset and codes are publicly available11https://github.com/Hua-YS/Prototype-based-Memory-Network..

Joint feature embedding learning and correlation filters for aircraft tracking with infrared imagery

Infrared object tracking is a key technology for infrared imaging guidance. Blurred imaging, strong ego-motion and frequent occlusion make it difficult to maintain robust tracking. We observe that the features trained on ImageNet are not suitable for aircraft tracking with infrared imagery. In addition, for deep feature-based tracking, the main computational burden comes from the feedforward pass through the pretrained deep network. To this end, we present an airborne infrared target tracking algorithm that employs feature embedding learning and correlation filters to obtain improved performance. We develop a shallow network and a contrastive center loss function to learn the prototypical representation of the aircraft in the embedding space. The feature embedding module is lightweight and integrated into the efficient convolution operator framework for aircraft tracking. Finally, to demonstrate the effectiveness of our tracking algorithm, we conduct extensive experiments on airborne infrared imagery and benchmark trackers.

Frog-GNN: Multi-perspective aggregation based graph neural network for few-shot text classification

Few-shot text classification aims to learn a classifier from very few labeled text instances per class. Previous few-shot research works in NLP are mainly based on Prototypical Networks, which encode support set samples of each class to prototype representations and compute distance between query and each class prototype. In the prototype aggregation progress, much useful information of support set and discrepancy between samples from different class are ignored. In contrast, our model focuses on all query-support pairs without information loss. In this paper, we propose a multi-perspective aggregation based graph neural network (Frog-GNN) that observes through eyes (support and query instance) and speaks by mouth (pair) for few-shot text classification. We construct a graph by pre-trained pair representations and aggregate information from neighborhoods by instance-level representations for message-passing. The final relational features of pairs imply intra-class similarity and inter-class dissimilarity after iteratively interactions among instances. In addition, our Frog-GNN with meta-learning strategy can well generalize to unseen class. Experimental results demonstrate that the proposed GNN model outperforms existing few-shot approaches in both few-shot text classification and relation classification on three benchmark datasets.

Hysteresis in centrosymmetric CuPbI3 perovskite halide: apolar dielectric or orientable dielectric?

We demonstrated the change in polarization behaviour at the surface/interface before and after light through Havriliak–Negami equation of lesser known CuPbI3. We have synthesized CuPbI3 through cold sintering technique and the polarization mechanisms are altered by increasing (cold) sintering temperature. The structure of CuPbI3 was not known and we predicted it to be hexagonal (Rm) with 21R prototype representation. The hysteresis is reported to be affected by ferroelectricity (reorientable dipoles with non-centrosymmetry), to inspect this a centrosymmetric CuPbI3 is taken. In spite of centrosymmetry, we observed that the hysteresis area and shape of IV curve in AM 1.5 G sunlight shows the drastic variation with the change in polarization behaviour. Our experimental results suggest that apolar dielectric behaviour is the cause of I–V hysteresis rather than robust ferroelectric polarization (which was absent in the present case).

Few-Shot Image and Sentence Matching via Aligned Cross-Modal Memory.

Image and sentence matching has attracted much attention recently, and many effective methods have been proposed to deal with it. But even the current state-of-the-arts still cannot well associate those challenging pairs of images and sentences containing few-shot content in their regions and words. In fact, such a few-shot matching problem is seldom studied and has become a bottleneck for further performance improvement in real-world applications. In this work, we formulate this challenging problem as few-shot image and sentence matching, and accordingly propose an Aligned Cross-Modal Memory (ACMM) model to deal with it. The model can not only softly align few-shot regions and words in a weakly-supervised manner, but also persistently store and update cross-modal prototypical representations of few-shot classes as references, without using any groundtruth region-word correspondence. The model can also adaptively balance the relative importance between few-shot and common content in the image and sentence, which leads to better measurement of overall similarity. We perform extensive experiments in terms of both few-shot and conventional image and sentence matching, and demonstrate the effectiveness of the proposed model by achieving the state-of-the-art results on two public benchmark datasets.

Comparison detector for cervical cell/clumps detection in the limited data scenario

Automated detection of cervical cancer cells/clumps has the potential to significantly reduce error rate and increase productivity in cervical cancer screening. However, most traditional methods rely on the success of accurate cell segmentation and discriminative hand-crafted features extraction. Recently there are emerging deep learning-based methods which train Convolutional Neural Networks (CNN) to classify cell patches or to detect cells from the whole image. But the former is computationally expensive, while the latter often requires a large-scale dataset with expensive annotations. In this paper we propose an efficient cervical cancer cells/clumps detection method, called Comparison detector, to deal with the limited data problem. Specifically, we utilize the state-of-the-art proposal-based object detection method, Faster R-CNN with Feature Pyramid Network (FPN) as the baseline and replace the classification of each proposal by comparing it with the prototype representation of each category. In addition, we propose to learn the prototype representation of the background category from data instead of manually choosing them by some heuristic rules. Experimental results show that the proposed Comparison detector yields significant improvement on the small dataset, achieving a mean Average Precision (mAP) of 26.3% and an Average Recall (AR) of 35.7%, both improved by about 20% comparing to the baseline. Moreover, when training on the medium-sized dataset, our Comparison detector gains a mAP of 48.8% and an AR of 64.0%, improving the AR by 5.1% and the mAP by 3.6% respectively. Our method is promising for the development of automation-assisted cervical cancer screening systems. Code and datasets are available athttps://github.com/kuku-sichuan/ComparisonDetector.

Attribute Prototype Learning for Interactive Face Retrieval

Interactive face retrieval aims at finding target subjects in face databases through human and machine interaction, which involves user feedback based on human perception and machine similarity measure in feature spaces. In this article, we propose an attribute prototype learning method to tackle the semantic gap between human and machine in face perception for fast interactive face retrieval. We reformulate the theoretical explanation of the interactive retrieval model and develop the algorithm of the heuristic solution of the model. Each module of the prototype model is learned with a set of identity-related facial attributes. The outputs of the prototype modules form the semantic representation. To adapt the prototype models across different databases, we propose a transfer selection algorithm based on the coherence measurements in interactive face retrieval. Coherence analysis proves that the proposed attribute prototype representation can effectively narrow down the semantic gap even in the case of cross-database transfer learning. The prototype representation can effectively reduce the feature dimension in the retrieval process. Real user retrieval with the Bayesian relevance feedback model shows that attribute prototype space is superior to low-level feature space and proves that interactive retrieval with attribute prototype representation can converge fast in large face databases.

The Middle Class of Malgudi

This paper discusses the depiction of ‘middle-class values’ by the celebrated Indian writer R.K. Narayan. Malgudi, a fictional South Indian town, is a prototypical representation of typical Indian society and the literary works of Narayan vividly mirror the everyday happenings. Content analysis of selected short stories, namely, ‘Forty-Five a Month,’ “Iswaran” and “Sweets for Angels,”; reveal the peculiarity of the Indian middle class and the author’s apt derision of certain lifestyle practices associated with modernity. Set in the backdrop of the 1930s, the stories capture the labyrinth of social issues and problems that plague South Asian societies. For instance, educational achievement is considered to be of utmost importance for preserving one’s status and the achievement of social mobility by the middle class. Ironically, educational success is often unsuccessful in buying freedom from alienating and oppressive livelihood practices engaging the middle class. There is also an accompanying erosion of ‘social capital’ and the expanding “radius of distrust.”

Few-Shot Ship Classification in Optical Remote Sensing Images Using Nearest Neighbor Prototype Representation

With the development of ship detection in optical remote sensing images, it is convenient to obtain accurate detection results and ship images. Owing to the superior performance of convolutional neural networks (CNNs), one way to acquire the category of ship is to train a classifier using numerous ship images. However, the classification performance of CNN may degrade in the case of a small number of training samples. To solve this problem, we propose a metric-based few-shot method to generate novel concept (class) representation using nearest neighbor prototype. Different from image-to-image measure in common few-shot methods, we use an image-to-feature measure. We map small number of samples to the feature space through CNN, and generate prototypes by computing nearest neighbor value on each dimension of the feature separately. Our method is validated on patch-level ship image dataset, a reproduced ship classification dataset based on HRSC2016. The experimental results demonstrate the accuracy and robustness of our method for ship classification with a small amount of labeled data.

Unsupervised Domain Adaptation Network With Category-Centric Prototype Aligner for Biomedical Image Segmentation

With the widespread success of deep learning in biomedical image segmentation, domain shift becomes a critical and challenging problem, as the gap between two domains can severely affect model performance when deployed to unseen data with heterogeneous features. To alleviate this problem, we present a novel unsupervised domain adaptation network, for generalizing models learned from the labeled source domain to the unlabeled target domain for cross-modality biomedical image segmentation. Specifically, our approach consists of two key modules, a conditional domain discriminator (CDD) and a category-centric prototype aligner (CCPA). The CDD, extended from conditional domain adversarial networks in classifier tasks, is effective and robust in handling complex cross-modality biomedical images. The CCPA, improved from the graph-induced prototype alignment mechanism in cross-domain object detection, can exploit precise instance-level features through an elaborate prototype representation. In addition, it can address the negative effect of class imbalance via entropy-based loss. Extensive experiments on a public benchmark for the cardiac substructure segmentation task demonstrate that our method significantly improves performance on the target domain.

Tracking prototype and exemplar representations in the brain across learning.

There is a long-standing debate about whether categories are represented by individual category members (exemplars) or by the central tendency abstracted from individual members (prototypes). Neuroimaging studies have shown neural evidence for either exemplar representations or prototype representations, but not both. Presently, we asked whether it is possible for multiple types of category representations to exist within a single task. We designed a categorization task to promote both exemplar and prototype representations and tracked their formation across learning. We found only prototype correlates during the final test. However, interim tests interspersed throughout learning showed prototype and exemplar representations across distinct brain regions that aligned with previous studies: prototypes in ventromedial prefrontal cortex and anterior hippocampus and exemplars in inferior frontal gyrus and lateral parietal cortex. These findings indicate that, under the right circumstances, individuals may form representations at multiple levels of specificity, potentially facilitating a broad range of future decisions.

Deep prototypical networks based domain adaptation for fault diagnosis

Due to the existence of dataset shifts, the distributions of data acquired from different working conditions show significant differences in real-world industrial applications, which leads to performance degradation of traditional machine learning methods. This work provides a framework that combines supervised domain adaptation with prototype learning for fault diagnosis. The main idea of domain adaptation is to apply the Siamese architecture to learn a latent space where the mapped features are inter-class separable and intra-class similar for both source and target domains. Moreover, the prototypical layer utilizes the features from Siamese architecture to learn prototype representations of each class. This supervised method is attractive because it needs very few labeled target samples. Moreover, it can be further extended to address the problem when the classes from the source and target domains are not completely overlapping. The model must generalize to unseen classes in the source dataset, given only a few examples of each new target class. Experimental results, on the Case Western Reserve University bearing dataset, show the effectiveness of the proposed framework. With increasing target samples in training, the model quickly converges with high classification accuracy.