Prototype Learning Research Articles

ME-PLAN: A deep prototypical learning with local attention network for dynamic micro-expression recognition

As one of the important psychological stress reactions, Micro-expressions (MEs) are spontaneous and subtle facial movements, which usually occur in a high-stake situation and can reveal genuine human feelings and cognition. ME, Recognition (MER) has essential applications in many fields such as lie detection, criminal investigation, and psychological healing. However, due to the challenges of learning discriminative ME features via fleeting facial subtle reactions as well as the shortage of available MEs data, this research topic is still far from well-studied. To this end, in this paper, we propose a deep prototypical learning framework, namely ME-PLAN, with a local attention mechanism for the MER problem. Specifically, ME-PLAN consists of two components, i.e., a 3D residual prototypical network and a local-wise attention module, where the former aims to learn the precise ME feature prototypes through expression-related knowledge transfer and episodic training, and the latter could facilitate the attention to the local facial movements. Furthermore, to alleviate the dilemma that most MER methods need to depend on manually annotated apex frames, we propose an apex frame spotting method with Unimodal Pattern Constrained (UPC) and further extract ME key-frames sequences based on the detected apex frames to train our proposed ME-PLAN in an end-to-end manner. Finally, through extensive experiments and interpretable analysis regarding the apex frame spotting and MER on composite-database, we demonstrate the superiority and effectiveness of the proposed methods.

Cross-modal prototype learning for zero-shot handwritten character recognition

Traditional methods of handwritten character recognition rely on extensive labeled data. However, humans can generalize to unseen handwritten characters by watching a few printed examples in textbooks. To simulate this ability, we propose a cross-modal prototype learning method (CMPL) to realize zero-shot recognition. For each character class, a prototype is generated by mapping the printed character into a deep neural network feature space. For unseen character class, its prototype can be directly produced from a printed character sample, therefore, not requiring any handwritten samples to realize class-incremental learning. Specifically, CMPL considers different modalities simultaneously - online handwritten trajectories, offline handwritten images, and auxiliary printed character images. The joint learning of the above modalities is achieved through sharing printed prototypes between online and offline data. In zero-shot inference, we feed CMPL the printed samples to obtain corresponding class prototypes, and then the unseen handwritten character can be recognized by the nearest prototype. Our experimental results demonstrate that CMPL outperforms the state-of-the-art methods in both online and offline zero-shot handwritten Chinese character recognition. Moreover, we also show the cross-domain generalization of CMPL from two perspectives: cross-language and modern-to-ancient handwritten character recognition, focusing on the transferability between different languages and different styles (i.e., modern and historical handwritings).

Generalized zero-shot domain adaptation with target unseen class prototype learning

Generalized zero-shot domain adaptation with target unseen class prototype learning

Multi-Level Functional Connectivity Fusion Classification Framework for Brain Disease Diagnosis.

Brain disease diagnosis is a new hotspot in the cross research of artificial intelligence and neuroscience. Quantitative analysis of functional magnetic resonance imaging (fMRI) data can provide valuable biomarkers that contributes to clinical diagnosis, and the analysis of functional connectivity (FC) has become the primary method. However, previous studies mainly focus on brain disease classification based on the low-order FC features, ignoring the potential role of high-order functional relationships among brain regions. To solve this problem, this study proposed a novel multi-level FC fusion classification framework (MFC) for brain disease diagnosis. We firstly designed a deep neural network (DNN) model to extract and learn abstract feature representations for the constructed low-order and high-order FC patterns. Both unsupervised and supervised learning steps were performed during the DNN model training, and the prototype learning was introduced in the supervised fine-tuning to improve the intra-class compactness and inter-class separability of the feature representation. Then, we combined the learned multi-level abstract FC features and trained an ensemble classifier with a hierarchical stacking learning strategy for the brain disease classification. Systematic experiments were conducted on two real large-scale fMRI datasets. Results showed that the proposed MFC model obtained robust classification performance for different preprocessing pipelines, different brain parcellations, and different cross-validation schemes, suggesting the effectiveness and generality of the proposed MFC model. Overall, this study provides a promising solution to combine the informative low-order and high-order FC patterns to further promote the classification of brain diseases.

A similarity-guided segmentation model for garbage detection under road scene

The development of computer vision technology provides a possible path for realizing intelligent control of road sweepers to reduce energy waste in urban street cleaning work. For garbage segmentation of seven categories under road scene, we introduce an efficient deep-learning-based method. Our model follows a lightweight structure with a feature pyramid attention (FPA) module employed in the decoder to enhance feature integration at multi-levels. Besides, a similarity guidance (SG) module is added to the decoder branches, which calculates the cosine distance between learned prototypes and feature maps to guide the segmentation results from a metric learning perspective. Our model has less than 3 M parameters and can run at over 65 FPS in an RTX 2070 GPU. Experimental results demonstrate that our method can yield competitive results in terms of speed and accuracy trade-off, with overall mean intersection-over-union (mIoU) reaching 0.87 and 0.67, respectively, on two garbage data sets we built. Besides, our model can perform acceptable category-balanced segmentation from less than 20 annotated images per category by introducing the SG module.Graphical

A New Competitive Neural Architecture for Object Classification

In this paper, we propose a new neural architecture for object classification, made up from a set of competitive layers whose number and size are dynamically learned from training data using a two-step process that combines unsupervised and supervised learning modes. The first step consists in finding a set of one or more optimal prototypes for each of the c classes that form the training data. For this, it uses the unsupervised learning and prototype generator algorithm called fuzzy learning vector quantization (FLVQ). The second step aims to assess the quality of the learned prototypes in terms of classification results. For this, the c classes are reconstructed by assigning each object to the class represented by its nearest prototype, and the obtained results are compared to the original classes. If one or more constructed classes differ from the original ones, the corresponding prototypes are not validated and the whole process is repeated for all misclassified objects, using additional competitive layers, until no difference persists between the constructed and the original classes or a maximum number of layers is reached. Experimental results show the effectiveness of the proposed method on a variety of well-known benchmark data sets.

Efficient Open-Set Recognition for Interference Signals Based on Convolutional Prototype Learning

Interference classification plays an important role in anti-jamming communication. Although the existing interference signal recognition methods based on deep learning have a higher accuracy than traditional methods, these have poor robustness while rejecting interference signals of unknown classes in interference open-set recognition (OSR). To ensure the classification accuracy of the known classes and the rejection rate of the unknown classes in interference OSR, we propose a new hollow convolution prototype learning (HCPL) in which the inner-dot-based cross-entropy loss (ICE) and the center loss are used to update prototypes to the periphery of the feature space so that the internal space is left for the unknown class samples, and the radius loss is used to reduce the impact of the prototype norm on the rejection rate of unknown classes. Then, a hybrid attention and feature reuse net (HAFRNet) for interference signal classification was designed, which contains a feature reuse structure and hybrid domain attention module (HDAM). A feature reuse structure is a simple DenseNet structure without a transition layer. An HDAM can recalibrate both time-wise and channel-wise feature responses by constructing a global attention matrix automatically. We also carried out simulation experiments on nine interference types, which include single-tone jamming, multitone jamming, periodic Gaussian pulse jamming, frequency hopping jamming, linear sweeping frequency jamming, second sweeping frequency jamming, BPSK modulation jamming, noise frequency modulation jamming and QPSK modulation jamming. The simulation results show that the proposed method has considerable classification accuracy of the known classes and rejection performance of the unknown classes. When the JNR is −10 dB, the classification accuracy of the known classes of the proposed method is 2–7% higher than other algorithms under different openness. When the openness is 0.030, the unknown class rejection performance plateau of the proposed method reaches 0.9883, while GCPL is 0.9403 and CG-Encoder is 0.9869; when the openness is 0.397, the proposed method is more than 0.89, while GCPL is 0.8102 and CG-Encoder is 0.9088. However, the rejection performance of unknown classes of CG-Encoder is much worse than that of the proposed method under low JNR. In addition, the proposed method requires less storage resources and has a lower computational complexity than CG-Encoder.

Federating recommendations using differentially private prototypes

Machine learning methods exploit similarities in users’ activity patterns to provide recommendations in applications across a wide range of fields including entertainment, dating, and commerce. However, in domains that demand protection of personally sensitive data, such as medicine or banking, how can we learn recommendation models without accessing the sensitive data and without inadvertently leaking private information? Many situations in the medical field prohibit centralizing the data from different hospitals and thus require learning from information kept in separate databases. We propose a new federated approach to learning global and local private models for recommendation without collecting raw data, user statistics, or information about personal preferences. Our method produces a set of locally learned prototypes that allow us to infer global behavioral patterns while providing differential privacy guarantees for users in any database of the system. By requiring only two rounds of communication, we both reduce the communication costs and avoid excessive privacy loss associated with typical federated learning iterative procedures. We test our framework on synthetic data, real federated medical data, and a federated version of Movielens ratings. We show that local adaptation of the global model allows the proposed method to outperform centralized matrix-factorization-based recommender system models, both in terms of the accuracy of matrix reconstruction and in terms of the relevance of recommendations, while maintaining provable privacy guarantees. We also show that our method is more robust and has smaller variance than individual models learned by independent entities.

Prototype Regularized Manifold Regularization Technique for Semi-Supervised Online Extreme Learning Machine.

Data streaming applications such as the Internet of Things (IoT) require processing or predicting from sequential data from various sensors. However, most of the data are unlabeled, making applying fully supervised learning algorithms impossible. The online manifold regularization approach allows sequential learning from partially labeled data, which is useful for sequential learning in environments with scarcely labeled data. Unfortunately, the manifold regularization technique does not work out of the box as it requires determining the radial basis function (RBF) kernel width parameter. The RBF kernel width parameter directly impacts the performance as it is used to inform the model to which class each piece of data most likely belongs. The width parameter is often determined off-line via hyperparameter search, where a vast amount of labeled data is required. Therefore, it limits its utility in applications where it is difficult to collect a great deal of labeled data, such as data stream mining. To address this issue, we proposed eliminating the RBF kernel from the manifold regularization technique altogether by combining the manifold regularization technique with a prototype learning method, which uses a finite set of prototypes to approximate the entire data set. Compared to other manifold regularization approaches, this approach instead queries the prototype-based learner to find the most similar samples for each sample instead of relying on the RBF kernel. Thus, it no longer necessitates the RBF kernel, which improves its practicality. The proposed approach can learn faster and achieve a higher classification performance than other manifold regularization techniques based on experiments on benchmark data sets. Results showed that the proposed approach can perform well even without using the RBF kernel, which improves the practicality of manifold regularization techniques for semi-supervised learning.

Thesis Camp As An Effort To Complete The Student's Final Project (Tesis) Of The Unimus Nursing Master’s Degree Program During The COVID-19 Pandemic

This research is a research and development that aims to develop technical thesis learning by using thesis camp as one of the learning techniques in preparing a thesis proposal for Unimus nursing masters students. The technical development of this learning is based on the length of time it takes to complete a thesis proposal experienced by most of the Unimus master's nursing students who are preparing a thesis proposal. The results of the survey on students showed that 85% had difficulty in compiling a thesis proposal for various reasons, including difficulty finding the research title, difficulty finding problems, and difficulty managing time because most of the students were already working. This has an impact on the length of the student's study period which affects the discrepancy with the educational calendar. This study was designed to: 1) examine students' understanding in the preparation of a thesis proposal, 2) create a technical prototype for thesis camp learning, 3) test the effectiveness of the thesis camp, and 4) compile a thesis camp guidebook. This research generally uses a research and development approach. The research stages used descriptive qualitative methods in exploratory studies, and development through workshop activities in the context of preparing a camp thesis guide book. During the COVID-19 pandemic, it was not possible for students to consult directly with the thesis supervisor, so this thesis camp was carried out with blended learning (70% online learning and 30% learning offline) and while offline still using the Covid-19 prevention protocol, consist of here are wearing masks, washing hands, and keeping a distance. Thesis camp is proven to be effective in increasing the timeliness of completing thesis writing for nursing master students of Unimus.

ERP prototypical matching net: a meta-learning method for zero-calibration RSVP-based image retrieval

Objective. A rapid serial visual presentation (RSVP)-based brain-computer interface (BCI) is an efficient information detection technology through detecting event-related potentials (ERPs) evoked by target visual stimuli. The BCI system requires a time-consuming calibration process to build a reliable decoding model for a new user. Therefore, zero-calibration has become an important topic in BCI research. Approach. In this paper, we construct an RSVP dataset that includes 31 subjects, and propose a zero-calibration method based on a metric-based meta-learning: ERP prototypical matching net (EPMN). EPMN learns a metric space where the distance between electroencephalography (EEG) features and ERP prototypes belonging to the same category is smaller than that of different categories. Here, we employ prototype learning to learn a common representation from ERP templates of different subjects as ERP prototypes. Additionally, a metric-learning loss function is proposed for maximizing the distance between different classes of EEG and ERP prototypes and minimizing the distance between the same classes of EEG and ERP prototypes in the metric space. Main results. The experimental results showed that EPMN achieved a balanced-accuracy of 86.34% and outperformed the comparable methods. Significance. Our EPMN can realize zero-calibration for an RSVP-based BCI system.

E-Learning course design and implementation in fuzzy logic

The goal of e-learning in fuzzy logic courses is to assist students in the learning process, create menu structures and simple operation techniques, and create prototypes for e-learning in fuzzy logic courses. The difficulty of students in implementing what they have learned stems from the fact that the advent of e-learning can make it simpler for students to access material that they do not comprehend. In this study, an analytic learning prototype was used as the research approach. The outcomes of e-learning products based on online apps in this fuzzy logic course can be used as learning material. The menu structure developed in this e-learning is a home page with an introduction to e-learning, a site page with participants, calendars, and notes. Pages that can be used to grow the network and courses are the most significant aspects of e-learning. E-learning includes material, discussion, forums, quizzes, and other activities. The findings of the validation by media specialists on this e-learning application are pretty good, indicating that it is suitable for use. According to the results of material expert validation, the material used is excellent, suggesting that it is ideal for use in fuzzy logic courses. The limited test results for Informatics study program students were in the very good category, indicating that this e-learning tool was simple to use.

Improving projected fuzzy K-means clustering via robust learning

Fuzzy K-Means clustering has been an attractive research area for many multimedia tasks. Due to the interference of the noise and outliers, the performance of fuzzy K-Means clustering has been limited. In this paper, a projected fuzzy K-Means clustering method, referred to as Robust Projected Fuzzy K-Means (RPFKM) is proposed as a response to such a challenge. RPFKM improves fuzzy K-Means clustering in three perspectives. First, unlike existing fuzzy K-Means algorithms where the clustering process is conducted in the original space, RPFKM learns the fuzzy membership relationship between samples and prototypes in the low-dimensional space to eliminate the influence of the noise and irrelevant features. Second, it employs the ℓ21 norm to reduce the contribution of outliers to the learning of prototypes. Third, it also considers the sensitivity of fuzzy clustering to the number of reduced dimensions, and the reconstruction term is introduced to hold the main energy of the original data. Furthermore, an iterative re-weighted algorithm is developed to solve the proposed method. The evaluation results of our proposed method and the state-of-the-art methods on real-world and synthetic data sets show the effectiveness and efficiency of our approach.

Decoding lip language using triboelectric sensors with deep learning

Lip language is an effective method of voice-off communication in daily life for people with vocal cord lesions and laryngeal and lingual injuries without occupying the hands. Collection and interpretation of lip language is challenging. Here, we propose the concept of a novel lip-language decoding system with self-powered, low-cost, contact and flexible triboelectric sensors and a well-trained dilated recurrent neural network model based on prototype learning. The structural principle and electrical properties of the flexible sensors are measured and analysed. Lip motions for selected vowels, words, phrases, silent speech and voice speech are collected and compared. The prototype learning model reaches a test accuracy of 94.5% in training 20 classes with 100 samples each. The applications, such as identity recognition to unlock a gate, directional control of a toy car and lip-motion to speech conversion, work well and demonstrate great feasibility and potential. Our work presents a promising way to help people lacking a voice live a convenient life with barrier-free communication and boost their happiness, enriches the diversity of lip-language translation systems and will have potential value in many applications.

Deep Convolutional Neural Networks Implementation for the Analysis of Urine Culture.

Urine culture images collected using bacteriology automation are currently interpreted by technologists during routine standard-of-care workflows. Machine learning may be able to improve the harmonization of and assist with these interpretations. A deep learning model, BacterioSight, was developed, trained, and tested on standard BD-Kiestra images of routine blood agar urine cultures from 2 different medical centers. BacterioSight displayed performance on par with standard-of-care-trained technologist interpretations. BacterioSight accuracy ranged from 97% when compared to standard-of-care (single technologist) and reached 100% when compared to a consensus reached by a group of technologists (gold standard in this study). Variability in image interpretation by trained technologists was identified and annotation "fuzziness" was quantified and found to correlate with reduced confidence in BacterioSight interpretation. Intra-testing (training and testing performed within the same institution) performed well giving Area Under the Curve (AUC) ≥0.98 for negative and positive plates, whereas, cross-testing on images (trained on one institution's images and tested on images from another institution) showed decreased performance with AUC ≥0.90 for negative and positive plates. Our study provides a roadmap on how BacterioSight or similar deep learning prototypes may be implemented to screen for microbial growth, flag difficult cases for multi-personnel review, or auto-verify a subset of cultures with high confidence. In addition, our results highlight image interpretation variability by trained technologist within an institution and globally across institutions. We propose a model in which deep learning can enhance patient care by identifying inherent sample annotation variability and improving personnel training.

PIP: Pictorial Interpretable Prototype Learning for Time Series Classification.

Time series classifiers are not only challenging to design, but they are also notoriously difficult to deploy for critical applications because end users may not understand or trust black-box models. Despite new efforts, explanations generated by other interpretable time series models are complicated for non-engineers to understand. The goal of PIP is to provide time series explanations that are tailored toward specific end users. To address the challenge, this paper introduces PIP, a novel deep learning architecture that jointly learns classification models and meaningful visual class prototypes. PIP allows users to train the model on their choice of class illustrations. Thus, PIP can create a user-friendly explanation by leaning on end-users definitions. We hypothesize that a pictorial description is an effective way to communicate a learned concept to non-expert users. Based on an end-user experiment with participants from multiple backgrounds, PIP offers an improved combination of accuracy and interpretability over baseline methods for time series classification.

Implicit prototype learning in patients with memory deficit: Evidence from a study with Alzheimer's disease patients.

In a previous study (Zannino et al., 2012), it was demonstrated that individuals with amnestic mild cognitive impairment (MCI) were unimpaired on a new prototype learning task consisting of morphed faces (face prototype learning task [FPLT]). This paradigm was devised to improve on the classical dot pattern task by ruling out any reliance on residual episodic memory or working memory resources. In the present study, we aimed to demonstrate: first, that people with even more severe episodic memory impairment than MCI are unimpaired on a fully implicit prototype learning task and second, that the dot pattern task, at variance with the FPLT, requires a no negligible contribution from the episodic memory system. Twenty-four persons with Alzheimer's disease (AD) and 48 healthy controls took part in this experiment. As in the original study, in addition to the FPLT, they were also administered the classical dot pattern task and an ordinary forced-choice face recognition task. AD performed like normal controls in the FPLT but scored significantly worse on the dot pattern task and the face recognition task. Interestingly, although performance on the face recognition task did not correlate with that on the FPLT, a significant correlation was observed between the face recognition and the dot pattern task. Results support both of our claims: first, that also severe amnesic people can learn new visual prototypes with a fully implicit paradigm and, second, that the classical dot pattern task requires some degree of episodic resources. Further research is needed to rule out the role of working memory in solving the FPLT. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Few-Shot Semantic Segmentation via Frequency Guided Neural Network

Prototype learning is extensively used in few-shot semantic segmentation due to its excellent capability of semantic information extraction and effective prevention of overfitting. The previous prototype based methods ignore the frequency discrepancy inside the object, thereby leading to semantic confusion of the object. In this paper, we propose a frequency guided network (FGNet) which explicitly models the semantic information of different frequencies and precisely guides the semantic alignment of the object. Specifically, the proposed FGNet consists of two modules: a frequency separation module (FSM) and a multi-guided feature enrichment module (MG-FEM) to complete the multi-frequency semantic information extraction and alignment, respectively. Experiments on PASCAL- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$5^{i}$</tex-math></inline-formula> dataset show that our FGNet achieves mIoU score of 61.2% in 1-shot which surpasses the state-of-the-art methods.

Class-Specific Semantic Reconstruction for Open Set Recognition.

Open set recognition enables deep neural networks (DNNs) to identify samples of unknown classes, while maintaining high classification accuracy on samples of known classes. Existing methods based on auto-encoder (AE) and prototype learning show great potential in handling this challenging task. In this study, we propose a novel method, called Class-Specific Semantic Reconstruction (CSSR), that integrates the power of AE and prototype learning. Specifically, CSSR replaces prototype points with manifolds represented by class-specific AEs. Unlike conventional prototype-based methods, CSSR models each known class on an individual AE manifold, and measures class belongingness through AE's reconstruction error. Class-specific AEs are plugged into the top of the DNN backbone and reconstruct the semantic representations learned by the DNN instead of the raw image. Through end-to-end learning, the DNN and the AEs boost each other to learn both discriminative and representative information. The results of experiments conducted on multiple datasets show that the proposed method achieves outstanding performance in both close and open set recognition and is sufficiently simple and flexible to incorporate into existing frameworks.

Incremental Learning With Open-Set Recognition for Remote Sensing Image Scene Classification

Image scene classification aiming to assign specific semantic labels for each image is vital important for the applications of remote sensing (RS) data. In real world, since the observation environment is open and dynamic, RS images are collected sequentially and the numbers of images and classes grow rapidly over time. Most existing scene classification methods are offline learning algorithms which are inefficient and unscalable for this scenario. In this paper, an incremental learning with open set recognition framework is proposed for RS image scene classification in the open and dynamic environment, called ILOSR, which can identify the unknown classes from a stream of data and learn these new classes incrementally. Specifically, a controllable convex hull-based exemplar selection strategy is designed to address the catastrophic forgetting issue in incremental learning, which can reduce training time and memory footprint effectively. In addition, a new loss function based on prototype learning and uncertainty measurement is proposed for open set recognition to enhance the inter-class discrimination and intra-class compactness of the learned deep features. Experimental results on real remote sensing datasets demonstrate that the proposed method can not only outperform the state-of-the-art approaches on offline classification, incremental learning and OSR problem separately, but also achieve better and more stable performance in the experiments for incremental learning with OSR.