Unseen Classes Research Articles

Towards open-set text recognition via label-to-prototype learning

Scene text recognition is a popular research topic which is also extensively utilized in the industry. Although many methods have achieved satisfactory performance for the close-set text recognition challenges, these methods lose feasibility in open-set scenarios, where collecting data or retraining models for novel characters could yield a high cost. For example, annotating samples for foreign languages can be expensive, whereas retraining the model each time when a “novel” character is discovered from historical documents costs both time and resources. In this paper, we introduce and formulate a new open-set text recognition task which demands the capability to spot and recognize novel characters without retraining. A label-to-prototype learning framework is also proposed as a baseline for the new task. Specifically, the framework introduces a generalizable label-to-prototype mapping function to build prototypes (class centers) for both seen and unseen classes. An open-set predictor is then utilized to recognize or reject samples according to the prototypes. The implementation of rejection capability over out-of-set characters allows automatic spotting of unknown characters in the incoming data stream. Extensive experiments show that our method achieves promising performance on a variety of zero-shot, close-set, and open-set text recognition datasets.

Unbiased feature generating for generalized zero-shot learning

Generalized zero-shot learning (GZSL) aims at training a model on seen data to recognize objects from both seen and unseen classes. Existing generated-based methods show encouraging performance by directly generating unseen samples. However, due to insufficient exploration of unseen label space and limited class-wise semantic descriptions, existing methods still face the bias problem. In this paper, we divide the bias problem into seen-biased and neighbor-biased problems and propose a GZSL method named Unbiased Feature Generating. For the seen-biased problem, we train a classifier in complete label space by introducing the discriminative information contained in fake unseen samples. For the neighbor-biased problem, we generate untypical samples and refine the classification boundaries among neighbor classes. The classifier in complete label space and generator are trained in an iterative process to complement each other. The experimental results on four widely used datasets verify our method achieves encouraging performance compared with the state-of-the-art methods.

Multilevel Features-Guided Network for Few-Shot Segmentation

The purpose of few-shot semantic segmentation is to segment unseen classes with only a few labeled samples. However, most methods ignore the guidance of low-level features for segmentation, leading to unsatisfactory results. Therefore, we propose a multilevel features-guided network using convolutional neural network techniques, which fully utilizes features from each level. It includes two novel designs: (1) a similarity-guided feature reinforcement module (SRM), which uses features from different levels, it enables sufficient guidance from the support set to the query set, thus avoiding the situation that some feature information is ignored in deep network computation, (2) a method that bridges query features at each level to the decoder to guide the segmentation, making full use of local and edge information to improve model performance. We experiment on PASCAL-5i and COCO-20i datasets to demonstrate the effectiveness of the model, the results in 1-shot setting and 5-shot setting on PASCAL-5i are 64.7% and 68.0%, which are 3.9% and 6.1% higher than the baseline model, respectively, and the results on the COCO-20i are also improved.

Residual-Prototype Generating Network for Generalized Zero-Shot Learning

Conventional zero-shot learning aims to train a classifier on a training set (seen classes) to recognize instances of novel classes (unseen classes) by class-level semantic attributes. In generalized zero-shot learning (GZSL), the classifier needs to recognize both seen and unseen classes, which is a problem of extreme data imbalance. To solve this problem, feature generative methods have been proposed to make up for the lack of unseen classes. Current generative methods use class semantic attributes as the cues for synthetic visual features, which can be considered mapping of the semantic attribute to visual features. However, this mapping cannot effectively transfer knowledge learned from seen classes to unseen classes because the information in the semantic attributes and the information in visual features are asymmetric: semantic attributes contain key category description information, while visual features consist of visual information that cannot be represented by semantics. To this end, we propose a residual-prototype-generating network (RPGN) for GZSL that extracts the residual visual features from original visual features by an encoder–decoder and synthesizes the prototype visual features associated with semantic attributes by a disentangle regressor. Experimental results show that the proposed method achieves competitive results on four GZSL benchmark datasets with significant gains.

OCSTN: One-class time-series classification approach using a signal transformation network into a goal signal

One-class classification (OCC) is a classification task where the training data have only one class. The goal is to classify input data into one seen class or other unseen classes. This paper proposes an OCC approach using a signal transformation network (OCSTN), which aims to process univariate time-series data. The main contribution is developing a signal transformation network (STN) that aims to transform input signals into one signal, namely the goal signal. Moreover, the model error of the STN is a distance metric between the goal signal and the model output. The STN model learns from one-class signals. Therefore, model error for one class is small relative to other classes. Accordingly, OCSTN could discriminate between seen and unseen classes using the model errors. The proposed OCSTN is evaluated using two ballistocardiography (BCG) datasets. The OCSTN achieves fair results in both AUC scores and processing speed. OCSTN has a weak point in training diverse signals. In addition, the entropy and smoothness of the goal signal are highly related to the AUC score.

A randomized prospective study of a hybrid rule- and data-driven virtual patient

AbstractRandomized prospective studies represent the gold standard for experimental design. In this paper, we present a randomized prospective study to validate the benefits of combining rule-based and data-driven natural language understanding methods in a virtual patient dialogue system. The system uses a rule-based pattern matching approach together with a machine learning (ML) approach in the form of a text-based convolutional neural network, combining the two methods with a simple logistic regression model to choose between their predictions for each dialogue turn. In an earlier, retrospective study, the hybrid system yielded a nearly 50% error reduction on our initial data, in part due to the differential performance between the two methods as a function of label frequency. Given these gains, and considering that our hybrid approach is unique among virtual patient systems, we compare the hybrid system to the rule-based system by itself in a randomized prospective study. We evaluate 110 unique medical student subjects interacting with the system over 5,296 conversation turns, to verify whether similar gains are observed in a deployed system. This prospective study broadly confirms the findings from the earlier one but also highlights important deficits in our training data. The hybrid approach still improves over either rule-based or ML approaches individually, even handling unseen classes with some success. However, we observe that live subjects ask more out-of-scope questions than expected. To better handle such questions, we investigate several modifications to the system combination component. These show significant overall accuracy improvements and modest F1 improvements on out-of-scope queries in an offline evaluation. We provide further analysis to characterize the difficulty of the out-of-scope problem that we have identified, as well as to suggest future improvements over the baseline we establish here.

Benchmarking knowledge-driven zero-shot learning

External knowledge (a.k.a. side information) plays a critical role in zero-shot learning (ZSL) which aims to predict with unseen classes that have never appeared in training data. Several kinds of external knowledge, such as text and attribute, have been widely investigated, but they alone are limited with incomplete semantics. Some very recent studies thus propose to use Knowledge Graph (KG) due to its high expressivity and compatibility for representing kinds of knowledge. However, the ZSL community is still in short of standard benchmarks for studying and comparing different external knowledge settings and different KG-based ZSL methods. In this paper, we proposed six resources covering three tasks, i.e., zero-shot image classification (ZS-IMGC), zero-shot relation extraction (ZS-RE), and zero-shot KG completion (ZS-KGC). Each resource has a normal ZSL benchmark and a KG containing semantics ranging from text to attribute, from relational knowledge to logical expressions. We have clearly presented these resources including their construction, statistics, data formats and usage cases w.r.t. different ZSL methods. More importantly, we have conducted a comprehensive benchmarking study, with a few classic and state-of-the-art methods for each task, including a method with KG augmented explanation. We discussed and compared different ZSL paradigms w.r.t. different external knowledge settings, and found that our resources have great potential for developing more advanced ZSL methods and more solutions for applying KGs for augmenting machine learning. All the resources are available at https://github.com/China-UK-ZSL/Resources_for_KZSL.

Discrete asymmetric zero-shot hashing with application to cross-modal retrieval

In recent years, cross-modal retrieval technology has attracted extensive attention with the massive growth of multimedia data. However, most cross-modal hashing methods mainly focus on exploring the retrieval of seen classes, while ignoring the retrieval of unseen classes. Therefore, traditional cross-modal hashing methods cannot achieve satisfactory performances in zero-shot retrieval. To mitigate this challenge, in this paper, we propose a novel zero-shot cross-modal retrieval method called discrete asymmetric zero-shot hashing(DAZSH), which fully exploits the supervised knowledge of multimodal data. Specifically, it integrates pairwise similarity, class attributes and semantic labels to guide zero-shot hashing learning. Moreover, our proposed DAZSH method combines the data features with the class attributes to obtain a semantic category representation for each category. Therefore, the relationships between seen and unseen classes can be effectively captured by learning a category representation vector for each instance. Therefore, the supervised knowledge can be transferred from the seen classes to the unseen classes. In addition, we develop an efficient discrete optimization strategy to solve the proposed model. Massive experiments on three benchmark datasets show that our proposed approach has achieved promising results in cross-modal retrieval tasks. The source code of this paper can be obtained from https://github.com/szq0816/DAZSH.

MFF: Multi-modal feature fusion for zero-shot learning

Generative Zero-Shot Learning (ZSL) methods generally generate pseudo-samples/features based on the semantic description information of unseen classes, thereby transforming ZSL tasks into traditional supervised learning tasks. Under this learning paradigm, the quality of pseudo-samples/features guided by the classes’ semantic description information is the key to the success of the model. However, the semantic description information used in the existing generative methods is mainly the low-dimensional representation (e.g., attributes) of classes, which leads to the low quality of the generated pseudo-samples/features and may aggravate the problem of domain shift. To alleviate this problem, we introduce the visual principal component feature, which is extracted by a principal component analysis network, to make up for the deficiency of using only semantic description information and propose a novel Variational Auto-Encoder (VAE) and Generative Adversarial Network (GAN) based generative method for ZSL, which we call Multi-modal Feature Fusion algorithm (MFF). In MFF, the input of different modal information enables VAE better fit the original data distribution and the proposed alignment loss ensures the consistency of the generated visual features and the corresponding semantic features. With the help of high-quality pseudo-samples/features, the ZSL model can make more accurate predictions for unseen classes. Extensive experiments on five public datasets demonstrate that our proposed algorithm outperforms several state-of-the-art methods under both ZSL and generalized ZSL settings.

Semantic guided knowledge graph for large-scale zero-shot learning

Zero-shot learning has received growing attention, which aims to improve generalization to unseen concepts. The key challenge in zero-shot tasks is to precisely model the relationship between seen and unseen classes. Most existing zero-shot learning methods capture inter-class relationships via a shared embedding space, leading to inadequate use of relationships and poor performance. Recently, knowledge graph-based methods have emerged as a new trend of zero-shot learning. These methods use a knowledge graph to accurately model the inter-class relationships. However, the currently dominant method for zero-shot learning directly extracts the fixed connection from off-the-shelf WordNet, which will inherit the inherent noise in WordNet. In this paper, we propose a novel method that adopts class-level semantic information as a guidance to construct a new semantic guided knowledge graph (SG-KG), which can correct the errors in the existing knowledge graph and accurately model the inter-class relationships. Specifically, our method includes two main steps: noise suppression and semantic enhancement. Noise suppression is used to eliminate noise edges in the knowledge graph, and semantic enhancement is used to connect two classes with strong relations. To promote high efficient information propagation among classes, we develop a novel multi-granularity fusion network (MGFN) that integrates discriminative information from multiple GCN branches. Extensive experiments on the large-scale ImageNet-21K dataset and AWA2 dataset demonstrate that our method consistently surpasses existing methods and achieves a new state-of-the-art result.

Tf-GCZSL: Task-free generalized continual zero-shot learning

Learning continually from a stream of training data or tasks with an ability to learn the unseen classes using a zero-shot learning framework is gaining attention in the literature. It is referred to as continual zero-shot learning (CZSL). Existing CZSL requires clear task-boundary information during training which is not practically feasible. This paper proposes a task-free generalized CZSL (Tf-GCZSL) method with short-term/long-term memory to overcome the requirement of task-boundary in training. A variational autoencoder (VAE) handles the fundamental ZSL tasks. The short-term and long-term memory help to overcome the condition of the task boundary in the CZSL framework. Further, the proposed Tf-GCZSL method combines the concept of experience replay with dark knowledge distillation and regularization to overcome the catastrophic forgetting issues in a continual learning framework. Finally, the Tf-GCZSL uses a fully connected classifier developed using the synthetic features generated at the latent space of the VAE. The performance of the proposed Tf-GCZSL is evaluated in the existing task-agnostic prediction setting and the proposed task-free setting for the generalized CZSL over the five ZSL benchmark datasets. The results clearly indicate that the proposed Tf-GCZSL improves the prediction at least by 12%, 1%, 3%, 4%, and 3% over existing state-of-the-art and baseline methods for CUB, aPY, AWA1, AWA2, and SUN datasets, respectively in both settings (task-agnostic prediction and task-free learning). The source code is available at https://github.com/Chandan-IITI/Tf-GCZSL.

Few-shot learning with unsupervised part discovery and part-aligned similarity

Few-shot learning aims to recognize novel concepts with only a few examples. To this end, previous studies resort to acquiring a strong inductive bias via meta-learning on a group of similar tasks, which however needs a large labeled base dataset to sample training tasks. In this paper, we show that such inductive bias can be learned from a flat collection of unlabeled images, and instantiated as transferable representations among seen and unseen classes. Specifically, we propose a novel unsupervised Part Discovery Network (PDN) to learn transferable representations from unlabeled images, which automatically selects the most discriminative part from an input image and then maximizes its similarities to the global view of the input and other neighbors with similar semantics. To better leverage the learned representations for few-shot learning, we further propose Part-Aligned Similarity (PAS), the key of which is to measure image similarities based on a set of discriminative and aligned parts. We conduct extensive studies on five popular few-shot learning datasets to evaluate our approach. The experimental results show that our approach outperforms previous unsupervised methods by a large margin and is even comparable with state-of-the-art supervised methods.

Bidirectional Flow Decision Tree for Reliable Remote Sensing Image Scene Classification

Remote sensing image scene classification (RSISC), which aims to classify scene categories for remote sensing imagery, has broad applications in various fields. Recent deep learning (DL) successes have led to a new wave of RSISC applications; however, they lack explainability and trustworthiness. Here, we propose a bidirectional flow decision tree (BFDT) module to create a reliable RS scene classification framework. Our algorithm combines BFDT and Convolutional Neural Networks (CNNs) to make the decision process easily interpretable. First, we extract multilevel feature information from the pretrained CNN model, which provides the basis for constructing the subsequent hierarchical structure. Then the model uses the discriminative nature of scene features at different levels to gradually refine similar subsets and learn the interclass hierarchy. Meanwhile, the last fully connected layer embeds decision rules for the decision tree from the bottom up. Finally, the cascading softmax loss is used to train and learn the depth features based on the hierarchical structure formed by the tree structure that contains rich remote sensing information. We also discovered that superclass results can be obtained well for unseen classes due to its unique tree structure hierarchical property, which results in our model having a good generalization effect. The experimental results align with theoretical predictions using three popular datasets. Our proposed framework provides explainable results, leading to correctable and trustworthy approaches.

Coarse-to-fine few-shot classification with deep metric learning

Few-shot classification predicts the labels of unseen samples using only a few labeled samples, and employs the samples of the classes disjoint with unseen classes for model training. It faces two primary challenges, i.e., handling sample pairs with different similarity degrees by single classifier, and learning discriminant patterns from very few labeled samples per class. To address them, this work presents a Coarse-to-Fine few-shot classification framework under the guidance of Metric-based Auxiliary learning, abbreviated as CFMA. In particular, it discriminates the image pairs with large difference by capturing global features, and models the similarity relation of the image pairs with small difference by mining the local region of interests. Moreover, CFMA adopts deep metric learning to improve the model adaptivity on the set of limited samples, and generates pseudo labels to dynamically guide the coarse learning in iteration. Empirical studies on several benchmark databases, including mini-ImageNet, tiered-ImageNet, and CUB, demonstrate that our method achieves more promising classification performance compared to many state-of-the-art alternatives.

OCFSP: self-supervised one-class classification approach using feature-slide prediction subtask for feature data

One-class classification (OCC) is a machine learning problem where training data has only one class. Recently, self-supervised OCC algorithms have been increasing attention. These algorithms train the model for pretext tasks and use the model error for OCC. However, these tasks are specialized for images, and applying them to feature data is not practical or appropriate for such a purpose. The motivation of this study is to apply self-supervised OCC to feature data. For this purpose, this paper proposes an OCC approach using feature-slide prediction (FSP) subtask for feature data (OCFSP). The main originality is the FSP subtask, which is the first classification subtask for feature data. In particular, the proposed method creates a self-labeled dataset by generating additional feature vectors with the feature slide of original vectors and self-annotating these vectors as the number of the slides. Such a dataset is applied to train a multi-class classifier to predict the number of feature slides. Since this classification model learns data from only one class, the FSP accuracy for a seen class is higher relative to unseen classes. Accordingly, OCC could be made using the accuracy of FSP. The proposed methods are experimented with using the imbalanced-learn, covtype, and kddcup datasets. OCFSP shows fair accuracy where few training data is given. In addition, classification subtask for feature data shows a relatively fast testing speed, unlike image data. Therefore, the bottleneck of the self-supervised approach is considered the memory size, which is the main difference between image and feature data. Source code is uploaded at https://github.com/ToshiHayashi/OCFSP

A framework to enhance generalization of deep metric learning methods using general discriminative feature learning and class adversarial neural networks

Deep Metric Learning (DML) methods automatically extract features from data and learn a non-linear transformation from the input to a semantically embedding space. Many DML methods focused to enhance the discrimination power of the learned metric by proposing novel sampling strategies or loss functions. This approach is very helpful when both the training and test examples are selected from the same set of categories. However, it is less effective in many applications of DML such as image retrieval and person-reidentification. Here, the DML should learn general semantic concepts from observed classes and employ them to rank or identify objects from unseen categories. Neglecting the generalization ability of the learned representation and just emphasizing to learn a more discriminative embedding on the observed classes may lead to the overfitting problem. To address this limitation, we propose a framework to enhance the generalization power of existing DML methods in a Zero-Shot Learning (ZSL) setting by general yet discriminative representation learning and employing a class adversarial neural network. To learn a general representation, we employ feature maps of intermediate layers in a deep neural network and enhance their discrimination power through an attention mechanism. Besides, a class adversarial network is utilized to force the deep model to seek class invariant features. We evaluate our work on widely used machine vision datasets in a ZSL setting. Extensive experimental results confirm that our framework can improve the generalization of existing DML methods, and it consistently outperforms baseline DML algorithms on unseen classes.

Zero-shot unsupervised image-to-image translation via exploiting semantic attributes

Recent studies have shown remarkable success in unsupervised image-to-image translation. However, if there is no access to enough images in target classes, learning a mapping from source classes to the target classes always suffers from mode collapse, especially the zero shot case, which limits the application of the existing methods. In this work, we propose a zero-shot unsupervised image-to-image translation framework to address this limitation, by effectively associating categories with their side information like attributes. To generalize the translator to previously unseen classes, we introduce two strategies for exploiting the semantic attribute space. First, we propose to preserve semantic relations to the visual space for effective guidance on where to map the input image. Second, expanding attribute space is introduced by utilizing attribute vectors of unseen classes, which alleviates the mapping bias for unseen classes. Both of these strategies encourage the translator to explore the modes of unseen classes. Quantitative and qualitative results on different datasets validate the effectiveness of our proposed approach. Moreover, we demonstrate that our framework can be applied to fashion design task. The code is available at https://github.com/cyq373/ZUNIT . • A framework for zero-shot unsupervised image-to-image translation • Preserving semantic relations to the visual space • Expanding attribute space using attribute vectors of unseen classes • Capable of fashion design task

Caption generation on scenes with seen and unseen object categories

Image caption generation is one of the most challenging problems at the intersection of vision and language domains. In this work, we propose a realistic captioning task where the input scenes may incorporate visual objects with no corresponding visual or textual training examples. For this problem, we propose a detection-driven approach that consists of a single-stage generalized zero-shot detection model to recognize and localize instances of both seen and unseen classes, and a template-based captioning model that transforms detections into sentences. To improve the generalized zero-shot detection model, which provides essential information for captioning, we define effective class representations in terms of class-to-class semantic similarities, and leverage their special structure to construct an effective unseen/seen class confidence score calibration mechanism. We also propose a novel evaluation metric that provides additional insights for the captioning outputs by separately measuring the visual and non-visual contents of generated sentences. Our experiments highlight the importance of studying captioning in the proposed zero-shot setting, and verify the effectiveness of the proposed detection-driven zero-shot captioning approach.

An Entropy-Guided Reinforced Partial Convolutional Network for Zero-Shot Learning

Zero-Shot Learning (ZSL) aims to transfer learned knowledge from observed classes to unseen classes via semantic correlations. A promising strategy is to learn a global-local representation that incorporates global information with extra localities (i.e., small parts/regions of inputs). However, existing methods discover localities based on explicit features without digging into the inherent properties and relationships among regions. In this work, we propose a novel Entropy-guided Reinforced Partial Convolutional Network (ERPCNet), which extracts and aggregates localities progressively based on semantic relevance and visual correlations without human-annotated regions. ERPCNet uses reinforced partial convolution and entropy guidance; it not only discovers global-cooperative localities dynamically but also converges faster for policy gradient optimization. We conduct extensive experiments to demonstrate ERPCNet’s performance through comparisons with state-of-the-art methods under ZSL and Generalized Zero-Shot Learning (GZSL) settings on four benchmark datasets. We also show that ERPCNet is time efficient and explainable through visualization analysis.

Few Shot Object Detection for SAR Images via Feature Enhancement and Dynamic Relationship Modeling

Current Synthetic Aperture Radar (SAR) image object detection methods require huge amounts of annotated data and can only detect the categories that appears in the training set. Due to the lack of training samples in the real applications, the performance decreases sharply on rare categories, which largely inhibits the detection model from reaching robustness. To tackle this problem, a novel few-shot SAR object detection framework is proposed, which is built upon the meta-learning architecture and aims at detecting objects of unseen classes given only a few annotated examples. Observing the quality of support features determines the performance of the few-shot object detection task, we propose an attention mechanism to highlight class-specific features while softening the irrelevant background information. Considering the variation between different support images, we also employ a support-guided module to enhance query features, thus generating high-qualified proposals more relevant to support images. To further exploit the relevance between support and query images, which is ignored in single class representation, a dynamic relationship learning paradigm is designed via constructing a graph convolutional network and imposing orthogonality constraint in hidden feature space, which both make features from the same category more closer and those from different classes more separable. Comprehensive experiments have been completed on the self-constructed SAR multi-class object detection dataset, which demonstrate the effectiveness of our few-shot object detection framework in learning more generalized features to both enhance the performance on novel classes and maintain the performance on base classes.