Zero-shot Learning Methods Research Articles

Deep Attention Relation Network: A Zero-Shot Learning Method for Bearing Fault Diagnosis Under Unknown Domains

Deep learning (DL) method are extensively used for bearing fault diagnosis (BFD). Due to severe data distribution difference under variable working conditions, they have unsatisfactory performance of the BFD. Although the existing transfer learning (TL) methods might improve the diagnostic performance in different data distributions, fault data from these different domains in training have to be obtained. When a given bearing operates in a new working condition and fault data are not available, the TL methods might be invalid, and the BFD would be postponed. To solve the above problem, a novel zero-shot learning method named deep attention relation network (DARN) is proposed for the BFD under multiple unknown domains. The built DARN only trained by the data from a known domain might be used to diagnose fault types from unknown, but related domains without prior data input. In this method, a feature extraction module is constructed to generate representations of input samples, and a relation module is designed to calculate the relation score between the sample pairs to determine their categories. Meanwhile, a parallel attention mechanism is introduced into the DARN so as to enhance the representative ability of the built model. The results of experimental study indicate that the proposed method can make use of fault knowledge learnt from the single known domain for the BFD in the several unknown domains. The proposed DARN significantly outperforms the existing popular TL methods in diagnostic performance.

Low-Illumination Image Enhancement Based on Deep Learning Techniques: A Brief Review

As a critical preprocessing technique, low-illumination image enhancement has a wide range of practical applications. It aims to improve the visual perception of a given image captured without sufficient illumination. Conventional low-illumination image enhancement methods are typically implemented by improving image brightness, enhancing image contrast, and suppressing image noise simultaneously. Nevertheless, recent advances in this area are dominated by deep-learning-based solutions, and consequently, various deep neural networks have been proposed and applied to this field. Therefore, this paper briefly reviews the latest low-illumination image enhancement, ranging from its related algorithms to its unsolved open issues. Specifically, current low-illumination image enhancement methods based on deep learning are first sorted out and divided into four categories: supervised learning methods, unsupervised learning methods, semi-supervised learning methods, and zero-shot learning methods. Then, existing low-light image datasets are summarized and analyzed. In addition, various quality assessment indices for low-light image enhancement are introduced in detail. We also compare 14 representative algorithms in terms of both objective evaluation and subjective evaluation. Finally, the future development trend of low-illumination image enhancement and its open issues are summarized and prospected.

Data driven recurrent generative adversarial network for generalized zero shot image classification

Traditional Generative Adversarial Network (GAN) based Generalized Zero Shot Learning (GZSL) methods usually suffer from a problem that these methods ignore the differences between classes when using the standard normal distribution to fit the true distribution of each category, and the incompleteness of a single adversarial training makes the model unable to capture all the characteristics of the samples. To address this problem, a data-driven recurrent adversarial generative network is proposed in this paper. We first synthesize visual prototypes for unseen classes using the transformation from semantic attributes to visual prototypes learned on seen classes. Then, some noise is generated from these prototypes to synthesize the unseen samples according to the corresponding semantic attributes. During the sample generation process, a recurrent generative adversarial network is designed to facilitate the generated visual features to be more representative. Extensive experiments on five popular datasets as well as detailed ablation studies demonstrate the effectiveness and superiority of the proposed method.

Zero-Shot Learning with Noisy Labels

Zero-shot learning (ZSL) is an attractive technique that can recognize novel object classes without any visual examples, but most existing methods assume that the class labels of the training instances from seen classes are accurate and reliable. However, in some real-world scenarios, the label quality may be compromised by various factors, and ZSL may suffer from performance degradation under label noise. In this paper, we propose an effective approach that utilizes a robust loss function to handle label noise in seen classes. Specifically, we construct a new denoising framework to reduce the impact of outliers and mislabeled samples. To mitigate overfitting to noisy labelled samples, a robust loss function called ramp-style loss is employed to filter out the negative influence of data with anomalous loss values, using only samples within a normal loss range for training, thereby enhancing the overall robust performance. Additionally, we introduce the concave-convex procedure (CCCP) method to address the non-convexity issue of ramp-style loss and devise an efficient update scheme based on ADMM. Extensive experiments conducted on several benchmark datasets (AWA2, SUN, CUB) demonstrate the superior performance of our framework over state-of-the-art ZSL methods in various noisy-label environments.

Rebalanced Zero-Shot Learning.

Zero-shot learning (ZSL) aims to identify unseen classes with zero samples during training. Broadly speaking, present ZSL methods usually adopt class-level semantic labels and compare them with instance-level semantic predictions to infer unseen classes. However, we find that such existing models mostly produce imbalanced semantic predictions, i.e. these models could perform precisely for some semantics, but may not for others. To address the drawback, we aim to introduce an imbalanced learning framework into ZSL. However, we find that imbalanced ZSL has two unique challenges: (1) Its imbalanced predictions are highly correlated with the value of semantic labels rather than the number of samples as typically considered in the traditional imbalanced learning; (2) Different semantics follow quite different error distributions between classes. To mitigate these issues, we first formalize ZSL as an imbalanced regression problem which offers empirical evidences to interpret how semantic labels lead to imbalanced semantic predictions. We then propose a re-weighted loss termed Re-balanced Mean-Squared Error (ReMSE), which tracks the mean and variance of error distributions, thus ensuring rebalanced learning across classes. As a major contribution, we conduct a series of analyses showing that ReMSE is theoretically well established. Extensive experiments demonstrate that the proposed method effectively alleviates the imbalance in semantic prediction and outperforms many state-of-the-art ZSL methods.

LVQ Treatment for Zero-Shot Learning

In image classification, there are no labeled training instances for some classes, which are therefore called unseen classes or test classes. To classify these classes, zero-shot learning (ZSL) was developed, which typically attempts to learn a mapping from the (visual) feature space to the semantic space in which the classes are represented by a list of semantically meaningful attributes. However, the fact that this mapping is learned without using instances of the test classes affects the performance of ZSL, which is known as the domain shift problem. In this study, we propose to apply the learning vector quantization (LVQ) algorithm in the semantic space once the mapping is determined. First and foremost, this allows us to refine the prototypes of the test classes with respect to the learned mapping, which reduces the effects of the domain shift problem. Secondly, the LVQ algorithm increases the margin of the 1-NN classifier used in ZSL, resulting in better classification. Moreover, for this work, we consider a range of LVQ algorithms, from initial to advanced variants, and applied them to a number of state-of-the-art ZSL methods, then obtained their LVQ extensions. The experiments based on five ZSL benchmark datasets showed that the LVQ-empowered extensions of the ZSL methods are superior to their original counterparts in almost all settings.

Zero-Shot SAR Target Recognition Based on Classification Assistance

As one of main active learning methods, zero-shot target recognition with synthetic aperture radar (SAR) data has received considerable attention in recent years. Its goal is to distinguish new targets from the known classes without requiring additional training data. Existing zero-shot learning (ZSL) methods perform well on optical targets, but they fail to recognize zero-shot SAR targets. Due to the strong similarity of SAR targets between different classes, the ZSL task usually suffers from the distribution concentration problem. To tackle this problem, a Dual Branch Auto-Encoder (DBAE) network is proposed in this letter. DBAE effectively alleviates the distribution concentration problem by adding a classification assistance net. Its dual branch structure is specially designed for further improving the intra-class similarity and inter-class dissimilarity of SAR targets in the embedding space. By training with the defined hybrid loss function, DBAE automatically builds a stable embedding space. Extensive experiments on the public data set of Moving and Stationary Target Acquisition and Recognition (MSTAR) show that DBAE is of rational design and provides better or comparable ZSL recognition results.

Zero-Shot Predicate Prediction for Scene Graph Parsing

The scene graph is a structured semantic representation of an image, which represents objects and relationships with vertices and edges, respectively. Since it is impossible to manually label all potential relationships in the real world, some previous methods try to apply the zero-shot method for scene graph generation. However, existing methods take triplet (i.e., hsubject-predicate-objecti) as the basic unit of a relationship. Each element (i.e., subject, predicate, or object) of the unseen relationship is actually seen in the training data. Therefore, they ignore the unseen predicate. To predict the unseen predicate, we introduce a novel task named zero-shot predicate prediction, which is crucial to extending existing scene graph generation methods to recognize more relationship classes. The new task is challenging and cannot be simply resolved through conventional zero-shot learning methods because there is a large intra-class variation of each predicate. Firstly, the large intra-class variation leads to the difficulty of computing the discriminative instancelevel feature of the predicate class. Secondly, the large intraclass variation also brings more difficulties when knowledge is transferred from seen classes to unseen classes. For the first challenge, we propose distilling lexical knowledge of different objects and construct multi-modal representations of pairwise objects to reduce the intra-class variation of the predicate. To respond to the second challenge, we build a compact semantic space where the representations of unseen classes are reconstructed based on the seen classes for zero-shot predicate classification. We evaluate the proposed method on the public dataset Visual Genome. The extensive experiment results under the zeroshot/few-shot/supervised settings demonstrate the effectiveness of the proposed method.

An end-to-end deep generative approach with meta-learning optimization for zero-shot object classification

Zero-shot object classification aims to recognize the object of unseen classes whose supervised data are unavailable in the training stage. Recent zero-shot learning (ZSL) methods usually propose to generate new supervised data for unseen classes by designing various deep generative networks. In this paper, we propose an end-to-end deep generative ZSL approach that trains the data generation module and object classification module jointly, rather than separately as in the majority of existing generation-based ZSL methods. Due to the ZSL assumption that unseen data are unavailable in the training stage, the distribution of generated unseen data will shift to the distribution of seen data, and subsequently causes the projection domain shift problem. Therefore, we further design a novel meta-learning optimization model to improve the proposed generation-based ZSL approach, where the parameters initialization and the parameters update algorithm are meta-learned to assist model convergence. We evaluate the proposed approach on five standard ZSL datasets. The average accuracy increased by the proposed jointly training strategy is 2.7% and 23.0% for the standard ZSL task and generalized ZSL task respectively, and the meta-learning optimization further improves the accuracy by 5.0% and 2.1% on two ZSL tasks respectively. Experimental results demonstrate that the proposed approach has significant superiority in various ZSL tasks.

Incremental Zero-Shot Learning.

The goal of zero-shot learning (ZSL) is to recognize objects from unseen classes correctly without corresponding training samples. The existing ZSL methods are trained on a set of predefined classes and do not have the ability to learn from a stream of training data. However, in many real-world applications, training data are collected incrementally; this is one of the main reasons why ZSL methods cannot be applied to certain real-world situations. Accordingly, in order to handle practical learning tasks of this kind, we introduce a novel ZSL setting, referred to as incremental ZSL (IZSL), the goal of which is to accumulate historical knowledge and alleviate Catastrophic Forgetting to facilitate better recognition when incrementally trained on new classes. We further propose a novel method to realize IZSL, which employs a generative replay strategy to produce virtual samples of previously seen classes. The historical knowledge is then transferred from the former learning step to the current step through joint training on both real new and virtual old data. Subsequently, a knowledge distillation strategy is leveraged to distill the knowledge from the former model to the current model, which regularizes the training process of the current model. In addition, our method can be flexibly equipped with the most generative-ZSL methods to tackle IZSL. Extensive experiments on three challenging benchmarks indicate that the proposed method can effectively tackle the IZSL problem effectively, while the existing ZSL methods fail.

Semantic-aligned reinforced attention model for zero-shot learning

Zero-shot learning (ZSL) aims to recognize unseen images from invisible classes, by transferring semantic knowledge from visible classes to invisible classes. Such as, although humans have never seen a zebra, if we know that “a horse with stripes is a zebra”, then we can easily recognize it when we see a zebra. Given semantic descriptions, the human can capture intrinsic visual clues from different channels or appearance factors (e.g., color, texture) on salient parts. But computers are not smart enough to recognize it with high accuracy, they still need to make progress in the learning of semantic-aligned visual representations. Therefore, we propose a semantic-aligned reinforced attention (SRA) model to improve the attributes localization ability. We aim to discover invariable features related to class-level semantic attributes from variable intra-class vision information, and thereby avoid misalignment between much visual information and simple semantic representations. Specially, during the localization of spatial attention, we develop an efficient constraint directly on feature map to ensure the intra-attention compactness and inter-attention dispersion characteristics like human gaze. While for the channel, we proposed a novel attributes attention cross entropy loss to exploit the supervision effect of each semantic attribute subset. Experiments on three ZSL benchmarks, i.e., CUB, SUN and AWA2, indicate the competitiveness of our proposed method against the state-of-the-art ZSL methods. • Propose a ZSL model to mimic human cognitive process by exploiting the reinforced attention from both space and channel. • Develop an efficient loss for constraint on feature map for better interpretation. • Achieve superior or competitive performance on widely used standard benchmarks.

Generalized Zero-Shot Learning for Image Classification—Comparing Performance of Popular Approaches

There are many areas where conventional supervised machine learning does not work well, for instance, in cases with a large, or systematically increasing, number of countably infinite classes. Zero-shot learning has been proposed to address this. In generalized settings, the zero-shot learning problem represents real-world applications where test instances are present during inference. Separately, recently, there has been increasing interest in meta-classifiers, which combine the results from individual classifications to improve the overall classification quality. In this context, the purpose of the present paper is two-fold: First, the performance of five state-of-the-art, generalized zero-shot learning methods is compared for five popular benchmark datasets. Second, six standard meta-classification approaches are tested by experiment. In the experiments undertaken, all meta-classifiers were applied to the same datasets; their performance was compared to each other and to the original classifiers.

Hyperspectral Imaging Zero-Shot Learning for Remote Marine Litter Detection and Classification

This paper addresses the development of a novel zero-shot learning method for remote marine litter hyperspectral imaging data classification. The work consisted of using an airborne acquired marine litter hyperspectral imaging dataset that contains data about different plastic targets and other materials and assessing the viability of detecting and classifying plastic materials without knowing their exact spectral response in an unsupervised manner. The classification of the marine litter samples was divided into known and unknown classes, i.e., classes that were hidden from the dataset during the training phase. The obtained results show a marine litter automated detection for all the classes, including (in the worst case of an unknown class) a precision rate over 56% and an overall accuracy of 98.71%.

Benchmarking knowledge-driven zero-shot learning

External knowledge (a.k.a. side information) plays a critical role in zero-shot learning (ZSL) which aims to predict with unseen classes that have never appeared in training data. Several kinds of external knowledge, such as text and attribute, have been widely investigated, but they alone are limited with incomplete semantics. Some very recent studies thus propose to use Knowledge Graph (KG) due to its high expressivity and compatibility for representing kinds of knowledge. However, the ZSL community is still in short of standard benchmarks for studying and comparing different external knowledge settings and different KG-based ZSL methods. In this paper, we proposed six resources covering three tasks, i.e., zero-shot image classification (ZS-IMGC), zero-shot relation extraction (ZS-RE), and zero-shot KG completion (ZS-KGC). Each resource has a normal ZSL benchmark and a KG containing semantics ranging from text to attribute, from relational knowledge to logical expressions. We have clearly presented these resources including their construction, statistics, data formats and usage cases w.r.t. different ZSL methods. More importantly, we have conducted a comprehensive benchmarking study, with a few classic and state-of-the-art methods for each task, including a method with KG augmented explanation. We discussed and compared different ZSL paradigms w.r.t. different external knowledge settings, and found that our resources have great potential for developing more advanced ZSL methods and more solutions for applying KGs for augmenting machine learning. All the resources are available at https://github.com/China-UK-ZSL/Resources_for_KZSL.

MFF: Multi-modal feature fusion for zero-shot learning

Generative Zero-Shot Learning (ZSL) methods generally generate pseudo-samples/features based on the semantic description information of unseen classes, thereby transforming ZSL tasks into traditional supervised learning tasks. Under this learning paradigm, the quality of pseudo-samples/features guided by the classes’ semantic description information is the key to the success of the model. However, the semantic description information used in the existing generative methods is mainly the low-dimensional representation (e.g., attributes) of classes, which leads to the low quality of the generated pseudo-samples/features and may aggravate the problem of domain shift. To alleviate this problem, we introduce the visual principal component feature, which is extracted by a principal component analysis network, to make up for the deficiency of using only semantic description information and propose a novel Variational Auto-Encoder (VAE) and Generative Adversarial Network (GAN) based generative method for ZSL, which we call Multi-modal Feature Fusion algorithm (MFF). In MFF, the input of different modal information enables VAE better fit the original data distribution and the proposed alignment loss ensures the consistency of the generated visual features and the corresponding semantic features. With the help of high-quality pseudo-samples/features, the ZSL model can make more accurate predictions for unseen classes. Extensive experiments on five public datasets demonstrate that our proposed algorithm outperforms several state-of-the-art methods under both ZSL and generalized ZSL settings.

Semantic guided knowledge graph for large-scale zero-shot learning

Zero-shot learning has received growing attention, which aims to improve generalization to unseen concepts. The key challenge in zero-shot tasks is to precisely model the relationship between seen and unseen classes. Most existing zero-shot learning methods capture inter-class relationships via a shared embedding space, leading to inadequate use of relationships and poor performance. Recently, knowledge graph-based methods have emerged as a new trend of zero-shot learning. These methods use a knowledge graph to accurately model the inter-class relationships. However, the currently dominant method for zero-shot learning directly extracts the fixed connection from off-the-shelf WordNet, which will inherit the inherent noise in WordNet. In this paper, we propose a novel method that adopts class-level semantic information as a guidance to construct a new semantic guided knowledge graph (SG-KG), which can correct the errors in the existing knowledge graph and accurately model the inter-class relationships. Specifically, our method includes two main steps: noise suppression and semantic enhancement. Noise suppression is used to eliminate noise edges in the knowledge graph, and semantic enhancement is used to connect two classes with strong relations. To promote high efficient information propagation among classes, we develop a novel multi-granularity fusion network (MGFN) that integrates discriminative information from multiple GCN branches. Extensive experiments on the large-scale ImageNet-21K dataset and AWA2 dataset demonstrate that our method consistently surpasses existing methods and achieves a new state-of-the-art result.

Auto‐encode the synthesis pseudo features for generalized zero‐shot learning

Zero-shot learning (ZSL) is to identify target categories without labeled data, in which semantic information is used to transfer knowledge from some seen categories. In the existing Generalized Zero-Shot Learning (GZSL) methods, domains shift problem always appeared during generating feature stage. In order to solve this problem, a new method to Auto-Encode the Synthesis Pseudo Features for the GZSL task (AESPF-GZSL) is proposed in this manuscript. Specifically, the AESPF-GZSL method trains the generated features under the semantic auto-encoder framework and exploits attention mechanism to train the generated features again. Then, the generated features are input to the classifier. The proposed method is performed on three benchmark data sets referred as to AWA, CUB and SUN. The experimental results show that the proposed method achieves the state-of-the art classifier accuracy both in ZSL and GZSL settings. In ZSL setting, the classification accuracy of our method is superior to the compared algorithms, improved by 0.40% in AWA and 0.30% in SUN, respectively. And in GZSL setting, the classification accuracy of the method is superior to the comparison algorithm 0.41% in Harmonic mean on AWA, and 1.01%, 0.62%, and 1.05% in training data set, testing data set, and harmonic average on SUN.

Zero-shot face recognition: Improving the discriminability of visual face features using a Semantic-Guided Attention Model

Zero-shot learning enables the recognition of classes not seen during training through the use of semantic information comprising a visual description of the class either in textual or attribute form. Despite the advances in the performance of zero-shot learning methods, most of the works do not explicitly exploit the correlation between the visual attributes of the image and their corresponding semantic attributes for learning discriminative visual features. In this paper, we introduce an attention-based strategy for deriving features from the image regions regarding the most prominent attributes of the image class. In particular, we train a Convolutional Neural Network (CNN) for image attribute prediction and use a gradient-weighted method for deriving the attention activation maps of the most salient image attributes. These maps are then incorporated into the feature extraction process of Zero-Shot Learning (ZSL) approaches for improving the discriminability of the features produced through the implicit inclusion of semantic information. For experimental validation, the performance of state-of-the-art ZSL methods was determined using features with and without the proposed attention model. Surprisingly, we discover that the proposed strategy degrades the performance of ZSL methods in classical ZSL datasets (AWA2), but it can significantly improve performance when using face datasets. Our experiments show that these results are a consequence of the interpretability of the dataset attributes, suggesting that existing ZSL datasets attributes are, in most cases, difficult to be identifiable in the image. Source code is available at https://github.com/CristianoPatricio/SGAM.

Zero-Shot Learning on 3D Point Cloud Objects and Beyond

Zero-shot learning, the task of learning to recognize new classes not seen during training, has received considerable attention in the case of 2D image classification. However, despite the increasing ubiquity of 3D sensors, the corresponding 3D point cloud classification problem has not been meaningfully explored and introduces new challenges. In this paper, we identify some of the challenges and apply 2D Zero-Shot Learning (ZSL) methods in the 3D domain to analyze the performance of existing models. Then, we propose a novel approach to address the issues specific to 3D ZSL. We first present an inductive ZSL process and then extend it to the transductive ZSL and Generalized ZSL (GZSL) settings for 3D point cloud classification. To this end, a novel loss function is developed that simultaneously aligns seen semantics with point cloud features and takes advantage of unlabeled test data to address some known issues (e.g., the problems of domain adaptation, hubness, and data bias). While designed for the particularities of 3D point cloud classification, the method is shown to also be applicable to the more common use-case of 2D image classification. An extensive set of experiments is carried out, establishing state-of-the-art for ZSL and GZSL on synthetic (ModelNet40, ModelNet10, McGill) and real (ScanObjectNN) 3D point cloud datasets.

Region interaction and attribute embedding for zero-shot learning

Zero-shot learning (ZSL) aims to recognize unseen class images only by using seen class images for training. In ZSL tasks, unseen classes share local attribute semantics with seen classes. However, most ZSL methods directly learn an embedding from global feature space to semantic space, which may fail to discover local attribute semantics and cause the strong bias problem. In this paper, a novel method called region interaction and attribute embedding (RIAE) is proposed. RIAE consists of two modules: the region graph network (RGN) and the attribute feature embedding (AFE). RGN constructs a region graph where each region (image patch) is regarded as a graph node for region interaction. That is, through the graph-convolution operations of RGN, the node features can aggregate the information from their neighboring node features and update themselves. In order to learn the compatibility between the updated node features and local attribute semantics, AFE is designed to assign high attention weights to the node features which have large compatibilities with attribute semantic vectors. Relying on attribute semantic vectors rather than human annotations, RIAE can adaptively extract and update local attribute features, and further learn an embedding from attribute feature space to semantic space. Extensive experiments on the benchmark datasets show that RIAE gets state-of-the-art or competitive performances.